Intrinsic Dimension Part 1: How Learning in Large Models Is Driven by a Few Parameters and Its Impact on Fine-Tuning
Learned over-parameterized models inherently exist within a low intrinsic dimension {Li et al.¹ and Aghajanyan et al.³}. To understand this concept better, let’s delve into the following questions:What is a model’s dimension?What is an Objective function and its landscape?What is ID and how can it be visualized through graphs of convex and non-convex landscapes?How do you identify the ID of a network?If the ID of an objective function is so low, why do we have such large networks in the...
Read