Intrinsic Dimension

Intrinsic Dimension Part 2: Measuring the True Complexity of a Model via Random Subspace Training

In Part I of this series, we delved into the concept of Intrinsic Dimension (ID) and its implications on fine-tuning. To recap, the intrinsic dimension of an objective function measures the minimal number of parameters needed to achieve satisfactory performance on a given task (Li et al.,¹). For example, in their seminal work, the authors demonstrated that a fully connected neural network with a total of parameters (architecture: 784–200–200–10) achieved a 90% performance threshold on the MNIST dataset with an...

Read

Intrinsic Dimension Part 1: How Learning in Large Models Is Driven by a Few Parameters and Its Impact on Fine-Tuning

Learned over-parameterized models inherently exist within a low intrinsic dimension {Li et al.¹ and Aghajanyan et al.³}. To understand this concept better, let’s delve into the following questions:What is a model’s dimension?What is an Objective function and its landscape?What is ID and how can it be visualized through graphs of convex and non-convex landscapes?How do you identify the ID of a network?If the ID of an objective function is so low, why do we have such large networks in the...

Read