What Gradient Descent Is Doing

Almost every model you have heard of was trained the same way. Understanding that one procedure explains a surprising amount of what models can and cannot do.

The setup#

A model starts with parameters set to arbitrary values, so its predictions are arbitrary too. A loss function measures how wrong they are — one number, and lower is better.

Training is the process of changing the parameters to make that number smaller.

Walking downhill in fog#

Imagine standing somewhere on a landscape in thick fog, trying to reach the lowest point. You cannot see the valley. What you can do is feel which way the ground slopes beneath your feet and take a step in that direction.

Repeat that a few million times and you tend to end up somewhere low.

That is gradient descent. The gradient is the slope — which direction increases the loss fastest — and the algorithm steps the other way.

What this explains#

Several things that seem mysterious about models follow directly from this:

• Training is not exact. It is an approximate search, so two runs on the same data can produce different models.
• Local minima exist. The lowest point you can reach by walking downhill is not necessarily the lowest point on the landscape.
• More data changes the landscape, not just the walk. The shape being descended is defined by the examples the model was shown.
• The model optimises the loss, not your intention. If the loss function rewards something subtly wrong, gradient descent will find it faithfully.

The last one is the most important, and it is not a technical limitation. It is a specification problem, and it does not go away with more compute.

The algorithm is indifferent to what you meant. It minimises exactly what you wrote down.