Learning rate

The learning rate is the size of each step in Gradient descent — how far we move along the negative gradient before recomputing it. It’s conventionally written $\eta$ (the Greek letter eta) and is one of the most important hyperparameters in training a machine-learning model.

The update rule for gradient descent:

${\mathbf{w}}_{t+1}={\mathbf{w}}_t-\eta \nabla J({\mathbf{w}}_t)$

The negative gradient says which direction decreases the loss. The learning rate says how far to step in that direction.

Too small

If $\eta$ is too small, training is very slow — we creep down the hill in tiny steps and need many iterations to reach the minimum. With $\eta =0.0001$, an optimizer that would converge in 100 steps at $\eta =0.1$ might take 100,000 steps. The loss curve is smooth but flat — barely decreasing over many iterations.

Too large

If $\eta$ is too large, the updates may overshoot the minimum. We compute the gradient at the current point, take a big step in that direction, and end up on the other side of the valley — possibly at a point with higher loss than where we started. The loss may oscillate, bounce around without settling, or even diverge entirely with the parameters growing without bound.

Just right

A good learning rate produces a loss curve that drops quickly at first, then levels off as it approaches the minimum. The descent should be steady — neither glacial nor spasmodic.

Picking it

Picking $\eta$ is partly an art. Common starting values are 0.01 or 0.001. Run training for a while, look at how the loss changes, adjust:

• If the loss drops quickly and smoothly, $\eta$ is roughly right.
• If the loss stays flat, $\eta$ is too small — increase by 10×.
• If the loss bounces around or grows, $\eta$ is too large — decrease by 10×.

A worked example from the textbook: minimizing $f(x,y)=3x^2+4y^2+2x-4xy$ starting from $(1,1)$. With $\eta =0.1$, the iterations converge to the analytical minimum $(-0.5,-0.25)$ in about 25 steps. With $\eta =0.3$, the path overshoots — taking large steps past the minimum, then correcting back. With $\eta$ much larger, the path spirals outward and diverges. With $\eta =0.001$, the path creeps slowly inward and doesn’t reach the minimum within 200 iterations.

Beyond a constant

Modern training algorithms often use a learning-rate schedule — $\eta$ that varies over time. Common patterns:

• Step decay: drop $\eta$ by a factor of 10 every $K$ epochs.
• Cosine annealing: $\eta$ follows a cosine curve from ${\eta }_{\max }$ to ${\eta }_{\min }$.
• Warmup: start small, ramp up linearly to ${\eta }_{\max }$, then decay.

These help training converge faster early (when far from the minimum, big steps help) and more precisely later (when near the minimum, small steps prevent overshoot).

Adaptive methods like Adam, RMSProp, and AdaGrad adjust the learning rate per parameter based on the history of gradients, removing some of the need to tune $\eta$ by hand.