Why does gradient descent oscillate when features are not scaled?

Conditions

• Used in linear regression optimization with gradient descent.
• Features have significantly different scales (e.g., one ranges from 0 to 1, another from 100 to 1000).

Reasoning, step by step

1. Analyze the gradient descent update rule: the change in parameter $\theta_i$ is proportional to the feature value $x_i$.
2. Observe that if $x_1$ is small, the step size for $\theta_1$ will be small.
3. Observe that if $x_2$ is large, the step size for $\theta_2$ will be large.
4. Conclude that this disparity in step sizes causes the optimization path to oscillate inefficiently across the narrow dimension of the cost function's contours, delaying convergence.

Example

The video visualizes this with a contour plot showing an elongated valley and a red zigzagging path moving towards the minimum, caused by $x_1 \in (0, 1)$ and $x_2 \in (100, 1000)$.

Common misconceptions

• Believing that a smaller learning rate fixes the oscillation without scaling; it only slows down the large steps, making the small steps even slower.
• Assuming that gradient descent always takes a direct path to the minimum regardless of feature scales.

Watch the explanation

Explore next

• How does a larger feature range affect distance calculations in algorithms like KNN?
• What is the parameter update rule for gradient descent?
• Why is feature scaling important in machine learning?