What is the parameter update rule for gradient descent?

Conditions

• The cost function J must be differentiable with respect to the parameters $\theta$.
• The learning rate $\alpha$ must be chosen appropriately to ensure convergence.

Reasoning, step by step

1. Identify the current parameter value $\theta_i$.
2. Calculate the partial derivative of the cost function $J$ with respect to $\theta_i$: $\frac{\partial J(\theta)}{\partial \theta_i}$.
3. Multiply the gradient by the learning rate $\alpha$.
4. Subtract this product from the current parameter value.
5. The update rule is $\theta_i := \theta_i - \alpha \frac{\partial J(\theta)}{\partial \theta_i}$.

Example

The video displays the general update rule on-screen as "$\theta_i := \theta_i - \alpha \frac{\partial J(\theta)}{\partial \theta_i}$" and later shows the specific rules for two parameters in linear regression.

Common misconceptions

• Assuming the update adds the gradient rather than subtracting it; gradient descent moves in the opposite direction of the gradient to minimize the cost function.
• Forgetting the learning rate $\alpha$, which controls the step size of the update.

Watch the explanation

Explore next

• Why does gradient descent oscillate when features are not scaled?
• What is the least squares cost function?
• Why is feature scaling important in machine learning?