Why is feature scaling important in machine learning?

Conditions

• Applies to iterative optimization algorithms like gradient descent.
• Applies to distance-based machine learning algorithms like KNN or K-means.

Reasoning, step by step

1. Identify the first reason: Faster Convergence. Unscaled features with vastly different ranges cause uneven step sizes in gradient descent, leading to inefficient oscillations and delayed convergence.
2. Identify the second reason: Computing Distance Appropriately. In distance-based algorithms, features with larger numerical ranges disproportionately influence the total distance, masking the true importance of other features.
3. Conclude that scaling features to comparable ranges resolves both issues, improving model training efficiency and fairness in distance calculations.

Example

The video summarizes the reasons on a slide titled "Why Feature Scaling is Important?" with two bullet points: "Faster Convergence" and "Computing Distance Appropriately".

Common misconceptions

• Assuming feature scaling is only necessary for neural networks; it is also crucial for linear regression with gradient descent and distance-based algorithms like KNN.
• Believing that a feature dominating a distance calculation is inherently more important for the prediction task, rather than just having a larger numerical scale.

Watch the explanation

Explore next

• Why does gradient descent oscillate when features are not scaled?
• How does a larger feature range affect distance calculations in algorithms like KNN?
• What is the formula for min-max normalization?