How does a larger feature range affect distance calculations in algorithms like KNN?

Conditions

• Applies to distance-based machine learning algorithms like KNN or K-means.
• Features are not scaled or normalized.
• Distance is computed using a standard metric like Euclidean distance.

Reasoning, step by step

1. Calculate the Euclidean distance between two points using the formula $\sqrt{(\Delta x_{horizontal})^2 + (\Delta x_{vertical})^2}$.
2. Observe that if the horizontal feature has a much larger range, $\Delta x_{horizontal} \gg \Delta x_{vertical}$.
3. Conclude that the vertical component becomes negligible in the sum under the square root, so the distance is approximately $|\Delta x_{horizontal}|$.
4. Recognize that this biases the algorithm to only consider the larger-scale feature.

Example

The video shows a scatter plot where a yellow line segment connecting two points is almost entirely horizontal because the horizontal feature has a much larger range. After normalization, the line redraws with significant contributions from both horizontal and vertical components.

Common misconceptions

• Believing that a feature dominating a distance calculation is inherently more important for the prediction task; it is often merely an artifact of the feature's numerical range.
• Assuming that distance-based algorithms automatically handle different feature scales correctly.

Watch the explanation

Explore next

• What is the visual and mathematical effect of applying normalization to features before computing distances?
• Why is feature scaling important in machine learning?
• What is the formula for min-max normalization?