Why is the least-squares fit Ax* identified with the orthogonal projection of b onto $C(A)$?

The solution to the normal equations is considered the least-squares solution because it satisfies the necessary and sufficient condition for minimizing the residual norm $\|\vec{b} - A\vec{x}\|$. The derivation shows that minimizing this norm is equivalent to requiring the residual $A\vec{x}^* - \vec{b}$ to be orthogonal to the column space $C(A)$.

Conditions: The original system $A\vec{x}=\vec{b}$ may be inconsistent; $A^T A \vec{x}^* = A^T \vec{b}$ has a solution

Every product $A\vec{x}$ is a member of the column space $C(A)$ because matrix-vector multiplication is defined as a linear combination of the columns of $A$. Specifically, if $A = [\vec{a}_1 \ \vec{a}_2 \ \cdots \ \vec{a}_k]$ and $\vec{x} = [x_1, \dots, x_k]^T$, then $A\vec{x} = x_1\vec{a}_1 + \dots + x_k\vec{a}_k$.

Conditions: $A$ is an $n \times k$ matrix; $\vec{x} \in \mathbb{R}^k$; $C(A)$ is the span of the columns of $A$

The equation $A\vec{x}=\vec{b}$ has no solution because solving it is equivalent to finding weights $x_1, \dots, x_k$ such that $x_1\vec{a}_1 + \dots + x_k\vec{a}_k = \vec{b}$. The column space $C(A)$ is defined as the set of all possible linear combinations of the columns of $A$.

Conditions: $A$ is an $n \times k$ matrix; $\vec{x} \in \mathbb{R}^k$ and $\vec{b} \in \mathbb{R}^n$; $C(A)$ denotes the column space of $A$

Minimizing the Euclidean norm $\|\vec{b} - A\vec{x}^*\|$ is equivalent to minimizing its square, $\|\vec{b} - A\vec{x}^*\|^2$. The squared norm expands algebraically into the sum of the squared differences of corresponding components: $(b_1 - v_1)^2 + (b_2 - v_2)^2 + \cdots + (b_n - v_n)^2$, where $\vec{v} = A\vec{x}^*$.

Conditions: $\vec{b}, \vec{v} \in \mathbb{R}^n$; Standard Euclidean norm is used; $\vec{v} = A\vec{x}^*$

When $A\vec{x}=\vec{b}$ has no exact solution, the least-squares solution $\vec{x}^*$ is defined as the vector that minimizes the Euclidean norm of the residual, $\|\vec{b} - A\vec{x}^*\|$. Geometrically, this means choosing $\vec{x}^*$ such that $A\vec{x}^*$ is the closest possible vector to $\vec{b}$ within the column space $C(A)$.

Conditions: The system $A\vec{x}=\vec{b}$ is inconsistent (no exact solution exists); Distance is measured by the standard Euclidean norm