What is the least-squares objective function for simple linear regression?

The least-squares fit $A\vec{x}^*$ is identified with the orthogonal projection of $\vec{b}$ onto $C(A)$ because the goal of least squares is to find the vector in the subspace $C(A)$ that is closest to $\vec{b}$. A fundamental geometric property of subspaces states that the unique closest point in a subspace to an external vector is its orthogonal projection.

Conditions: $C(A)$ is a subspace of $\mathbb{R}^n$; $\vec{b} \in \mathbb{R}^n$; Distance is measured by the Euclidean norm

The computation requires the sums of $x$, $y$, $xy$, and $x^2$.

Conditions: Computing the slope and intercept by hand for a least-squares regression line.

By setting the partial derivative of the least-squares objective with respect to the intercept to zero, we obtain the relationship: intercept = mean(y) − slope · mean(x). This implies that the fitted line always passes through the point of sample means.

Conditions: The model includes an intercept term.; The derivative is taken with respect to the intercept parameter.

The least-squares slope is calculated as the sum of the products of centered x and y values, divided by the sum of squared centered x values. Mathematically, this is $\Sigma[(x_i - \bar{x})(y_i - \bar{y})] / \Sigma(x_i - \bar{x})^2$.

Conditions: The denominator $\Sigma(x_i - \bar{x})^2$ is positive.; The model is simple linear regression with an intercept.

The solution to the normal equations is considered the least-squares solution because it satisfies the necessary and sufficient condition for minimizing the residual norm $\|\vec{b} - A\vec{x}\|$. The derivation shows that minimizing this norm is equivalent to requiring the residual $A\vec{x}^* - \vec{b}$ to be orthogonal to the column space $C(A)$.

Conditions: The original system $A\vec{x}=\vec{b}$ may be inconsistent; $A^T A \vec{x}^* = A^T \vec{b}$ has a solution