What is the orthogonality principle in terms of projections?

The orthogonality principle in terms of projections relates to the projection of a vector onto a subspace, specifically within the context of linear regression and least squares problems. The principle states that the error vector, representing the difference between the actual data points and their projections onto the subspace, is orthogonal to (i.e., perpendicular to) the subspace itself. This orthogonality is what ensures that the projection minimizes the error, or the squared distance, between the actual data points and their estimated values.

Let’s consider a vector y ∈ ℝⁿ and a subspace S within ℝⁿ, which can be spanned by vectors {x₁, x₂, ..., x_k}. The orthogonal projection of y onto the subspace S is defined as the vector ŷ, which lies in S.

The error vector e equals the difference between the original vector y and its projection ŷ:

e = y − ŷ

According to the orthogonality principle, the error vector e is orthogonal to the subspace S and satisfies the following conditions:

e^Tx_i = 0  for  i = 1, 2, ..., k

Consider a linear regression model where we want to estimate the unknown parameters β using the data matrix X and the response vector y. We can write the linear regression model as:

y = Xβ + ϵ

Here, y is an n × 1 vector of response values, X is an n × p matrix of features, β is a p × 1 vector of parameters, and ϵ is an n × 1 vector of errors.

To solve for the unknown parameter vector β, we want to find the projection of y onto the column space of X (i.e., the subspace S spanned by the columns of X). According to the orthogonality principle, the error vector ϵ is orthogonal to the column space of X:

ϵ^TX = 0

The unknown parameter vector can be found by solving the following normal equations:

X^T(y − Xβ) = 0

 ⇒ X^Ty = X^TXβ

 ⇒ β = (X^TX)^− 1X^Ty

In conclusion, the orthogonality principle in terms of projections helps us find the best approximation or projection of a vector onto a subspace by ensuring that the error vector is orthogonal to that subspace. In the context of linear regression and least squares problems, this ensures that we can find the best possible estimated values for the model parameters.