Introduction to the Mathematical and Statistical Foundations of Econometrics

Projections, Projection Matrices, and Idempotent Matrices

Consider the following problem: Which point on the line through the origin and point a in Figure I.3 is the closest to point b? The answer is point p in Figure

I.4. The line through b and p is perpendicular to the subspace spanned by a, and therefore the distance between b and any other point in this subspace is larger than the distance between b and p. Point p is called the projection of b on the subspace spanned by a. To find p, let p = c ■ a, where c is a scalar. The distance between b and p is now ||b - c ■ a\; consequently, the problem is to find the scalar c that minimizes this distance. Because ||b - c ■ a\ is minimal if and only if

||b — c ■ a||2 = (b — c ■ a)T(b — c ■ a) = bTb — 2c ■ aTb + c2aTa

is minimal, the answer is c = aTb/aTa; hence, p = (aTb/aTa) ■ a.

Similarly, we can project a vector y in Rn on the subspace of Rn spanned by a basis [x,xk} as follows. Let X be the n x к matrix with columns

x1,...,xk. Any point p in the column space R(X) of X can be written as p = Xb, where b є Rk. Then the squared distance between y and p = Xb is

IIy - XbII2 = (y - Xb)T(y - Xb)

= yTy - bTXTy - yTXb + bTXTXb = yTy - 2bTXTy + bTXTXb, (I.39)

where the last equality follows because yTXb is a scalar (or, equivalently, a 1 x 1 matrix); hence, yTXb = (yTXb)T = bTXTy. Given Xand y, (I.39) is a quadratic function of b. The first-order condition for a minimum of (I.39) is given by

2 XTy + 2 XT Xb = 0,

which has the solution

b = (XT X)-1XTy.

Thus, the vectorp in R(X) closest to y is

p = X(XT X)-1XTy, (I.40)

which is the projection of y on R(X).

Matrices of the type in (I.40) are called projection matrices:

Definition I.11: Let A be an n x k matrix with rankk. Then the n x n matrix P = A( AT A)-1 AT is called a projection matrix: For each vector x in Rn, Px is the projection ofx on the column space of A.

Note that this matrix P is such that PP = A(ATA)-1 ATA(ATA)-1 AT) = A(ATA)-1 AT = P. This is not surprising, though, because p = Px is already in R(A); hence, the point in R(A) closest to p is p itself.

Definition I.12: Ann x n matrix M is called idempotent if MM = M.

Thus, projection matrices are idempotent.