A COMPANION TO Theoretical Econometrics

Diagnosing collinearity using the singular value decomposition

The singular-value decomposition is a factorization of X. The matrix X may be decomposed as X = UA1/2C', where U'U = C' C = CC = IK and A1/2 is a diagonal matrix with nonnegative diagonal values X1/2, XY2,..., X1/2, called the singular values of X. The relation to eigenanalysis is that the singular values are the positive square roots of the eigenvalues of X'X, and the K x K matrix C is the matrix whose columns contain the eigenvectors of X X. Thus

C'X'XC = Л, (12.5)

where Л is a diagonal matrix with the real values X1, X2,..., XK on the diagonal. The matrix U is T x K, and its properties are discussed in Belsley (1991, pp. 42-3). The columns of the matrix C, denoted c;, are the eigenvectors (or characteristic vectors) of the matrix X X, and the real values Xi are the corresponding eigenvalues (or characteristic roots). It is customary to assume that the columns of C are arranged so that the eigenvalues are ordered by magnitude, X1 > X2 > ... > XK.

If X is of full column rank K, so that there are no exact linear dependencies among the columns of X, then X'X is a positive definite and symmetric matrix, and all its eigenvalues are not only real but also positive. If we find a "small" eigenvalue, Xj ~ 0, then c'X'Xq = (Xc;)'(Xc;) = X, ~ 0 and therefore Xc, ~ 0. Thus we have located a near exact linear dependency among the columns of X. If there is a single small eigenvalue, then the linear relation Xc, ~ 0 indicates the form of the linear dependency, and can be used to determine which of the explanatory variables are involved in the relationship.

1.1 Collinearity and the least squares estimator

Using equation (12.5), and the orthogonality of C, C'C = CC = IK, we can write X'X = ЄЛО, and therefore

K

(X'X)-1 = ^-1C' = £ X-1qc'i. (12.6)

Z=1

The covariance matrix of the least squares estimator b is cov(b) = o2(X'X) a, and using equation (12.6) the variance of bj is

where cjk is the element in the jth row and ith column of C. The orthogonality of C implies that Xk=1 c2k = 1. Thus the variance of bj depends upon three distinct factors. First, the magnitude of the error variance, a2; second, the magnitudes
of the constants Cjk; and third, the magnitude of the eigenvalues, Xk. A small eigenvalue may cause a large variance for bj if it is paired with a constant cjk that is not close to zero. The constants Cjk = 0 when Xj and xk, the jth and kth columns of X, respectively, are orthogonal, so that x'jXk = 0. This fact is an important one for it will allow us to determine which variables are "not" involved in collinear relationships.

Suppose Pj is a critical parameter in the model, and there is one small eigenvalue, XK ~ 0. If Xj is not involved in the corresponding linear dependency XcK ~ 0, then CjK will be small, and the fact that XK ~ 0, i. e. the Kth eigenvalue is very small, will not adversely affect the precision of estimation of Pj. The presence of collinearity in the data does not automatically mean that “all is lost." If X'X has one or more small eigenvalues, then you must think clearly about the objectives of your research, and determine if the collinearity reduces the precision of estimation of your key parameters by an unacceptable amount. We address the question What is a small eigenvalue? in Section 3. For more about the geometry of characteristic roots and vectors see Fomby, Hill, and Johnson (1984, pp. 288-93).