Introduction to the Mathematical and Statistical Foundations of Econometrics
A.2. A Hilbert Space of Random Variables
Let U0 be the vector space of zero-mean random variables with finite second moments defined on a common probability space {&, .^, P} endowed with the innerproduct (X, Y) = E[X ■ Y],norm ||X|| = ^E[X2],andmetric ||X — Y||.
Theorem 7.A.2: The space U0 defined above is a Hilbert space.
Proof: To demonstrate that U0 is a Hilbert space, we need to show that every Cauchy sequence Xn, n > 1, has a limit in U0. Because, by Chebishev’s inequality,
P [|Xn — Xm | > є] < E[(Xn — Xm )2]/є2
= ||X„ — Xm ||2/є2 ^ 0 as n, m ^ то
forevery є > 0, it follows that | Xn — Xmp 0as n, m ^ж. In Appendix 6.B of Chapter 6, we have seen that convergence in probability implies convergence a. s. along a subsequence. Therefore, there exists a subsequence nk such that Xnk — Xnm ^ 0 a. s. as n, m ^ж. The latter implies that there exists a null set N such that for every ш є N, Xnk (ш) is a Cauchy sequence in R; hence,
limk^TOXnt (ш) = X(ш) exists for every ш є ЙN. Now for every fixed m,
(Xnk — Xm)2 ^ (X — Xm)2 a. s. as k ^ж.
By Fatou’s lemma (see Lemma 7.A.1) and the Cauchy property, the latter implies that
IIX — Xm II2 = E [(X — Xm)2]
< liminfE (Xnt — Xm)21 ^ 0 as m ^ж.
k^ж L J
Moreover, it is easy to verify that E[X] = 0 and E[X2] < ж. Thus, every Cauchy sequence in U0 has a limit in U0; hence, U0 is a Hilbert space. Q. E.D.
Lemma 7.A.1: (Fatou’s lemma). Let Xn, n > 1, be a sequence of nonnegative random variables. Then E[liminf „^жXn] < liminf „^жE[Xn].
Proof: Put X = liminf^^,Xn and let ф be a simple function satisfying 0 < q>(x) < x. Moreover, put Yn = min^(X), Xn). Then Yn ^p <p(X) because, for arbitrary є > 0,
P[Yn — v(X) >є] = P[Xn < ф(X) — є] < P[Xn < X — є] ^ 0.
Given that E[<p(X)] < ж because ф is a simple function, and Yn < ф(^), it follows from Yn ^p ф(^) and the dominated convergence theorem that
E[ф(X)] = lim E[Yn] = liminfE[Yn] < liminfE[Xn]. (7.64)
п^ж п^ж п^ж
If we take the supremum over all simple functions ф satisfying 0 < ф(x) < x, it follows now from (7.64) and the definition of E [X] that E [X] < liminfn^ E [Xn ]. Q. E.D.
7.A.3. Projections
As for the Hilbert space Rn, two elements x andy in a Hilbert space H are said to be orthogonal if (x, y) = 0, and orthonormal if, in addition, ||x || = 1 and |y | = 1. Thus, in the Hilbert space U0, two random variables are orthogonal if they are uncorrelated.
Definition 7.A.4: A linear manifold of a real Hilbert space H is a nonempty subset M of H such that for each pair x, y in M and all real numbers a and в, a ■ x + в ■ y e M. The closure M ofM is called a subspace of H. The subspace spanned by a subset C of H is the closure of the intersection of all linear manifolds containing C.
In particular, if S is the subspace spanned by a countable infinite sequence x1, x2, x3,... of vectors in H, then each vector x in S takes the form x = J2T cn ■ xn, where the coefficients cn are such that ||x || < to.
It is not hard to verify that a subspace of a Hilbert space is a Hilbert space itself.
Definition 7.A.5: The projection of an element y in a Hilbert space H on a subspace S of His an element x of S such that || y — x || = minzeS ||y — z||.
For example, if S is a subspace spanned by vectors x1,...,xk in H and y e HS, then the projection ofy on S is a vector x = c1 ■ x1 + ■ ■ ■ + ck ■ xk e S, where the coefficients cj are chosen such that ||y — c1 ■ x1 — ■■■ — ck ■ xk || is minimal. Of course, if y e S, then the projection of y on S is y itself.
Projections always exist and are unique:
Theorem 7.A.3: (Projection theorem) IfS is a subspace of a Hilbert space H andy is a vector in H, then there exists a unique vectorx in Ssuch that ||y — x || = minzeS IIy — z||. Moreover, the residual vector u = y — x is orthogonal to any z in S.
Proof: Let y e HS and infzeS ||y — z|| = 8. By the definition of infimum it is possible to select vectors xn in S such that ||y — xn || <8 + 1/n. The existence of the projection x of y on S then follows by showing that xn is a Cauchy sequence as follows. Observe that
l|x„ — xm II2 = ||(x„ — y) — (x„ — y)II2
= l|x„ — y||2 + ||xm — y||2 — 2(x„ — y, xm — y> and
4||(x„ + xm )/2 — y||2 = ||(x„ — y) + (xm — y )||2
= l|x„ — y ||2 + ||xm — y||2 + 2(x„ — y, xm — y>. Adding these two equations up yields
l|x„ — xm II2 = 2||x„ — y||2 + 2||xffl — y||2 — 4||(x„ + xm)/2 — y||2.
(7.65)
Because (x„ + xm)/2 e S, it follows that ||(x„ + xm)/2 — y |2 > 82; hence, it follows from (7.65) that
| xn — xm | 2 < 2| xn — y| 2 + 2| xm — y| 2 — 482 < 48/n + 1/n2 + 48/m + 1/m2.
Thus, xn is a Cauchy sequence in S, and because S is a Hilbert space itself, xn has a limit x in S.
As to the orthogonality of u = y — x with any vector z in S, note that for every real number c and every z in S, x + c ■ z is a vector in S, and thus
S2 <\y — x — c ■ z\2 = ||u — c ■ z\2
= ІІУ — x ||2 + ||c ■ z||2 — 2{u, c ■ z)
= S2 + c2||z||2 — 2c{u, z). (7.66)
Minimizing the right-hand side of (7.66) to c yields the solution c0 = {u, z)/||z||2, and substituting this solution in (7.66) yields the inequality
({u, z))2/||z||2 < 0. Thus, {u, z) = 0.
Finally, suppose that there exists another vector p in S such that ||y — p|| = S. Then y — p is orthogonal to any vector z in S : {y — p, z) = 0. But x — p is a vector in S, and thus {y — p, x — p) = 0 and {y — x, x — p) = 0; hence, 0 = {y — p, x — p) — {y — x, x — p) = {x — p, x — p) = \x — p||2. Therefore, p = x. Q. E.D.
