Advanced Econometrics Takeshi Amemiya

Laws of Large Numbers and Central Limit Theorems

Given a sequence of random variables {X,}, define Xn = и-1 2JL, X,.A law of large numbers (LLN) specifies the conditions under which X„ — EX„ con­verges to 0 either almost surely or in probability. The former is called a strong law of large numbers and the hitter a weak law. In manyapplications the simplest way to show X„ — EX„ 0 is to show X„ — EX „ 0 and then

apply Theorem 3.2.1 (Chebyshev). We shall state two strong laws of large numbers. For proofs, the reader is referred to Rao (1973, pp. 114-115).

T heorem 3.3.1 (Kolmogorov LLN 1). Let {X,) beindependent with finite variance VX, = of. If27-1 of/*2 < then X„ — EX„^ 0.

Theorem 3.3.2 (Kolmogorov LLN 2). Let {Xt) be i. i.d. (independent and identically distributed). Then a necessary and sufficient condition that X„ ^ H is that EX, exists and is equal to ц.

Theorem 3.3.2 and the result obtained by putting X„ — EXn into the Xn of Theorem 3.2.1 are complementary. If we were to use Theorem 3.2.1 to prove X„ — EX„ 0, we would need the finite variance ofX,, which Theorem 3.3.2

does not require; but Theorem 3.3.2 requires {X,} to be i. i.d., which Theorem 3.2.1 does not.10 _ _

When all we want is the proof of X„ — EX„ 0, Theorem 3.3.1 does not

add anything to Theorem 3.2.1 because the assumptions of Theorem 3.3.1 imply X „ — EX „ 0. The proof is left as an exercise. _

Now we ask the question, What is an approximate distribution ofX„ when n is large? Suppose a law of large numbers holds for a sequence {X,} so that Xn — EX„ 0. It follows from Theorem 3.2.2 that X„ — EXn 0. How­ever, it is an uninteresting limit distribution because it is degenerate. It is more meaningful to inquire into the limit distribution of Z„ = (VX„)~i/2 (Xn — EX n). For, if the limit distribution of Z„ exists, it should be nondegen­erate because VZn = 1 for all n. A central limit theorem (CLT) specifies the conditions under which Z„ converges in distribution to a standard normal random variable [we shall write Z„ —»N(0, 1)].

We shall state three central limit theorems—Lindeberg-Levy, Liapounov, and Lindeberg-Feller—and shall prove only the first. For proofs of the other two, see Chung (1974) or Gnedenko and Kolmogorov (1954). Lindeberg-

Levy and Liapounov are special cases of Lindeberg-Feller, in the sense that the assumptions of either one of the first two central limit theorems imply those of Lindeberg-Feller. The assumptions of Lindeberg-L6vy are more restrictive in some respects and less restrictive in other respects than those of Liapounov.

Before we state the central limit theorems, however, we shall define the characteristic function of a random variable and study its properties.

Definition 3.3.1. The characteristic function of a random variable X is defined by Ееax.

Thus if the distribution function of X is F( •), the characteristic function is eiijc dF(x). The characteristic function is generally a complex number. However, because eiXx = cos Ax + і sin Ax, the characteristic function of a random variable with a density function symmetric around 0 is real. The characteristic functon of N(0, 1) can be evaluated as

(3.3.1)

Define g(X) = Log /“«, e** dF(x), where Log denotes the principal loga­rithm.11 Then the following Taylor expansion is valid provided EX' exists:

(3.3.2)

where Kj = (d^/dA/)A_0/iv. The coefficients Kj are called the cumulants of X. The first four cumulants are given by к, = EX, k2 = VX, k3 = E(X— EX)3, and ka = E(X - EX)4 - 3( VX)2.

The following theorem is essential for proving central limit theorems.

Theorem 3.3.3. If E exp (iXX„) —*E exp (iAX) for every A and if E ехр(г'АДГ) is continuous at A = 0, then Xn X.

The proof of Theorem 3.3.3 can be found in Rao (1973, p. 119).

We can now prove the following theorem.

Theorem 3.3.4 (Lindeberg-Levy CLT). Let {X,) be i. i.d. with EX, = p and VX, = o2. Then Z„ -» N(0, 1).

Proof We assume p = 0 without loss of generality, for we can consider {X, — p) if// Ф 0. Define g(A) = Log E exp (iXX,). Then from (3.3.2) we have

We have, using (3.3.3),

n Г ,'IV "I

LogEexp(ikZn) = 2 LogEexp(3.3.4)

Therefore the theorem follows from (3.3.1) and Theorem 3.3.3.

Theorem 3.3.5 (Liapounov CLT). Let (Xt) be independent with EX, = p„ VX, — of, and E[X, — p,3] = m3t. If

л "1-1/2 Г n І1/3

s**] Is4 - o'

then Z„ —* N(0, 1).

Theorem 3.3.6 (Lindeberg-Feller CLT). Let {X,} be independent with distribution functions {F,} and EX, = p, and VX, = of. Define Cn —

W-rfY'2- If

for every e > 0, then Z„ —» N{0, 1).

In the terminology of Definition 3.2.3, central limit theoremsprovidecon - ditions under which the limit distribution of Zn — (VXn)~l/2(Xn — EX„) is N(0, 1). We now introduce the term asymptotic distribution. It simply means the “approximate distribution when n is large.” Given the mathematical result Z„ N(0, 1), we shall make statements such as “the asymptotic distri­

bution of Z„ is iV(0, 1)” (written asZ„ ~ N(0, 1)) or “the asymptotic distribu­tion of X„ is N(EXn, VX„).” These statements should be regarded merely as more intuitive paraphrases of the result Z„ N{0, 1). Note that it would be

meaningless to say “the limit distribution of Xn is N(EXn, VXn)."

When the asymptotic distribution of X„ is normal, we also say that Xn is asymptotically normal.

The following theorem shows the accuracy of a normal approximation to the true distribution (see Bhattacharya and Rao, 1976, p. 110).

Theorem 3.3.7. Let {X,} be i. i.d. with EX, = p, VX, = c2, and ЕХ,Ъ = m3. Let Fn be the distribution function of Z„ and let Ф be that of N(0, 1). Then

|ад-Ф(х)|^ (0.7975)^-^

for all x.

It is possible to approximate the distribution function F„ of Z„ more accu­rately by expanding the characteristic function of Z„ in the powers of n~1/2. If EX* exists, we can obtain from (3.3.2)

. x{ikf 12a6n

This is called the Edgeworth expansion (see Cramer, 1946, p. 228). Because (see Cramer, 1946, p. 106)

J e0* ф(г)(х) dx={ — iX)r e~m, (3.3.6)

where 0(r)(x) is the rth derivative of the density ofN{0, 1), we can invert (3.3.5) to obtain

Fn(x) - Ф(х) - -^= Ф<3>(х) -I - 2^ Ф(4Кх) (3.3.7)

+ 7^Ф(6)(х) + 0(й-3/2).

We shall conclude this section by stating a multivariate central limit theorem, the proof of which can be found in Rao (1973, p. 128).

Theorem 3.3.8. Let (XB) be a sequence of A-dimensional vectors of ran­dom variables. If c'X„ converges to a normal random variable for every AT-di - mensional constant vector с Ф 0, then X„ converges to a multivariate normal random variable. (Note that showing convergence of each element of X„ separately is not sufficient.)