A COMPANION TO Theoretical Econometrics

Monte Carlo tests in the presence of nuisance parameters

In Dufour (1995), we discuss extensions of MC tests when nuisance parameters are present. We now briefly outline the underlying methodology. In this section, n refers to the sample size and N the number of MC replications.

Consider a test statistic S for a null hypothesis H0, and suppose the null distribu­tion of S depends on an unknown parameter vector 0. [16]

• Using 0°, generate N simulated samples and, from them, N simulated values of the test statistic. Then compute pN(S01 0on), where pN(x | 0) refers to pN (x) based on realizations of S generated given 0 = 0 and pN(x) is defined in (23.31).

• An MC test may be based on the critical region

Pn(S0 10П) < a, a < 0 < 1.

For further reference, we denote the latter procedure a local Monte Carlo (LMC) test. Under general conditions, this LMC test has the correct level asymptotically (as n ^ ro), i. e. under H0,

lim{P[Pn(S0 | 0П) < a] - P[Pn(S0 | 0) < a]} = 0. (23.35)

In particular, these conditions are usually met whenever the test criterion involved is asymptotically pivotal. We emphasize that no asymptotics on the number of replications is required to obtain (23.35).

• To obtain an exact critical region, the MC p-value ought to be maximized with respect to the intervening parameters. Specifically, in Dufour (1995), it is shown that the test (henceforth called a maximized Monte Carlo (MMC) test) based on the critical region

sup [Pn(S0 | 0)] < a (23.36)

Є ЄМ0

where M0 is the subset of the parameter space compatible with the null hypo­thesis (i. e. the nuisance parameter space) is exact at level a.

The LMC test procedure is closely related to a parametric bootstrap, with how­ever a fundamental difference. Whereas bootstrap tests are valid as N ^ <*>, the number of simulated samples used in MC tests is explicitly taken into account. Further the LMC p-value may be viewed as exact in a liberal sense, i. e. if the LMC fails to reject, we can be sure that the exact test involving the maximum p-value is not significant at level a.

In practical applications of exact MMC tests, a global optimization procedure is needed to obtain the maximal randomized p-value in (23.36). We use the simu­lated annealing (SA) algorithm (Corana, Marchesi, Martini, and Ridella, 1987; Goffe, Ferrier, and Rogers, 1994). SA starts from an initial point (it is natural to use 0n here) and sweeps the parameter space (user defined) at random. An uphill step is always accepted while a downhill step may be accepted; the decision is made using the Metropolis criterion. The direction of all moves is determined by probabilistic criteria. As it progresses, SA constantly adjusts the step length so that downhill moves are less and less likely to be accepted. In this manner, the algorithm escapes local optima and gradually converges towards the most probable area for optimizing. SA is robust with respect to nonquadratic and even noncontinuous surfaces and typically escapes local optima. The procedure is known not to depend on starting values. Most importantly, SA readily handles problems involving a fairly large number of parameters.7

To conclude this section, we consider another application of MC tests which is useful in the context of boundedly pivotal statistics. Using the above notation, the statistic at hand S is boundedly pivotal if it is possible to find another statistic S* such that

S < S*, V0 Є ©о, (23.37)

and S* is pivotal under the null. Let c and c* refer to the a size-correct cut-off points associated with S and S*. As emphasized earlier, inequality (23.37) entails that c* may be used to define a critical region for S. The resulting test will have the correct level and may be viewed as conservative in the following sense: if the test based on c* is significant, we can be sure that the exact test involving the (unknown!) c is significant at level a. The main point here is that it is easier to calculate c*, because S* is pivotal, whereas S is nuisance-parameter dependent. Of course, this presumes that the null exact distribution of S* is known and tractable; see Dufour (1989, 1990) for the underlying theory and several illustra­tive examples. Here we argue that the MC test technique may be used to produce simulation-based conservative p-values based on S* even if the analytic null dis­tribution of S* is unknown or complicated (but may be simulated). The procedure involved is the same as above, except that the S* rather than S is evaluated from the simulated samples. We denote the latter procedure a bound MC (BMC) test.

A sound test strategy would be to perform the bounds tests first and, on failure to reject, to apply randomized tests. We recommend the following computationally attractive exact a test procedure:

1. compute the test statistic from the data;

2. if a bounding criterion is available, compute a BMC p-value; reject the null if: BMC p-value < a;

3. if the observed value of the test statistic falls in the BMC acceptance region, obtain a LMC p-value; declare the test not significant if: LMC p-value > a;

4. if the LMC p-value < a < BMC p-value, obtain the MMC p-value and reject the null if the latter is less than or equal to a.