A COMPANION TO Theoretical Econometrics
Criteria for Choosing Concomitants in RCMs
Equations (19.9)-(19.12) incorporate in a consistent way all the prior information that is usually available about these equations. The most difficult step arises in the form of equation (19.13). Not much prior information is available about the proper concomitants that satisfy Assumptions 1 and 2. As a minimum exercise of caution, the applied econometrician who approaches the problem of estimating equation (19.15) should choose among various sets of concomitants after carefully examining their implications for the estimates of the direct effect components of the coefficients of equation (19.12). Different models of the form (19.15) are obtained by including different sets of concomitants in equation (19.13). The question we address in this section is the following: how can we validate these different models? In what follows, we briefly describe a set of validation criteria and relate them to the RCM described above.
A money demand model can be considered to be validated if (i) it fits within - sample values well; (ii) it fits out-of-sample values well; (iii) it has high explanatory power; (iv) it is derived from equation (19.12) by making assumptions that are consistent with the real-world interpretations of the coefficients of equation (19.12); (v) the signs and statistical significance of the estimates of direct effects remain virtually unchanged as one set of concomitants (other than the determinants of direct effects) after another is introduced into equation (19.13).
Condition (i) is used in almost all econometric work as a measure of fitted- model adequacy. The definition of the coefficient of determination (R1) that is appropriate to a regression equation with nonspherical disturbances can be applied to equation (19.15). Such a definition is given in Judge et al. (1985, p. 31). The coefficient is a measure of the proportion of weighted variation in mt, t = 1,
1,. .., T, explained by the estimated equation (19.15). Within the RCM framework, a low R1 implies that the set of concomitants included in equation (19.13) together with the explanatory variables of equation (19.11) do not adequately explain the weighted variation in mt, t = 1, 1,..., T. The problem with R1, however, is that a high value can result by arbitrarily increasing the number of concomitants in equation (19.13), even if all these concomitants are not relevant for explaining the coefficients of equation (19.11).
Condition (ii) is based on cross validation, which is used to assess the ability of equation (19.15) to predict out-of-sample values of mt. In this procedure, the data sample is divided into two subsamples. The choice of a model with a set of concomitants, including any necessary estimation, is based on one subsample and then its performance is assessed by measuring its prediction against the other subsample. This method is related to Stone's (1974) cross-validatory choice of statistical predictions.
The premise of this approach is that the validity of statistical estimates should be judged by data different from those used to derive the estimates (Mosteller and Tukey, 1977, pp. 36-40; Friedman and Schwartz, 1991, p. 47). Underlying this approach is the view that formal hypothesis tests of a model on the data that are used to choose its numerical coefficients are almost certain to overestimate performance. Also, statistical tests lead to false models with probability 1 if both the null and alternative hypotheses considered for these tests are false, as we have already shown in Section 1. This problem can arise in the present case because of the lack of any guarantee that either a null or an alternative hypothesis will be true if inconsistent restrictions are imposed on equation (19.9).
Predictive testing - extrapolation to data outside the sample - also has its limitations. All forecasts and forecast comparisons should take into account the result, due to Oakes (1985), that there is no universal algorithm to guarantee accurate forecasts forever. This result implies that equation (19.15) with a single set of concomitants cannot predict mt well in all future periods. This is especially true when the set of x* s in equation (19.9) changes over time. Also, past success does not guarantee future success. That is, if all we knew about equation (19.15) was that it had produced accurate forecasts in the past, there would be no way we could guarantee that future forecasts of equation (19.15) would be sufficiently accurate, since there are some sets of concomitants (e. g. dummy (or shift) variables that are appropriate for a past period) for which past values do not control future values. Even false models based on contradictory premises can sometimes predict their respective dependent variables well. To satisfy a necessary condition under which models are true, de Finetti (1974b) sets up minimal coherence criteria that forecasts should satisfy based on data currently available. By these criteria, different forecasts are equally valid now if they all satisfy the requirements for coherence, given currently available knowledge. Thus, a forecast from equation (19.15) can at best represent a measure of the confidence with which one expects that equation to predict an event in the future, based on currently available evidence and not on information yet to be observed, provided that the forecast satisfies the requirements for coherence.8
To choose models that satisfy de Finetti's criteria of coherence, we impose the additional conditions (iii)-(v) on RCMs. As with de Finetti's concept of coherence, condition (iv) also explicitly prohibits the use of contradictory premises.9 Together, conditions (i)-(v) provide an improved method of model validation.
Condition (iii) has also been advocated by Zellner (1988). If prediction were the only criterion of interest, there would be no need to separate direct effects from indirect and mismeasurement effects. But if we are interested in economic explanations - for example, a transmission mechanism of a particular policy action - we need to separate these effects. Equation (19.9) will have the highest explanatory power whenever it coincides with the true money demand function. It would be fortunate if Zn=з ajt^0jt were offset exactly by v0t and if ZjL3 ajt^1jt (or Zn= 3 ajWyt) and -(a 1t + з a^t) т (or - (a* + ZnjLз a^-t) y canceled each
other. In this case, y0t = a0t, y1t = a1t, y2t = a2t, and equation (19.12) has the same explanatory power as equation (19.9). Alternatively, when y1t Ф a1t and y2t Ф a2t, equation (19.12) explains well if it is closer to the true money demand function than to any other equation and cannot provide the proper explanations otherwise. To discern which one of these cases is true given the data on mt, rt, and yt, an accurate means for separating a1t and a2t from the other terms of y1t and y2t is needed. Equation (19.15) attempts to make such a separation.
Condition (iv) conforms to de Finetti's (1974b) requirement of coherence - namely, that statistical analysis applied to data should not violate probability laws. We apply this requirement in a somewhat different manner. To explain, consider the following example. Equation (19.12) represents a particular economic (i. e. money demand) relationship that we have in mind but cannot estimate. What we can estimate is equation (19.15). Underlying equation (19.15) are equation (19.12) and Assumptions 1 and 2. Thus, estimation requires that Assumptions 1 and 2 are consistent with the real-world interpretations of the coefficients of equation (19.12) so that de Finetti's condition is satisfied. Assumptions 1 and 2 are consistent with the real-world interpretations of the coefficients of equation (19.12) if the concomitants included in equation (19.1з) satisfy Assumptions 1 and 2. For example, Assumptions 1 and 2 are satisfied if the correlations between (y0t, y1t, y2t) and (rt, yt) arise because of their dependence on a common third set of variables, Zjt, j = 1,..., p, and if these z s together with ek/t-1, k = 0, 1, 2, capture all the variation in y1t and y2t and almost all the variation in y0t. An example of forecasts that do not satisfy de Finetti's requirement of coherence is a forecast of mt from equation (19.8), since the premises of this equation are inconsistent with the real-world interpretations of the coefficients of equation (19.12).
Condition (v) concerns the sensitivity of the signs and magnitudes of direct effects to changes in the set of concomitants. Following Pratt and Schlaifer (1988, p. 45), we state that the only convincing evidence that equation (19.12) under Assumptions 1 and 2 coincides with the true money demand function is of the following kind. It is found that the signs and statistical significance of the estimates of a 1t and a 2t remain virtually unchanged as one set of concomitants (other than those determining a1t and a 2t) after another is introduced into equation (19.13), until finally it is easier to believe that r* and y* have the effects they seem to have on m* than to believe that they are merely the proxies for some other, as yet undiscovered, variable or variables.
