A COMPANION TO Theoretical Econometrics
Model Selection Versus Hypothesis Testing
Hypothesis testing and model selection are different strands in the model evaluation literature. However, these strands differ in a number of important respects which are worth emphasizing here.10 Model selection begins with a given set of models, M, characterized by the (possibly) conditional pdfs
M = {f(Y|X;, ¥;), i = 1, 2,..., m},
with the aim of choosing one of the models under consideration for a particular purpose with a specific loss (utility) function in mind. In essence model selection is a part of decision making and as argued in Granger and Pesaran (2000) ideally it should be fully integrated into the decision making process. However, most of the current literature on model selection builds on statistical measure of fit such as sums of squares of residuals or more generally maximized loglikelihood values, rather than economic value which one would expect to follow from a model choice.11 As a result model selection seems much closer to hypothesis testing than it actually is in principle.
The model selection process treats all models under consideration symmetrically, while hypothesis testing attributes a different status to the null and to the alternative hypotheses and by design treats the models asymmetrically. Model selection always ends in a definite outcome, namely one of the models under consideration is selected for use in decision making. Hypothesis testing on the other hand asks whether there is any statistically significant evidence (in the Neyman-Pearson sense) of departure from the null hypothesis in the direction of one or more alternative hypotheses. Rejection of the null hypothesis does not necessarily imply acceptance of any one of the alternative hypotheses; it only warns the investigator of possible shortcomings of the null that is being advocated. Hypothesis testing does not seek a definite outcome and if carried out with due care need not lead to a favorite model. For example, in the case of nonnested hypothesis testing it is possible for all models under consideration to be rejected, or all models to be deemed as observationally equivalent.
Due to its asymmetric treatment of the available models, the choice of the null hypothesis plays a critical role in the hypothesis testing approach. When the models are nested the most parsimonious model can be used as the null hypothesis. But in the case of nonnested models (particularly when the models are globally nonnested) there is no natural null, and it is important that the null hypothesis is selected on a priori grounds.12 Alternatively, the analysis could be carried out with different models in the set treated as the null. Therefore, the results of nonnested hypothesis testing is less clear cut as compared to the case where the models are nested.13
It is also important to emphasize the distinction between paired and joint nonnested hypothesis tests. Letting/1 denote the null model and f Є M, i = 2,..., m index a set of m - 1 alternative models, a paired test is a test of f against a single member of M, whereas a joint test is a test of f1 against multiple alternatives in M McAleer (1995) is careful to highlight this distinction and in doing so points out a deficiency in many applied studies insofar as many authors have utilized a sequence of paired tests for problems characterized by multiple alternatives. Examples of studies which have applied nonnested tests to the choice between more than two models include Sawyer (1984), Smith and Maddala (1983) and Davidson and MacKinnon (1981). The paper by Sawyer is particularly relevant since he develops the multiple model equivalent of the Cox test.
The distinction between model selection and nonnested hypothesis tests can also be motivated from the perspective of Bayesian versus sampling-theory approaches to the problem of inference. For example, it is likely that with a large amount of data the posterior probabilities associated with a particular hypothesis will be close to one. However, the distinction drawn by Zellner (1971) between "comparing" and "testing" hypothesis is relevant given that within a Bayesian perspective the progression from a set of prior to posterior probabilities on M, mediated by the Bayes factor, does not necessarily involve a decision to accept or reject the hypothesis. If a decision is required it is generally based upon minimizing a particular expected loss function. Thus, model selection motivated by a decision problem is much more readily reconcilable with the Bayesian rather than the classical approach to model selection.
Finally, the choice between hypothesis testing and model selection clearly depends on the primary objective of the exercise. There are no definite rules. Model selection is more appropriate when the objective is decision making. Hypothesis testing is better suited to inferential problems where the empirical validity of a theoretical prediction is the primary objective. A model may be empirically adequate for a particular purpose but of little relevance for another use. Only in the unlikely event that the true model is known or knowable will the selected model be universally applicable. In the real world where the truth is elusive and unknowable both approaches to model evaluation are worth pursuing.
