Communications for Statistical Applications and Methods

The Korean Statistical Society (한국통계학회)
권호 표지
DOI QR코드

DOI QR Code

Robustness of model averaging methods for the violation of standard linear regression assumptions

  • Lee, Yongsu (Department of Statistics, University of Wisconsin-Madison) ;
  • Song, Juwon (Department of Statistics, Korea University)
  • Received : 2020.11.19
  • Accepted : 2021.02.12
  • Published : 2021.03.31
Download PDF
⟨ Previous Next ⟩

Abstract

In a regression analysis, a single best model is usually selected among several candidate models. However, it is often useful to combine several candidate models to achieve better performance, especially, in the prediction viewpoint. Model combining methods such as stacking and Bayesian model averaging (BMA) have been suggested from the perspective of averaging candidate models. When the candidate models include a true model, it is expected that BMA generally gives better performance than stacking. On the other hand, when candidate models do not include the true model, it is known that stacking outperforms BMA. Since stacking and BMA approaches have different properties, it is difficult to determine which method is more appropriate under other situations. In particular, it is not easy to find research papers that compare stacking and BMA when regression model assumptions are violated. Therefore, in the paper, we compare the performance among model averaging methods as well as a single best model in the linear regression analysis when standard linear regression assumptions are violated. Simulations were conducted to compare model averaging methods with the linear regression when data include outliers and data do not include them. We also compared them when data include errors from a non-normal distribution. The model averaging methods were applied to the water pollution data, which have a strong multicollinearity among variables. Simulation studies showed that the stacking method tends to give better performance than BMA or standard linear regression analysis (including the stepwise selection method) in the sense of risks (see (3.1)) or prediction error (see (3.2)) when typical linear regression assumptions are violated.

Keywords

References

  1. Agresti A (2013). Categorical Data Analysis (3rd ed.), JohnWiley & Sons, NJ.
  2. Ando T and Li KC (2014). A model-averaging approach for high-dimensional regression, Journal of the American Statistical Association, 109, 254-265. https://doi.org/10.1080/01621459.2013.838168
  3. Ando T and Li KC (2017). A weight-relaxed model averaging approach for high-dimensional generalized linear models, The Annals of Statistics, 45, 2654-2679.
  4. Breiman L (1996a). Bagging predictors, Machine Learning, 24, 123-140. https://doi.org/10.1007/BF00058655
  5. Breiman L (1996b). Stacked regressions, Machine Learning, 24, 49-64. https://doi.org/10.1007/BF00117832
  6. Breiman L (2001). Random forests, Machine Learning, 45, 5-32. https://doi.org/10.1023/A:1010933404324
  7. Chatterjee S and Hadi AS (2015). Regression Analysis by Example, John Wiley & Sons, New York.
  8. Clarke B (2003). Comparing bayes model averaging and stacking when model approximation error cannot be ignored, Journal of Machine Learning Research, 4, 683-712.
  9. Eklund J and Karlsson S (2007). Forecast combination and model averaging using predictive meansures, Econometric Reviews, 26, 329-363. https://doi.org/10.1080/07474930701220550
  10. Fernandez C, Ley E, and Steel MF (2001). Benchmark priors for bayesian model averaging, Journal of Econometrics, 100, 381-427. https://doi.org/10.1016/S0304-4076(00)00076-2
  11. Friedman J, Hastie T, and Tibshirani R (2001). The Elements of Statistical Learning(Vol. 1), Springer, New York.
  12. Furnival GM and Wilson RW (1974). Regressions by leaps and bounds, Technometrics, 16, 499-511. https://doi.org/10.1080/00401706.1974.10489231
  13. Haith DA (1976). Land use and water quality in new york rivers, Journal of the Environmental Engineering Division (EEL Proc. Paper 11902), 1-15.
  14. Hjort NL and Claeskens G (2003). Frequentist model average estimators, Journal of the American Statistical Association, 98, 879-899. https://doi.org/10.1198/016214503000000828
  15. Hoeting JA, Madigan D, Raftery AE, and Volinsky CT (1999). Bayesian model averaging: a tutorial, Statistical Science, 382-401.
  16. Kass RE and Raftery AE (1995). Bayes factors, Journal of the American Statistical Association, 90, 773-795. https://doi.org/10.1080/01621459.1995.10476572
  17. Liang H, Zou G, Wan AT, and Zhang X (2011). Optimal weight choice for frequentist model average estimators, Journal of the American Statistical Association, 106, 1053-1066. https://doi.org/10.1198/jasa.2011.tm09478
  18. Madigan D and Raftery AE (1994). Model selection and accounting for model uncertainty in graphical models using occam's window, Journal of the American Statistical Association, 89, 1535-1546. https://doi.org/10.1080/01621459.1994.10476894
  19. Montgomery DC, Peck EA, and Vining GG (2012). Introduction to Linear Regression Analysis (5th ed, Vol. 821), John Wiley & Sons, New York.
  20. Raftery AE (1995). Bayesian model selection in social research, Sociological Methodology (Vol. 25), 111-163. https://doi.org/10.2307/271063
  21. Schapire RE (2003). The boosting approach to machine learning: An overview. In Nonlinear Estimation and Classification (pp. 149-171), Springer.
  22. Wolpert DH (1992). Stacked generalization, Neural Networks, 5, 241-259. https://doi.org/10.1016/S0893-6080(05)80023-1
  23. Zhang X, Zou G, Liang H, and Carroll RJ (2020). Parsimonious model averaging with a diverging number of parameters, Journal of the American Statistical Association, 115, 972-984. https://doi.org/10.1080/01621459.2019.1604363