Communications for Statistical Applications and Methods

The Korean Statistical Society (한국통계학회)
권호 표지
DOI QR코드

DOI QR Code

A convenient approach for penalty parameter selection in robust lasso regression

  • Kim, Jongyoung (Department of Statistics, Hankuk University of Foreign Studies) ;
  • Lee, Seokho (Department of Statistics, Hankuk University of Foreign Studies)
  • Received : 2017.07.21
  • Accepted : 2017.10.14
  • Published : 2017.11.30
Download PDF
⟨ Previous Next ⟩

Abstract

We propose an alternative procedure to select penalty parameter in $L_1$ penalized robust regression. This procedure is based on marginalization of prior distribution over the penalty parameter. Thus, resulting objective function does not include the penalty parameter due to marginalizing it out. In addition, its estimating algorithm automatically chooses a penalty parameter using the previous estimate of regression coefficients. The proposed approach bypasses cross validation as well as saves computing time. Variable-wise penalization also performs best in prediction and variable selection perspectives. Numerical studies using simulation data demonstrate the performance of our proposals. The proposed methods are applied to Boston housing data. Through simulation study and real data application we demonstrate that our proposals are competitive to or much better than cross-validation in prediction, variable selection, and computing time perspectives.

Keywords

References

  1. Bishop CM (2006). Pattern Recognition and Machine Learning, Springer, New York.
  2. Buntine WL and Weigend AS (1991). Bayesian back-propagation, Complex Systems, 5, 603-643.
  3. Chang L, Roberts S, and Welsh A (2017). Robust lasso regression using Tukey's biweight criterion, Technometrics, from: https://dx.doi.org/10.1080/00401706.2017.1305299
  4. El Ghaoui L and Lebret H (1997). Robust solutions to least-squares problems with uncertain data, SIAM Journal on Matrix Analysis and Applications, 18, 1035-1064. https://doi.org/10.1137/S0895479896298130
  5. Figueiredo MAT (2003). Adaptive sparseness for supervised learning, IEEE Transactions on Pattern Analysis and Machine Intelligence, 25, 1150-1159. https://doi.org/10.1109/TPAMI.2003.1227989
  6. Hastie T, Tibshirani R, and Friedman JH (2001). The Elements of Statistical Learning: Data Mining Inference, and Prediction, Springer, New York.
  7. Koenker R (2005). Quantile Regression, Cambridge University Press, Cambridge.
  8. Lambert-Lacroix S and Zwald L (2011). Robust regression through the Huber's criterion and adaptive lasso penalty, Electronic Journal of Statistics, 5, 1015-1053. https://doi.org/10.1214/11-EJS635
  9. Lee A, Caron F, Doucet A, and Holmes C (2010). A hierarchical Bayesian framework for constructing sparsity-inducing priors (Technical report), University of Oxford, Oxford.
  10. Lee S, Shin H, and Lee SH (2016). Label-noise resistant logistic regression for functional data classification with an application to Alzheimer's disease study, Biometrics, 72, 1325-1335. https://doi.org/10.1111/biom.12504
  11. MacKay DJC (1995). Probable networks and plausible predictions: a review of practical Bayesian methods for supervised neural networks, Network: Computation in Neural Systems, 6, 469-505. https://doi.org/10.1088/0954-898X_6_3_011
  12. Maronna RA, Martin RD, and Yohai VJ (2006). Robust Statistics: Theory and Methods, Wiley, Chichester.
  13. Murphy KP (2012). Machine Learning: A Probabilistic Perspective, The MIT Press, Cambridge.
  14. Owen AB (2006). A robust hybrid of lasso and ridge regression (Technical report), Stanford University, Stanford.
  15. Park T and Casella G (2008). The Bayesian lasso, Journal of the American Statistical Association, 103, 681-686. https://doi.org/10.1198/016214508000000337
  16. Tipping ME (2001). Sparse Bayesian learning and the relevance vector machine, Journal of Machine Learning Research, 1, 211-244.
  17. West M (1987). On scale mixtures of normal distributions, Biometrika, 74, 646-648. https://doi.org/10.1093/biomet/74.3.646
  18. Zou H (2006). The adaptive lasso and its oracle properties, Journal of the American Statistical Association, 101, 1418-1429. https://doi.org/10.1198/016214506000000735
  19. Zou H and Li R (2008). One-step sparse estimates in nonconcave penalized likelihood models, The Annals of Statistics, 36, 1509-1533. https://doi.org/10.1214/009053607000000802