Communications for Statistical Applications and Methods

The Korean Statistical Society (한국통계학회)
권호 표지
DOI QR코드

DOI QR Code

ADMM for least square problems with pairwise-difference penalties for coefficient grouping

  • Received : 2021.11.30
  • Accepted : 2022.01.20
  • Published : 2022.07.31
Download PDF
⟨ Previous Next ⟩

Abstract

In the era of bigdata, scalability is a crucial issue in learning models. Among many others, the Alternating Direction of Multipliers (ADMM, Boyd et al., 2011) algorithm has gained great popularity in solving large-scale problems efficiently. In this article, we propose applying the ADMM algorithm to solve the least square problem penalized by the pairwise-difference penalty, frequently used to identify group structures among coefficients. ADMM algorithm enables us to solve the high-dimensional problem efficiently in a unified fashion and thus allows us to employ several different types of penalty functions such as LASSO, Elastic Net, SCAD, and MCP for the penalized problem. Additionally, the ADMM algorithm naturally extends the algorithm to distributed computation and real-time updates, both desirable when dealing with large amounts of data.

Keywords

Acknowledgement

This work is partially funded by the National Research Foundation of Korea (NRF) grants 2018R1D1A1B07043034 and 2019R1A4A1028134.

References

  1. Andrade, D., Fukumizu, K., and Okajima, Y. (2021) Convex covariate clustering for classification, Pattern Recognition Letters 151, 193-199. https://doi.org/10.1016/j.patrec.2021.08.012
  2. Bondell, H.D. and Reich, B.J. (2008) Simultaneous regression shrinkage, variable selection, and su- pervised clustering of predictors with oscar, Biometrics 64(1), 115-123. https://doi.org/10.1111/j.1541-0420.2007.00843.x
  3. Boyd, S., Boyd, S.P. and Vandenberghe, L. (2004) Convex optimization, Cambridge university press.
  4. Boyd, S., Parikh, N. and Chu, E. (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers, Now Publishers Inc.
  5. Chi, E.C. and Lange, K. (2015) Splitting methods for convex clustering, Journal of Computational and Graphical Statistics 24(4), 994-1013. https://doi.org/10.1080/10618600.2014.948181
  6. Fan, J. and Li, R. (2001) Variable selection via nonconcave penalized likelihood and its oracle prop- erties, Journal of the American statistical Association 96(456), 1348-1360. https://doi.org/10.1198/016214501753382273
  7. Harchaoui, Z. and Levy-Leduc, C. (2010) Multiple change-point estimation with a total variation penalty, Journal of the American Statistical Association 105 (492), 1480-1493. https://doi.org/10.1198/jasa.2010.tm09181
  8. Jeon, J.-J., Kwon, S., and Choi, H. (2017) Homogeneity detection for the high-dimensional general- ized linear model, Computational Statistics & Data Analysis 114, 61-74. https://doi.org/10.1016/j.csda.2017.04.001
  9. Ke, Z.T., Fan, J. and Wu, Y. (2015) Homogeneity pursuit, Journal of the American Statistical Associ- ation 110(509), 175-194. https://doi.org/10.1080/01621459.2014.892882
  10. Parikh, N. and Boyd, S. (2014) Proximal algorithms, Foundations and Trends in optimization 1(3), 127-239. https://doi.org/10.1561/2400000003
  11. Shen, X. and Huang, H.-C. (2010) Grouping pursuit through a regularization solution surface, Journal of the American Statistical Association 105(490), 727-739. https://doi.org/10.1198/jasa.2010.tm09380
  12. Tibshirani, R. (1996) Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society: Series B 58(1), 267-288.
  13. Tibshirani, R., Saunders, M., Rosset, S., Zhu, J. and Knight, K. (2005) Sparsity and smoothness via the fused lasso, Journal of the Royal Statistical Society: Series B (Statistical Methodology) 67(1), 91-108. https://doi.org/10.1111/j.1467-9868.2005.00490.x
  14. Wahlberg, B., Boyd, S., Annergren, M., and Wang, Y. (2012) An ADMM algorithm for a class of total variation regularized estimation problems, IFAC Proceedings Volumes 45(16), 83-88.
  15. Zhang, C.-H. (2010) Nearly unbiased variable selection under minimax concave penalty, The Annals of statistics, 38(2), 894-942.
  16. Zhu, Y. (2017) An augmented ADMM algorithm with application to the generalized lasso problem, Journal of Computational and Graphical Statistics 26(1), 195-204. https://doi.org/10.1080/10618600.2015.1114491
  17. Zou, H. and Hastie, T. (2005) Regularization and variable selection via the elastic net, Journal of the royal statistical society: series B 67(2), 301-320. https://doi.org/10.1111/j.1467-9868.2005.00503.x
  18. Zou, H. and Li, R.(2008) One-step sparse estimates in nonconcave penalized likelihood models, An- nals of statistics 36(4), 1509-1533.