Communications for Statistical Applications and Methods

The Korean Statistical Society (한국통계학회)
권호 표지
DOI QR코드

DOI QR Code

Concave penalized linear discriminant analysis on high dimensions

  • Sunghoon Kwon (Department of Applied Statistics, Konkuk University) ;
  • Hyebin Kim (Department of Applied Statistics, Konkuk University) ;
  • Dongha Kim (Department of Statistics, Sungshin Women's University) ;
  • Sangin Lee (Department of Information and Statistics, Chungnam National University)
  • Received : 2023.12.10
  • Accepted : 2024.04.09
  • Published : 2024.07.31
Download PDF
⟨ Previous Next ⟩

Abstract

The sparse linear discriminant analysis can be incorporated into the penalized linear regression framework, but most studies have been limited to specific convex penalties, including the least absolute selection and shrinkage operator and its variants. Within this framework, concave penalties can serve as natural counterparts of the convex penalties. Implementing the concave penalized direction vector of discrimination appears to be straightforward, but developing its theoretical properties remains challenging. In this paper, we explore a class of concave penalties that covers the smoothly clipped absolute deviation and minimax concave penalties as examples. We prove that employing concave penalties guarantees an oracle property uniformly within this penalty class, even for high-dimensional samples. Here, the oracle property implies that an ideal direction vector of discrimination can be exactly recovered through concave penalized least squares estimation. Numerical studies confirm that the theoretical results hold with finite samples.

Keywords

Acknowledgement

This paper was supported by Konkuk University in 2021.

References

  1. Bickel PJ and Levina E (2004). Some theory for fisher's linear discriminant function,naive bayes', and some alternatives when there are many more variables than observations, Bernoulli, 10, 989-1010.
  2. Cai T and Liu W (2011). A direct estimation approach to sparse linear discriminant analysis, Journal of the American Statistical Association, 106, 1566-1577. https://doi.org/10.1198/jasa.2011.tm11199
  3. Clemmensen L, Hastie T, Witten D, and Ersboll B (2011). Sparse discriminant analysis, Technometrics, 53, 406-413. https://doi.org/10.1198/TECH.2011.08118
  4. Fan J and Fan Y (2008). High dimensional classification using features annealed independence rules, Annals of Statistics, 36, 2605-2637.
  5. Fan J, Feng Y, and Tong X (2012). A road to classification in high dimensional space: The regularized optimal affine discriminant, Journal of the Royal Statistical Society Series B: Statistical Methodology, 74, 745-771. https://doi.org/10.1111/j.1467-9868.2012.01029.x
  6. Fan J and Li R (2001). Variable selection via nonconcave penalized likelihood and its oracle properties, Journal of the American Statistical Association, 96, 1348-1360. https://doi.org/10.1198/016214501753382273
  7. Fan J and Peng H (2004). Nonconcave penalized likelihood with a diverging number of parameters, The Annals of Statistics, 32, 928-961.
  8. Fan Y and Tang CY (2012). Tuning parameter selection in high dimensional penalized likelihood, Journal of the Royal Statistical Society Series B: Statistical Methodology, 75, 531-552. https://doi.org/10.1111/rssb.12001
  9. Fisher RA (1936). The use of multiple measurements in taxonomic problems, Annals of Eugenics, 7, 179-188. https://doi.org/10.1111/j.1469-1809.1936.tb02137.x
  10. Gordon GJ, Jensen RV, Hsiao LL et al. (2002). Translation of microarray data into clinically relevant cancer diagnostic tests using gene expression ratios in lung cancer and mesothelioma, Cancer Research, 62, 4963-4967.
  11. Hastie T, Tibshirani R, and Buja A (1994). Flexible discriminant analysis by optimal scoring, Journal of the American Statistical Association, 89, 1255-1270. https://doi.org/10.1080/01621459.1994.10476866
  12. Hastie T, Tibshirani R, and Friedman J (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer Science & Business Media, Berlin.
  13. James GM, Radchenko P, and Lv J (2009). Dasso: Connections between the dantzig selector and lasso, Journal of the Royal Statistical Society Series B: Statistical Methodology, 71, 127-142. https://doi.org/10.1111/j.1467-9868.2008.00668.x
  14. Kim D, Lee S, and Kwon S (2020). A unified algorithm for the non-convex penalized estimation: The ncpen package, The R Journal, 12, 120-133. https://doi.org/10.32614/RJ-2021-007
  15. Kim Y, Choi H, and Oh H-S (2008). Smoothly clipped absolute deviation on high dimensions, Journal of the American Statistical Association, 103, 1665-1673. https://doi.org/10.1198/016214508000001066
  16. Kim Y, Jeon J-J, and Han S (2016). A necessary condition for the strong oracle property, Scandinavian Journal of Statistics, 43, 610-624.
  17. Kim Y and Kwon S (2012). Global optimality of nonconvex penalized estimators, Biometrika, 99, 315-325. https://doi.org/10.1093/biomet/asr084
  18. Kwon S and Kim Y (2012). Large sample properties of the scad-penalized maximum likelihood estimation on high dimensions, Statistica Sinica, 22, 629-653.
  19. Kwon S, Moon H, Chang J, and Lee S (2021). Sufficient conditions for the oracle property in penalized linear regression, The Korean Journal of Applied Statistics, 34, 279-293. https://doi.org/10.5351/KJAS.2021.34.2.279
  20. Mai Q, Zou H, and Yuan M (2012). A direct approach to sparse discriminant analysis in ultra-high dimensions, Biometrika, 99, 29-42. https://doi.org/10.1093/biomet/asr066
  21. Na O and Kwon S (2018). Non-convex penalized estimation for the ar process, Communications for Statistical Applications and Methods, 25, 453-470. https://doi.org/10.29220/CSAM.2018.25.5.453
  22. Shen X, Pan W, Zhu Y, and Zhou H (2013). On constrained and regularized high-dimensional regression, Annals of the Institute of Statistical Mathematics, 65, 807-832. https://doi.org/10.1007/s10463-012-0396-3
  23. Singh D, Febbo PG, Ross K et al. (2002). Gene expression correlates of clinical prostate cancer behavior, Cancer Cell, 1, 203-209. https://doi.org/10.1016/S1535-6108(02)00030-2
  24. Tibshirani R, Hastie T, Narasimhan B, and Chu G (2002). Diagnosis of multiple cancer types by shrunken centroids of gene expression, Proceedings of the National Academy of Sciences, 99, 6567-6572. https://doi.org/10.1073/pnas.082099299
  25. Trendafilov NT and Jolliffe IT (2007). Dalass: Variable selection in discriminant analysis via the lasso, Computational Statistics & Data Analysis, 51, 3718-3736. https://doi.org/10.1016/j.csda.2006.12.046
  26. Witten DM and Tibshirani R (2011). Penalized classification using fisher's linear discriminant, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73, 753-772. https://doi.org/10.1111/j.1467-9868.2011.00783.x
  27. Zhang C-H (2010). Nearly unbiased variable selection under minimax concave penalty, The Annals of Statistics, 38, 894-942.
  28. Zhang C-H and Zhang T (2012). A general theory of concave regularization for high-dimensional sparse estimation problems, Statistical Science, 27, 576-593.
  29. Zhao P and Yu B (2006). On model selection consistency of lasso, Journal of Machine Learning Research, 7, 2541-2563.