Journal of the Korea Society of Computer and Information (한국컴퓨터정보학회논문지)

Korean Society of Computer Information (한국컴퓨터정보학회)
권호 표지
DOI QR코드

DOI QR Code

Multi-label Feature Selection Using Redundancy and Relevancy based on Regression Optimization

  • Hyunki Lim (Div. of AI Computer Science and Engineering, Kyonggi University)
  • Received : 2024.08.20
  • Accepted : 2024.11.14
  • Published : 2024.11.29
Download PDF
⟨ Previous Next ⟩

Abstract

High-dimensional data causes difficulties in machine learning due to high time consumption and large memory requirements. In particular, in a multi-label environment, higher complexity is required as much as the number of labels. This paper proposes a feature selection method to improve classification performance in multi-label settings. The method considers three types of relationships: between features, between features and labels, and between labels themselves. To achieve this, a regression-based objective function is designed. This objective function calculates the linear relationships between features and labels and uses mutual information to compute relationships between features and between labels. By minimizing this objective function, the optimal weights for feature selection are found. To optimize the objective function, a gradient descent method is applied to develop a fast-converging algorithm. The experimental results on six multi-label datasets show that the proposed method outperforms existing multi-label feature selection techniques. The classification performance of the proposed method, averaged over six datasets, showed a Hamming loss of 0.1285, a ranking loss of 0.1811, and a multi-label accuracy of 0.6416. Compared to the AMI(Approximating Mutual Information) algorithm, the performance was better by 0.0148, 0.0435, and 0.0852, respectively.

고차원 데이터는 기계학습에서 높은 시간 소모와 큰 메모리 요구로 학습에 어려움을 유발한다. 특히 다중 레이블 환경에서는 레이블의 개수만큼 더 높은 복잡도를 요구한다. 본 논문에서는 다중 레이블 환경에서 분류 성능 향상을 위한 특징 선별 기법을 제안한다. 중요한 특징을 선별하기 위해 특징과 특징, 특징과 레이블, 레이블과 레이블, 세 가지 관계를 고려했으며 이를 위해 회귀 기반 목적 함수를 설계하였다. 이 목적 함수는 특징과 레이블 사이의 선형적인 관계를 계산하고 상호 정보 기반의 특징과 특징, 레이블과 레이블 사이의 관계를 계산한다. 이 목적 함수를 최소화하는 가중치를 찾아 특징을 선별할 수 있다. 이 목적 함수 최적화를 위해 경사하강법 방식을 제시하여 빠르게 수렴할 수 있는 알고리즘을 제안하였다. 여섯 개의 다중 레이블 데이터에 대한 실험 결과 제안된 방법이 기존 다중 레이블 특징 선별 기법보다 높은 성능을 보여주었다. 여섯 개의 데이터 평균으로 본 제안 방법의 분류 성능은 해밍 로스 0.1285, 랭킹 로스 0.1811, 다중 레이블 정확도 0.6416으로, 비교 대상 알고리즘인 AMI(Approximating Mutual Information)에 비해 각각 0.0148, 0.0435, 0.0852 더 우수한 성능을 보였다.

Keywords

Acknowledgement

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (grant number 2020R1A6A1A03040583).

References

  1. E. Smirni, and G. Ciardo, "Workload-Aware Load Balancing for Cluster Web Servers," IEEE Trans. on Parallel and Distributed Systems, Vol. 16, No. 3, pp. 219-232, March 2005. DOI: 10.1016/j.patcog.2019.03.026
  2. E. Elhamifar and R. Vidal, "Sparse subspace clustering: Algorithm, theory, and applications," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 35, No. 11, pp. 2765-2781, March 2013. DOI: 10.1109/TPAMI.2013.57
  3. H. Lim, J. Lee, and D.-W. Kim, "Optimization approach for feature selection in multi-label classification," Pattern Recognition Letters, Vol. 89, pp. 25-30, April 2017. DOI: 10.1016/j.patrec.2017.02.004
  4. J. Lee, and D.-W. Kim, "Fast multi-label feature selection based on information-theoretic feature ranking," Pattern Recognition, Vol. 48, pp. 2761-2771, September 2015. DOI: 10.1016/j.patcog.2015.04.009
  5. S. Sharmin, M. Shoyaib, A. A. Ali, M. A. H. Khan, and O. Chae, "Simultaneous feature selection and discretization based on mutual information," Pattern Recognition, Vol. 91, pp. 162-174, July 2019. DOI: 10.1016/j.patcog.2019.02.016
  6. R. B. Pereira. A. Plastino, B. Zadrozny, L.H.C. Merschmann, "Categorizing feature selection methods for multi-label classification," Artificial Intelligence Review, Vol. 49, pp. 57-78, September 2018. DOI: 10.1007/s10462-016-9516-4
  7. G. Tsoumakas, and I. Vlahavas, "Random k-labelsets: an ensemble method for multilabel classification," In Proceedings of the European Conference on Machine Learning, pp 406-417, 2007.
  8. J. Read, "A pruned problem transformation method for multi-label classification," In Proceedings of the New Zealand Computer Science Research Student Conference, pp 143-150, 2008.
  9. M.L. Zhang, J.M. Pena, and V. Robles, "Feature selection for multi-label naive Bayes classification," Vol. 179, pp. 3218-3229, Information Sciences, September 2009. DOI: 10.1016/j.ins.2009.06.010
  10. A. Clare and R.D. King, "Knowledge discovery in multi-label phenotype data," In Proceedings of the European Conference on Principles of Data Mining and Knowledge Discovery, pp. 42-53, 2001.
  11. J. Lee and D.-W. Kim, "Feature selection for multi-label classification using multivariate mutual information," Pattern Recognition Letters, Vol. 34, pp. 349-357, February 2013. DOI: 10.1016/j.patrec.2012.10.005
  12. Y. Fan, B. Chen, W. Huang, J. Liu, and W. Weng, "Multi-label feature selection based on label correlations and feature redundancy," Vol. 241, pp. 108256, Knowledge-Based Systems, April 2022. DOI: 10.1016/j.knosys.2022.108256
  13. Y. Li, L. Hu, and W. Gao, "Multi-label feature selection via robust flexible sparse regularization," Vol. 134, pp. 109074, Pattern Recognition, February 2023. DOI: 10.1016/j.patcog.2022.109074
  14. Y. Fan, J. Liu, J. Tang, P. Liu, Y. Lin, and Y. Du, "Learning correlation information for multi-label feature selection," Vol. 145, pp. 109899, Pattern Recognition, January 2024. DOI: 10.1016/j.patcog.2023.109899
  15. Y. Li, L. Hu, and W. Gao, "Label correlations variation for robust multi-label feature selection," Vol. 609, pp. 1075-1097, Information Sciences, September 2022. DOI: 10.1016/j.ins.2022.07.154
  16. L. Hu, L. Gao, Y. Li, P. Zhang, and W. Gao, "Feature-specific mutual information variation for multi-label feature selection," Vol. 593, pp. 449-471, Information Sciences, May 2022. DOI: 10.1016/j.ins.2022.02.024
  17. J. Dai, W. Huang, C. Zhang, and J. Liu, "Multi-label feature selection by strongly relevant label gain and label mutual aid," Vol. 145, pp. 109945, Pattern Recognition, January 2024. DOI: 10.1016/j.patcog.2023.109945
  18. M. Faraji, S.A. Seyedi, F.A. Tab, and R. Mahmoodi, "Multi-label feature selection with global and local label correlation," Vol. 246, pp. 123198, Expert Systems with Applications, July 2024. DOI: 10.1016/j.eswa.2024.123198
  19. Z. He, Y. Lin, C. Wang, L. Guo, and W. Ding, "Multi-label feature selection based on correlation label enhancement," Vol. 647, pp. 119526, Information Sciences, November 2023. DOI: 10.1016/j.ins.2023.119526
  20. P. Zhang, G. Liu, and J. Song, "MFSJMI: Multi-label feature selection considering join mutual information and interaction weight," Vol. 138, pp. 109378, Pattern Recognition, June 2023. DOI: 10.1016/j.patcog.2023.109378
  21. Y. Yang, H. Chen, Y. Mi, C. Luo, S.-J. Horng, and Tianrui Li, "Multi-label feature selection based on stable label relevance and label-specific features," Vol. 648, pp. 119525, Information Sciences, November 2023. DOI: 10.1016/j.ins.2023.119525
  22. H. Peng, F. Long, and C. Ding, "Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 27, pp. 1226-1238, June 2005. DOI: 10.1109/TPAMI.2005.159
  23. T. Klonecki, P. Teisseyre, and J. Lee, "Cost-constrained feature selection in multilabel classification using an information-theoretic approach," Pattern Recognition, Vol. 141, pp. 1-18, September 2023. DOI: 10.1016/j.patcog.2023.109605
  24. J. Lee and D.-W. Kim, "Mutual information-based multi-label feature selection using interaction information," Expert Systems with Applications, Vol. 42, pp. 2013-2025, March 2015. DOI: 10.1016/j.eswa.2014.09.063
  25. M.-L. Zhang and Z.-H. Zhou, "ML-KNN: A lazy learning approach to multi-label learning," Pattern Recognition, Vol. 40, pp. 2038-2048, July 2007. DOI: 10.1016/j.patcog.2006.12.019
  26. K. Trohidis, G. Tsoumakas, G. Kalliris, and I. Vlahavas, "Multilabel classification of music into emotions," International Conference on Music Information Retrieval, pp. 325-330, 2008.
  27. S. Diplaris, G. Tsoumakas, P. Mitkas and I. Vlahavas, "Protein Classification with Multiple Algorithms," Panhellenic Conference on Informatics, pp. 448-456, 2005.
  28. J.P. Pestian, C. Brew, P. Matykiewicz, D.J. Hovermale, N. Johnson, K.B. Cohen, and W. Duch, "A shared task involving multi-label classification of clinical free text," Proceedings of the Workshop on BioNLP 2007: Biological, Translational, and Clinical Language Processing, pp. 97-104, 2007.
  29. M.R. Boutell, J. Luo, X. Shen, and C.M. Brown, "Learning multi-labelscene classiffication," Pattern Recognition, Vol. 37, pp. 1757-1771, September 2004. DOI: 10.1016/j.patcog.2004.03.009
  30. A. Elisseeff and J. Weston, "A kernel method for multi-labelled classification," Advances in Neural Information Processing Systems, 2001.
  31. J. Lee and D.-W. Kim, "SCLS: Multi-label feature selection based on scalable criterion for large label set," Pattern Recognition, Vol. 66, pp. 342-352, June 2017. DOI: https://doi.org/10.1016/j.patcog.2017.01.014
  32. J. Lee, H. Lim, and D.-W. Kim, "Approximating mutual information for multi-label feature selection," Electronics Letters, Vol. 48, pp. 1-2, July 2012. DOI: 10.1049/el.2012.1600
  33. Y. Lin, Q. Hu, J. Liu, and J. Duan, "Multi-label feature selection based on max-dependency and min-redundancy," Neurocomputing, Vol. 168, pp. 92-103, November 2015. DOI: 10.1016/j.neucom.2015.06.010