Journal of the Korean Society for Industrial and Applied Mathematics

Korean Society for Industrial and Applied Mathematics (한국산업응용수학회)
권호 표지
DOI QR코드

DOI QR Code

UNDERSTANDING NON-NEGATIVE MATRIX FACTORIZATION IN THE FRAMEWORK OF BREGMAN DIVERGENCE

  • KIM, KYUNGSUP (DEPARTMENT OF COMPUTER ENGINEERING, CHUNGNAM NATIONAL UNIVERSITY)
  • Received : 2021.04.30
  • Accepted : 2021.09.23
  • Published : 2021.09.25
Download PDF
⟨ Previous Next ⟩

Abstract

We introduce optimization algorithms using Bregman Divergence for solving non-negative matrix factorization (NMF) problems. Bregman divergence is known a generalization of some divergences such as Frobenius norm and KL divergence and etc. Some algorithms can be applicable to not only NMF with Frobenius norm but also NMF with more general Bregman divergence. Matrix Factorization is a popular non-convex optimization problem, for which alternating minimization schemes are mostly used. We develop the Bregman proximal gradient method applicable for all NMF formulated in any Bregman divergences. In the derivation of NMF algorithm for Bregman divergence, we need to use majorization/minimization(MM) for a proper auxiliary function. We present algorithmic aspects of NMF for Bregman divergence by using MM of auxiliary function.

Keywords

Acknowledgement

This work was supported by research fund of Chungnam National University.

References

  1. Masoud Ahookhosh, Le Thi Khanh Hien., Nicolas Gillis, and Panagiotis Patrinos. Multi-block Bregman proximal alternating linearized minimization and its application to orthogonal nonnegative matrix factorization. 30468160(30468160).
  2. Heinz H. Bauschke, Jerome Bolte, and Marc Teboulle. A descent Lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications. INFORMS, 79(0):21, 2007.
  3. Jerome Bolte, Shoham Sabach, Marc Teboulle, and Yakov Vaisbourd. First order methods beyond convexity and lipschitz gradient continuity with applications to quadratic inverse problems. SIAM Journal on Optimization, 28(3):2131-2151, 2018. https://doi.org/10.1137/17M1138558
  4. Andrzej Cichocki and Shun ichi Amari. Families of alpha- beta- and gamma- divergences: Flexible and robust measures of Similarities. Entropy, 12(6):1532-1568, 2010. https://doi.org/10.3390/e12061532
  5. Andrzej Cichocki, Hyekyoung Lee, Yong-deok Kim, and Seungjin Choi. Non-negative matrix factorization with α-divergence. Pattern Recognition Letters, 29:1433-1440, 2008. https://doi.org/10.1016/j.patrec.2008.02.016
  6. Andrzej Cichocki, Rafal Zdunek, and Shun-ichi AMARI. Csiszar's Divergences for Non-negative Matrix ' Factorization : Family of New Algorithms. In Independent Component Analysis and Blind Signal Separation. ICA 2006. Lecture Notes in Computer Science. 2006.
  7. Andrzej Cichocki, Rafal Zdunek, Seungjin Choi, Robert Plemmons, and Shun-ichi Amari. Non-negative tensor factorization using alpha and beta divergences. In In IEEE International Conference on Acoustics, Speech,, 2007.
  8. Cedric Fevotte, Nancy Bertin, and Jean Louis Durrieu. Nonnegative matrix factorization with the Itakura-Saito divergence: With application to music analysis. Neural Computation, 21(3):793-830, 2009. https://doi.org/10.1162/neco.2008.04-08-771
  9. Cedric Fevotte and Jerome Idier. Algorithms for nonnegative matrix factorization with the β-divergence. Neural Computation, 23(9):2421-2456, 2011. https://doi.org/10.1162/NECO_a_00168
  10. Romain Hennequin, Bertrand David, and Roland Badeau. Beta-divergence as a subclass of Bregman divergence. IEEE Signal Processing Letters, 18(2):83-86, 2011. https://doi.org/10.1109/LSP.2010.2096211
  11. Le Thi Khanh Hien and Nicolas Gillis. Algorithms for Nonnegative Matrix Factorization with the Kullback-Leibler Divergence. Accepted in the Journal of Scientific Computing, oct 2020.
  12. Le Thi Khanh Hien, Nicolas Gillis, and Panagiotis Patrinos. Inertial Block Proximal Methods for Non-Convex Non-Smooth Optimization. 2019.
  13. Le Thi Khanh Hien, Duy Nhat Phan, and Nicolas Gillis. An Inertial Block Majorization Minimization Framework for Nonsmooth Nonconvex Optimization. 2020.
  14. Kevin L. Keys, Hua Zhou, and Kenneth Lange. Proximal distance algorithms: Theory and practice. Journal of Machine Learning Research, 20:1-38, 2019.
  15. Keigo Kimura, Mineichi Kudo1, and Yuzuru Tanaka1. A column-wise update algorithm for nonnegative matrix factorization in Bregman divergence with an orthogonal constraint. Machine Learning, 103(2):285-306, 2016. https://doi.org/10.1007/s10994-016-5553-0
  16. Liangda Li, Guy Lebanon, and Haesun Park. Fast bregman divergence NMF using taylor expansion and coordinate descent. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 307-315, 2012.
  17. Haihao Lu, Robert M. Freund, and Yurii Nesterov. Relatively smooth convex optimization by first-order methods, and applications. SIAM Journal on Optimization, 28(1):333-354, 2018. https://doi.org/10.1137/16M1099546
  18. Julien Mairal. Optimization with first-order surrogate functions. 30th International Conference on Machine Learning, ICML 2013, 28(PART 3):1820-1828, 2013.
  19. Macoumba Ndour, Mactar Ndaw, and Papa Ngom. Relationship between the Bregman divergence and beta-divergence and their Applications. arXiv, (November), 2018.
  20. Vincent Y F Tan and Ce dric Fe Votte. Automatic Relevance Determination in Nonnegative Matrix Factorization with the beta-Divergence. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(7):1592-1605, 2013. https://doi.org/10.1109/TPAMI.2012.240
  21. Zhongming Wu, Chongshou Li, Min Li, and Andrew Lim. Inertial proximal gradient methods with Bregman regularization for a class of nonconvex optimization problems. Journal of Global Optimization, 79(3):617-644, 2021. https://doi.org/10.1007/s10898-020-00943-7
  22. Zhihua Zhang, James T. Kwok, and Dit Yan Yeung. Surrogate maximization/minimization algorithms and extensions. Machine Learning, 69(1):1-33, 2007. https://doi.org/10.1007/s10994-007-5022-x