Nutrition Research and Practice

The Korean Nutrition Society (한국영양학회)
권호 표지
DOI QR코드

DOI QR Code

The development of food image detection and recognition model of Korean food for mobile dietary management

  • Park, Seon-Joo (Department of Food and Nutrition, College of BioNano Technology, Gachon University) ;
  • Palvanov, Akmaljon (Department of Computer Engineering, College of IT, Gachon University) ;
  • Lee, Chang-Ho (Research Group of Functional Food Materials, Korea Food Research Institute) ;
  • Jeong, Nanoom (Department of Food and Nutrition, College of BioNano Technology, Gachon University) ;
  • Cho, Young-Im (Department of Computer Engineering, College of IT, Gachon University) ;
  • Lee, Hae-Jeung (Department of Food and Nutrition, College of BioNano Technology, Gachon University)
  • Received : 2019.05.28
  • Accepted : 2019.08.18
  • Published : 2019.12.01
Download PDF
⟨ Previous Next ⟩

Abstract

BACKGROUND/OBJECTIVES: The aim of this study was to develop Korean food image detection and recognition model for use in mobile devices for accurate estimation of dietary intake. MATERIALS/METHODS: We collected food images by taking pictures or by searching web images and built an image dataset for use in training a complex recognition model for Korean food. Augmentation techniques were performed in order to increase the dataset size. The dataset for training contained more than 92,000 images categorized into 23 groups of Korean food. All images were down-sampled to a fixed resolution of $150{\times}150$ and then randomly divided into training and testing groups at a ratio of 3:1, resulting in 69,000 training images and 23,000 test images. We used a Deep Convolutional Neural Network (DCNN) for the complex recognition model and compared the results with those of other networks: AlexNet, GoogLeNet, Very Deep Convolutional Neural Network, VGG and ResNet, for large-scale image recognition. RESULTS: Our complex food recognition model, K-foodNet, had higher test accuracy (91.3%) and faster recognition time (0.4 ms) than those of the other networks. CONCLUSION: The results showed that K-foodNet achieved better performance in detecting and recognizing Korean food compared to other state-of-the-art models.

Keywords

References

  1. De Keyzer W, Bracke T, McNaughton SA, Parnell W, Moshfegh AJ, Pereira RA, Lee HS, van't Veer P, De Henauw S, Huybrechts I. Cross-continental comparison of national food consumption survey methods--a narrative review. Nutrients 2015;7:3587-620. https://doi.org/10.3390/nu7053587
  2. Johnson RK, Soultanakis RP, Matthews DE. Literacy and body fatness are associated with underreporting of energy intake in US low-income women using the multiple-pass 24-hour recall: a doubly labeled water study. J Am Diet Assoc 1998;98:1136-40. https://doi.org/10.1016/S0002-8223(98)00263-6
  3. Stumbo PJ. New technology in dietary assessment: a review of digital methods in improving food record accuracy. Proc Nutr Soc 2013;72:70-6. https://doi.org/10.1017/S0029665112002911
  4. Martin CK, Kaya S, Gunturk BK. Quantification of food intake using food image analysis. Conf Proc IEEE Eng Med Biol Soc 2009;2009:6869-72.
  5. Lazarte CE, Encinas ME, Alegre C, Granfeldt Y. Validation of digital photographs, as a tool in 24-h recall, for the improvement of dietary assessment among rural populations in developing countries. Nutr J 2012;11:61. https://doi.org/10.1186/1475-2891-11-61
  6. Kikunaga S, Tin T, Ishibashi G, Wang DH, Kira S. The application of a handheld personal digital assistant with camera and mobile phone card (Wellnavi) to the general population in a dietary survey. J Nutr Sci Vitaminol (Tokyo) 2007;53:109-16. https://doi.org/10.3177/jnsv.53.109
  7. Wang DH, Kogashiwa M, Ohta S, Kira S. Validity and reliability of a dietary assessment method: the application of a digital camera with a mobile phone card attachment. J Nutr Sci Vitaminol (Tokyo) 2002;48:498-504. https://doi.org/10.3177/jnsv.48.498
  8. Khanna N, Boushey CJ, Kerr D, Okos M, Ebert DS, Delp EJ. An overview of the technology assisted dietary assessment project at Purdue University. ISM 2010:290-5.
  9. Martin CK, Correa JB, Han H, Allen HR, Rood JC, Champagne CM, Gunturk BK, Bray GA. Validity of the remote food photography method (RFPM) for estimating energy and nutrient intake in near real-time. Obesity (Silver Spring) 2012;20:891-9. https://doi.org/10.1038/oby.2011.344
  10. Beltran A, Dadabhoy H, Ryan C, Dholakia R, Jia W, Baranowski J, Sun M, Baranowski T. Dietary assessment with a wearable camera among children: feasibility and intercoder reliability. J Acad Nutr Diet 2018;118:2144-53. https://doi.org/10.1016/j.jand.2018.05.013
  11. Wang DH, Kogashiwa M, Kira S. Development of a new instrument for evaluating individuals' dietary intakes. J Am Diet Assoc 2006;106:1588-93. https://doi.org/10.1016/j.jada.2006.07.004
  12. He H, Kong F, Tan J. DietCam: multiview food recognition using a multikernel SVM. IEEE J Biomed Health Inform 2016;20:848-55. https://doi.org/10.1109/JBHI.2015.2419251
  13. Zhang XJ, Lu YF, Zhang SH. Multi-task learning for food identification and analysis with deep convolutional neural networks. J Comput Sci Technol 2016;31:489-500. https://doi.org/10.1007/s11390-016-1642-6
  14. Mezgec S, Korousic Seljak B. NutriNet: a deep learning food and drink image recognition system for dietary assessment. Nutrients 2017;9:9. https://doi.org/10.3390/nu9010009
  15. Chang UJ, Ko SA. A study on the dietary intake survey method using a cameraphone. Korean J Community Nutr 2007;12:198-205.
  16. Son HR, Lee SM, Khil JM. Evaluation of a dietary assessment method using photography for portion size estimation. J Korean Soc Food Cult 2017;32:162-73. https://doi.org/10.7318/KJFC/2017.32.2.162
  17. Jung H, Yoon J, Choi KS, Chung SJ. Feasibility of using digital pictures to examine individuals' nutrient intakes from school lunch: a pilot study. J Korean Diet Assoc 2009;15:278-85.
  18. Matsuda Y, Yanai K, editors. Multiple-food recognition considering co-occurrence employing manifold ranking. Proceedings of the 21st International Conference on Pattern Recognition; 2012 Nov 11-15; Tsukuba, Japan. Piscataway: IEEE; 2013 Feb. 2017-20 p.
  19. Zhu F, Bosch M, Woo I, Kim S, Boushey CJ, Ebert DS, Delp EJ. The use of mobile devices in aiding dietary assessment and evaluation. IEEE J Sel Top Signal Process 2010;4:756-66. https://doi.org/10.1109/JSTSP.2010.2051471
  20. Daugherty BL, Schap TE, Ettienne-Gittens R, Zhu FM, Bosch M, Delp EJ, Ebert DS, Kerr DA, Boushey CJ. Novel technologies for assessing dietary intake: evaluating the usability of a mobile telephone food record among adults and adolescents. J Med Internet Res 2012;14:e58. https://doi.org/10.2196/jmir.1967
  21. Kong F, Tan J. DietCam: automatic dietary assessment with mobile camera phones. Pervasive Mob Comput 2012;8:147-63. https://doi.org/10.1016/j.pmcj.2011.07.003
  22. Kawano Y, Yanai K, editors. Real-time mobile food recognition system. 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops; 2013 Jun 23-28; Portland, USA. Piscataway: IEEE; 2013 Sep. 1-7 p.
  23. Jia W, Li Y, Qu R, Baranowski T, Burke LE, Zhang H, Bai Y, Mancino JM, Xu G, Mao ZH, Sun M. Automatic food detection in egocentric images using artificial intelligence technology. Public Health Nutr 2019;22:1168-79.
  24. Kagaya H, Aizawa K, Ogawa M, editors. Food detection and recognition using convolutional neural network. Proceedings of the 22nd ACM International Conference on Multimedia; 2014 Nov 3-7; Orlando, USA. New York: ACM; 2014 Nov. 1085-8 p.
  25. Kingma DP, Ba J. Adam: a method for stochastic optimization. Published as a Conference Paper at the 3rd International Conference for Learning Representations; 2015 May 7-9; San Diego, USA. La Jolla: ICLR; 2017 Jan. 1-15 p.
  26. Pang T, Xu K, Dong Y, Du C, Chen N, Zhu J. Rethinking softmax cross-entropy loss for adversarial robustness [Internet]. Ithaca (NY): Cornell University; 2019 [cited 2019 June 26]. Available from: https://arxiv.org/abs/1905.10626.
  27. Farooq M, Doulah A, Parton J, McCrory MA, Higgins JA, Sazonov E. Validation of sensor-based food intake detection by multicamera video observation in an unconstrained environment. Nutrients 2019;11:E609. https://doi.org/10.3390/nu11030609
  28. Liu C, Cao Y, Luo Y, Chen G, Vokkarane V, Ma Y, editors. DeepFood: deep learning-based food image recognition for computer-aided dietary assessment. ICOST 2016 Proceedings of the 14th International Conference on Inclusive Smart Cities and Digital Health; 2016 May 25-27; Wuhan, China. Cham: Springer International Publishing; 2016 May. 37-48 p.
  29. Hassannejad H, Matrella G, Ciampolini P, De Munari I, Mordonini M, Cagnoni S, editors. Food image recognition using very deep convolutional networks. Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management; 2016 Oct 16; Amsterdam, The Netherlands. New York: ACM; 2016 Oct. 41-9 p.
  30. Christodoulidis S, Anthimopoulos M, Mougiakakou S, editors. Food recognition for dietary assessment using deep convolutional neural networks. New Trends in Image Analysis and Processing -- ICIAP 2015 Workshops; 2015 Sep 7-8; Genova, Italy. Cham: Springer International Publishing; 2015 Aug. 458-65 p.
  31. Lu Y. Food image recognition by using convolutional neural networks (CNNs) [Internet]. Ithaca (NY): Cornell University; 2019 [cited 2019 May 28]. Available from: https://arxiv.org/abs/1612.00983.
  32. Pandey P, Deepthi A, Mandal B, Puhan NB. FoodNet: recognizing food using ensemble of deep networks. IEEE Signal Process Lett 2017;24:1758-62.

Cited by

  1. Prediction Model of Aryl Hydrocarbon Receptor Activation by a Novel QSAR Approach, DeepSnap–Deep Learning vol.25, pp.6, 2019, https://doi.org/10.3390/molecules25061317