Journal of IKEEE (전기전자학회논문지)

Institute of Korean Electrical and Electronics Engineers (한국전기전자학회)
권호 표지
DOI QR코드

DOI QR Code

Implementation of the System Converting Image into Music Signals based on Intentional Synesthesia

의도적인 공감각 기반 영상-음악 변환 시스템 구현

  • Bae, Myung-Jin (Dept. of Convergence IT Engineering, Kyungnam University) ;
  • Kim, Sung-Ill (Dept. of Electronic Engineering, Kyungnam University)
  • Received : 2020.03.06
  • Accepted : 2020.03.27
  • Published : 2020.03.31
Download PDF
⟨ Previous Next ⟩

Abstract

This paper is the implementation of the conversion system from image to music based on intentional synesthesia. The input image based on color, texture, and shape was converted into melodies, harmonies and rhythms of music, respectively. Depending on the histogram of colors, the melody can be selected and obtained probabilistically to form the melody. The texture in the image expressed harmony and minor key with 7 characteristics of GLCM, a statistical texture feature extraction method. Finally, the shape of the image was extracted from the edge image, and using Hough Transform, a frequency component analysis, the line components were detected to produce music by selecting the rhythm according to the distribution of angles.

본 논문은 사전에 학습된 기억으로 공감각 현상을 지각할 수 있는 의도적인 공감각으로 영상에서 음악으로 변환하는 시스템을 구현하였다. 영상에서 변환정보로 색상(Color), 질감(Texture), 모양(Shape)을 사용하여 음악의 멜로디(Melody), 하모니(Harmony), 리듬(Rhythm) 정보로 변환하였다. 정적인 영상에서 단조로운 음이 반복되는 것을 최소화하고 영상에 있는 정보를 표현하기 위해 색상의 분포도에 따라 확률적으로 멜로디를 선택하여 출력함으로써 자연스럽게 음을 구성할 수 있도록 하였고, 영상에서 질감은 통계적 질감 특징 추출방식인 GLCM(Gray-Level Co-occurrence Matrix)의 7가지 특징으로 하모니의 장조와 단조를 표현하였다. 마지막으로 모양은 영상의 외곽선을 추출한 후 주파수 성분 분석인 허프 변환(Hough Transform)을 이용해 선 성분을 검출하여 각도의 분포에 따라 리듬을 선택하는 방식으로 음악을 생성하였다.

Keywords

References

  1. J. Ward, "Synesthesia," Annual Review of Psychology, Vol.64, No.1, pp.49-75, 2013. DOI: 10.1146/annurev-psych-113011-143840
  2. Suslick, S Kenneth. "Synesthesia in science and technology: more than making the unseen visible," Current opinion in chemical biology, Vol.16, No.5, pp.557-563, 2012. DOI: 10.1016/j.cbpa.2012.10.030
  3. Smilek, D., Dixon, M. J. & Merikle, P. M. "Synaesthesia: discordant male monozygotic twins," Neurocase, Vol.11, No.5, pp.363-370, 2005. DOI: 10.1080/13554790500205413
  4. Smilek, D. et al. "Synaesthesia: a case study of discordant monozygotic twins," Neurocase, Vol.8, No.4, pp.338-342, 2002. DOI: 10.1076/neur.8.3.338.16194
  5. S. Kim, "Conversion of Image into Sound Based on HSI Histogram," The Journal of the Acoustical Society of Korea, Vol.30, No.3, pp.142-148, 2011. DOI: 10.7776/ASK.2011.30.3.142
  6. M. Bae, S. Kim, "Implementation of the Visualization and Sonification System of Human Movements based on Intentional Synesthesia," Journal of Korean institute of intelligent systems, Vol.28, No.1, pp.83-90, 2018. DOI: 10.7471/ikeee.2018.22.2.362
  7. D. Bor, N. Rothen, D. Schwartzman, S. Clayton, A. Seth, "Adults Can Be Trained to Acquire Synesthetic Experiences," Scientific Reports 4, No.7089, 2014. DOI: 10.1038/srep07089
  8. Witthoft, Nathan, and Jonathan Winawer. "Learning, memory, and synesthesia," Psychological science, Vol.24, No.3, pp.258-265, 2013. DOI: 10.1177/0956797612452573
  9. M. Bae, S. Kim, "The System of Converting Muscular Sense Into both Color and Sound Based on the Synesthetic Perception," Journal of Korean institute of intelligent systems, Vol.24 No.5, pp.462-469, 2014. DOI: 10.5391/JKIIS.2014.24.5.462