Journal of the Korea Society of Computer and Information (한국컴퓨터정보학회논문지)

Korean Society of Computer Information (한국컴퓨터정보학회)
권호 표지
DOI QR코드

DOI QR Code

A Novel Method of Improving Cache Hit-rate in Hadoop MapReduce using SSD Cache

  • Kim, Jong-Chan (Intelligent IDC Project Office, Korea Electronics Technology Institute) ;
  • An, Jae-Hoon (Intelligent IDC Project Office, Korea Electronics Technology Institute) ;
  • Kim, Young-Hwan (Intelligent IDC Project Office, Korea Electronics Technology Institute) ;
  • Jeon, Ki-Man (Intelligent IDC Project Office, Korea Electronics Technology Institute)
  • Received : 2015.06.17
  • Accepted : 2015.07.31
  • Published : 2015.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

The MapReduce Program of Hadoop Distributed File System operates on any unspecified nodes due to distributed-parallel process and block replicate for data stability. Since it is difficult to guarantee the cache locality when a Solid State Drive is used as a cache in hadoop, cache hit-rate is decreased. In this paper, we suggest a method to improve cache hit rate by pre-loading the input data of the MapReduce onto the SSD cache. To perform this method, we estimated the blocks that are used on each node by using capacity scheduler and block metadata. Eventually we could increase the performance of SSD cache by loading the blocks onto SSD cache before the Map Task run.

Keywords

References

  1. "Hadoop.", from http://hadoop.apache.org
  2. "Solid-state drive, Wikipedia", https://en.wikipedia.org/wiki/Solid-state_drive
  3. Shvachko K., Kuang H., Radia S., and Chansler R, "The hadoop distributed file system.", In Mass Storage Systems and Technologies (MSST), 2010 IEEE 26th Symposium, pp. 1-10, May, 2010
  4. J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters." In Communications of the ACM, Vol.51, No.1, pp. 107-113, Jan, 2008 https://doi.org/10.1145/1327452.1327492
  5. "Flashcache.", https://wiki.archlinux.org/index.php/Flashcache
  6. "Flashcache project", https://github.com/facebook/flashcache
  7. "Hadoop's Capacity Scheduler", http://hadoop.apache.org/core/docs/current/capacity_scheduler.html
  8. Arun C. Murthy. "Apache Hadoop YARN, Moving beyond MapReduce and Batch Processing with Apache Hadoop 2", Pearson Education, pp. 153-170, 2014
  9. S. H. Kang, D. H. Koo, W. H. Kang, and S. W. Lee, "A case for flash memory ssd in hadoop applications." International Journal of Control and Automation, Vol.6, No.1, pp. 201-210, Feb, 2013 https://doi.org/10.14257/ijca.2013.6.6.19
  10. T. H. Keum, W. J. Lee, and C. H. Jeon, "A Performance Analysis Based on Hadoop Application's Characteristics in Cloud Computing", Journal of The Korea Society of Computer and Information, Vol.15, No.5, pp.49-56, May, 2010 https://doi.org/10.9708/jksci.2010.15.5.049
  11. J. S. Kim, C. H. Kim, W. J. Lee, and C. H. Jeon "A Block Relocation Algorithm for Reducing Network Consumption in Hadoop Cluster", Journal of The Korea Society of Computer and Information, Vol.19, No.11, pp.9-15, Nov,2014 https://doi.org/10.9708/JKSCI.2014.19.11.009