DOI QR코드

DOI QR Code

Delayed Block Replication Scheme of Hadoop Distributed File System for Flexible Management of Distributed Nodes

하둡 분산 파일시스템에서의 유연한 노드 관리를 위한 지연된 블록 복제 기법

  • Ryu, Woo-Seok (Dept. of Health Care Management, Catholic University of Pusan)
  • 류우석 (부산가톨릭대학교 병원경영학과)
  • Received : 2017.04.10
  • Accepted : 2017.04.24
  • Published : 2017.04.30

Abstract

This paper discusses management problems of Hadoop distributed node, which is a platform for big data processing, and proposes a novel technique for enabling flexible node management of Hadoop Distributed File System. Hadoop cannot configure Hadoop cluster dynamically because it judges temporarily unavailable nodes as a failure. Delayed block replication scheme proposed in this paper delays the removal of unavailable node as much as possible so as to be easily rejoined. Experimental results show that the proposed scheme increases flexibility of node management with little impact on distributed processing performance when the cluster size changes.

Acknowledgement

Supported by : 한국연구재단

References

  1. H. Yoon, "Development of Contents on the Marine Meteorology Service by Meteorology and Climate Big Data," J. of The Korea Institute of Electronic Communication Sciences, vol. 11, no. 2, 2016, pp. 125-138. https://doi.org/10.13067/JKIECS.2016.11.2.125
  2. H. Chen, R. Chiang, and V. C. Storey, "Business intelligence and analytics: From big data to big impact," MIS Quarterly, vol. 36, no. 4, 2012, pp. 1165-1188.
  3. C. Ryu, "Context Inference and Sensor Data Classification of Big Data Stream Environment," J. of The Korea Institute of Electronic Communication Sciences, vol. 9, no. 10, 2014, pp. 1079-1085. https://doi.org/10.13067/JKIECS.2014.9.10.1079
  4. W. Raghupathi and V. Raghupathi, "Big data analytics in healthcare: promise and potential," Health Information Science and Systems, vol. 2, no. 1, 2014, pp. 1-10. https://doi.org/10.1186/2047-2501-2-1
  5. J. Choi, "Utilization value of medical Big Data created in operation of medical information system," J. of The Korea Institute of Electronic Communication Sciences, vol. 10, no. 12, 2015, pp. 1403-1410. https://doi.org/10.13067/JKIECS.2015.10.12.1403
  6. K. Shvachko, H. Kuang, S. Radia, and R. Chansler, "The Hadoop Distributed File System," In Proc. IEEE Symp. on Mass Storage Systems and Technologies (MSST), NV, USA, May 2010, pp. 1-10.
  7. D. Borthakur, J. Sarma, and J. Gray, "Apache Hadoop Goes Realtime at Facebook, " In Proc. the 2011 ACM SIGMOD Int. Conf. on Management of data, Athens, Greece, 2011, pp. 1071-1080.
  8. W. Ryu, "Flexible management of data nodes for hadoop distributed file system," In Proc. Int. Conf. on Big Data, Small Data, Linked Data and Open Data (ALLDATA 2017), Venice, Italy, 2017.
  9. T. White, "Hadoop: The definitive guide, 4th Edition," O'Reilly Media, Inc., 2015.