KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

Job-aware Network Scheduling for Hadoop Cluster

  • Liu, Wen (School of Computer Science and Technology, Dalian University of Technology) ;
  • Wang, Zhigang (School of Computer Science and Technology, Dalian University of Technology) ;
  • Shen, Yanming (School of Computer Science and Technology, Dalian University of Technology)
  • Received : 2016.07.04
  • Accepted : 2016.11.22
  • Published : 2017.01.31
Download PDF
⟨ Previous Next ⟩

Abstract

In recent years, data centers have become the core infrastructure to deal with big data processing. For these big data applications, network transmission has become one of the most important factors affecting the performance. In order to improve network utilization and reduce job completion time, in this paper, by real-time monitoring from the application layer, we propose job-aware priority scheduling. Our approach takes the correlations of flows in the same job into account, and flows in the same job are assigned the same priority. Therefore, we expect that flows in the same job finish their transmissions at about the same time, avoiding lagging flows. To achieve load balancing, two approaches (Flow-based and Spray) using ECMP (Equal-Cost multi-path routing) are presented. We implemented our scheme using NS-2 simulator. In our evaluations, we emulate real network environment by setting background traffic, scheduling delay and link failures. The experimental results show that our approach can enhance the Hadoop job execution efficiency of the shuffle stage, significantly reduce the network transmission time of the highest priority job.

Keywords

References

  1. Dean J, Ghemawat S, "MapReduce: simplified data processing on large clusters," Communications of the ACM, vol. 51, no. 1, pp. 107-113, January, 2008. https://doi.org/10.1145/1327452.1327492
  2. Morton K, Balazinska M, Grossman D, "ParaTimer: a progress indicator for MapReduce DAGs," in Proc. of 29th ACM SIGMOD International Conference on Management of data, pp.507-518, June 6-11, 2010.
  3. Ferreira Cordeiro R L, Traina Junior C, Machado Traina A J, et al, "Clustering very large multi-dimensional datasets with mapreduce," in Proc of 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pp.690-698, August 21-24, 2011.
  4. Ghemawat, S., H. Gobioff, and S. Leung, "File and storage systems: The Google File System," in Proc. of 19th ACM Symposium on Operating Systems Principles Bolton Landing, pp.29-43, October 19-22, 2003.
  5. Chang F, Dean J, Ghemawat S, et al, "Bigtable: a distributed storage system for structured data," Acm Transactions on Computer Systems, vol. 26, no. 2, pp. 205-218, June, 2008.
  6. Chowdhury M, Zaharia M, Ma J, et al, "Managing data transfers in computer clusters with orchestra," ACM SIGCOMM Computer Communication Review, vol. 41, no. 4, pp. 98-109, August, 2011.
  7. Al-Fares, Mohammad, et al, "Hedera: dynamic flow scheduling for data center networks," in Proc. of 7th USENIX Symposium on Networked Systems Design and Implementation, pp. 281-296, April 28-30, 2010.
  8. Farrington N, Porter G, Radhakrishnan S, et al, "Helios: a hybrid electrical/optical switch architecture for modular data centers," ACM SIGCOMM Computer Communication Review, vol. 41, no. 4, pp. 339-350, August, 2011.
  9. Curtis A R, Kim W, Yalagandula P, "Mahout: Low-overhead datacenter traffic management using end-host-based elephant detection," in Proc. of 30th IEEE International Conference on Computer Communications, pp.1629-1637, April 11-15, 2011.
  10. Alizadeh, M., Greenberg, A., Maltz, D. A., Padhye, J., Patel, P., & Prabhakar, B., et al, "Data center tcp (DCTCP)," ACM Sigcomm Computer Communication Review, vol. 40, no. 4, pp. 63-74, August, 2011.
  11. Dogar F R, Karagiannis T, Ballani H, et al, "Decentralized task-aware scheduling for data center networks," ACM SIGCOMM Computer Communication Review, vol. 44, no. 4, pp. 431-442, August, 2014.
  12. Das, A., Lumezanu, C., Zhang, Y., Singh, V., Jiang, G., & Yu, C, "Transparent and flexible network management for big data processing in the cloud," in Proc. of 5th USENIX Workshop on Hot Topics in Cloud Computing, HotCloud'13, pp. 37-52, June 25-26, 2013.
  13. Peng Y, Chen K, Wang G, et al, "Hadoopwatch: A first step towards comprehensive traffic forecasting in cloud computing," in Proc. of 33th IEEE International Conference on Computer Communications, pp. 19-27, April 27-May 2, 2014.
  14. Chowdhury M, Stoica I, Chowdhury M, et al, "Coflow: An Application Layer Abstraction for Cluster Networking," ACM Hotnets, pp. 1-6, August 7, 2012.
  15. Chowdhury M, Stoica I, " Coflow: A networking abstraction for cluster applications," in Proc. of 11th ACM Workshop on Hot Topics in Networks, pp. 31-36, October 29-30, 2012.
  16. Isard M, Budiu M, Yu Y, et al, "Dryad: distributed data-parallel programs from sequential building blocks," ACM SIGOPS Operating Systems Review, vol. 41, no. 3, pp. 59-72, March, 2007.
  17. Vamanan B, Hasan J, Vijaykumar T N, "Deadline-aware datacenter tcp (D2TCP)," ACM SIGCOMM Computer Communication Review, vol. 42, no. 4, pp. 115-126, August, 2012.
  18. Hong C Y, Caesar M, Godfrey P, "Finishing flows quickly with preemptive scheduling," ACM SIGCOMM Computer Communication Review, vol. 42, no. 4, pp. 127-138, August, 2012.
  19. Ford A, Raiciu C, Handley M, et al, "TCP Extensions for Multi-path Operation with Multiple Addresses: draft-ietf-mptcp-multiaddressed-03," Roke Manor, March 2011.
  20. Dixit A, Prakash P, Hu Y C, et al, "On the impact of packet spraying in data center networks," in Proc. of 32nd IEEE International Conference on Computer Communications, pp. 2130-2138, April 14-19, 2013.
  21. Benson T, Anand A, Akella A, et al, "MicroTE: Fine grained traffic engineering for data centers," in Proc. of 17th International Conference on Emerging Networking Experiments and Technologies, pp. 1-12, December 6-9, 2011.
  22. McKeown N, Anderson T, Balakrishnan H, et al, "OpenFlow: enabling innovation in campus networks," ACM SIGCOMM Computer Communication Review, vol. 38, no. 2, pp. 69-74, April, 2012.
  23. Jalaparti V, Bodik P, Kandula S, et al, "Speeding up distributed request-response workflows," ACM SIGCOMM Computer Communication Review, vol. 43, no. 4, pp. 219-230, August, 2013.
  24. Ananthanarayanan G, Kandula S, Greenberg A G, et al, "Reining in the Outliers in Map-Reduce Clusters using Mantri," in Proc. of 9th USENIX Symposium on Operating Systems Design and Implementation , pp. 24-31, October 4-6, 2010.
  25. Nishtala R, Fugal H, Grimm S, et al, "Scaling memcache at facebook," in Proc. of 10th USENIX Symposium on Networked Systems Design and Implementation, pp. 385-398, April 2-5, 2013.
  26. Al-Fares M, Loukissas A, Vahdat A, "A scalable, commodity data center network architecture," ACM SIGCOMM Computer Communication Review, vol. 38, no. 4, pp. 63-74, August, 2008.
  27. Niranjan Mysore R, Pamboris A, Farrington N, et al, "Portland: a scalable fault-tolerant layer 2 data center network fabric," ACM SIGCOMM Computer Communication Review, vol. 39, no. 4, pp. 39-50, August, 2009.