Eager Data Transfer Mechanism for Reducing Communication Latency in User-Level Network Protocols

  • Won, Chul-Ho (Electrical and Computer Engineering Dept., California State University-Fresno) ;
  • Lee, Ben (School of Electrical Engineering and Computer Science, Oregon State University) ;
  • Park, Kyoung (Electronics and Telecommunications Research Institute) ;
  • Kim, Myung-Joon (Electronics and Telecommunications Research Institute)
  • Published : 2008.12.31


Clusters have become a popular alternative for building high-performance parallel computing systems. Today's high-performance system area network (SAN) protocols such as VIA and IBA significantly reduce user-to-user communication latency by implementing protocol stacks outside of operating system kernel. However, emerging parallel applications require a significant improvement in communication latency. Since the time required for transferring data between host memory and network interface (NI) make up a large portion of overall communication latency, the reduction of data transfer time is crucial for achieving low-latency communication. In this paper, Eager Data Transfer (EDT) mechanism is proposed to reduce the time for data transfers between the host and network interface. The EDT employs cache coherence interface hardware to directly transfer data between the host and NI. An EDT-based network interface was modeled and simulated on the Linux-based, complete system simulation environment, Linux/SimOS. Our simulation results show that the EDT approach significantly reduces the data transfer time compared to DMA-based approaches. The EDTbased NI attains 17% to 38% reduction in user-to-user message time compared to the cache-coherent DMA-based NIs for a range of message sizes (64 bytes${\sim}$4 Kbytes) in a SAN environment.


  1. Intel, Compaq and Microsoft Corporations, “Virtual Interface Architecture Specification, Version 1.0,” December 1997. Available at
  2. M. Banikazemi et al., “Design Alternatives for Virtual Interface Architecture (VIA) and an Implementation on IBM Netfinity NT Clusters,” Proc. of the Int'l Parallel and Distributed Processing Symposium, May 2000
  3. NERSC, “M-VIA: A High Performance Modular VIA for Linux,” Available from
  4. Infiniband Trade Association, “Infiniband Architecture Specification, Vol. 1,” InfiniBand Trade Association. Available from
  5. S.S. Mukherjee and M.D. Hill, “Making Network Interfaces Less Peripheral,” IEEE Computer, 31(10):70-76, October 1998
  6. S.S. Mukherjee et al., “Coherent network Interfaces for Fine-Grain Communication,” Proceedings of the 23rd International Symposium on Computer Architecture (ISCA), 1996
  7. S.S. Mukherjee et al., “The impact of Data Transfer and Buffering Alternatives on Network Interface Design,” Proceedings of the 4th International Symposium on High-Performance Computer Architecture (HPCA), Feb. 1998
  8. R.A.F. Bhoedjang, T. Ruhl, and H.E. Bal, “Design Issues for User-Level Network Interface Protocols on Myrinet,” IEEE Computer, 31(11):53-60, November 1998
  9. R.A.F. Bhoedjang et al., “Reducing Data and Control Transfer Overhead through Network-Interface Support,” First Myrinet User Group Conference (MUG), September 2000
  10. Won, C. et al., “Linux/SimOS - A Simulation Environment for Evaluating High-Speed Communication Systems,” Proceedings of the 2002 international Conference on Parallel Processing (ICPP), August 2002. An extended version of this paper appears in “Linux/SimOS: A Complete System simulation Environment for Evaluating High-Speed Communication Systems,” Journal of High Speed Networks, 2005
  11. H. Hellwagner, “Exploring the Performance of VI Architecture Communication Features in the Giganet Cluster LAN,” Proceedings of the International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA2000), 2000
  12. F. Briggs et al., “The Intel 870 Family of Enterprise Chipsets,” Proc. of the Hot Chips XIII, August 2001
  13. Intel Corporation, “Pentium(R) Processor Family Developer's Manual,” Available at intarch/manuals/241428.htm
  14. M. Rosenblum et al., “Using the SimOS Machine Simulator to study Complex Computer Systems,” ACM Transactions on Modeling and Computer Simulations, 7(1), January 1997
  15. J. R. Thorpe, “A Machine Independent DMA Framework for NetBSD,” USENIX 1998 Annual Technical Conference, June 15-19, 1998
  16. MIPS R1000 Microprocessor User's Manual, Version 2.0. Available from
  17. The PowerPC Architecture: A Specification for a New Family of RISC Processors, Edited by C. May, D. Silha, R. Simpson, and H. Warren, Morgan Kaufmann Publishers, Inc., 1994
  18. Pentium Processor Family Developer's Manual. Available from
  19. The Alchemy $Au1100^{TM}$ From AMD Internet Edge Processor Data Book. Available from
  20. Message Passing Interface Forum, “MPI: A Message Passing Interface Standard,” The International Journal of Supercomputer Applications and High Performance Computing, Vol. 8, 994
  21. MVICH: MPI for Virtual Interface Architecture, Berkeley Lab:
  22. T. Mehlan et al., “Providing a High-Performance VIA-Module for LAM/MPI,” Parallel Computing in Electrical Engineering, International Conference on (PARELEC'04), September 07 - 10, 2004
  23. M. Bertozzi, M. Panella, and M. Reggiani, “Design of a VIA based communication protocol for LAM/MPI Suite,” 9th Euromicro Workshop on Parallel Distributed Processing, Sept. 2001
  24. R. Dimitrov and A. Skjellum, “An efficient MPI implementation for Virtual Interface Architecture -- enabled cluster computing,” Proc. of the 3rd MPI developer's and user's conference, Atlanta, Georgia, March 1999
  25. MPICH-A Portable Implementation of MPI:
  26. Second Version of MPICH:
  27. MVAPICH: MPI for InfiniBand on VAPI Layer, Ohio State University:
  28. H. Tezuka et al., “Pin-down Cache: A Virtual Memory Management Technique for Zero-copy Communication,” 12th International Parallel Processing Symposium, Orlando, FL, March 1998
  29. J. Liu et al., “Design and Implementation of MPICH2 over InfiniBand with RDMA Support,” International Parallel and Distributed Processing Symposium (IPDPS 04), 2004
  30. R. Grabner, F. Mietke, and W. Rehm, “An MPICH2 Channel Device Implementation over VAPI on InfiniBand,” Proc. of CAC'04, Workshop on Communication Architecture for Clusters held in conjunction with IPDPS 2004, April 26-30 2004, Santa Fe, New Mexico
  31. N.J. Boden et al., “Myrinet: A gigabit-per-second local area network,” IEEE Micro, 15(1):29-36, February 1995

Cited by

  1. Evaluation of messaging middleware for high-performance cloud computing vol.17, pp.8, 2013,