Journal of Information Processing Systems

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

Shared Memory Model over a Switchless PCIe NTB Interconnect Network

  • Lim, Seung-Ho (Division of Computer Engineering, Hankuk University of Foreign Studies) ;
  • Cha, Kwangho (Division of Supercomputing, Korea Institute of Science and Technology Information)
  • Received : 2021.04.05
  • Accepted : 2021.10.18
  • Published : 2022.02.28
Download PDF
⟨ Previous Next ⟩

Abstract

The role of the interconnect network, which connects computing nodes to each other, is important in high-performance computing (HPC) systems. In recent years, the peripheral component interconnect express (PCIe) has become a promising interface as an interconnection network for high-performance and cost-effective HPC systems having the features of non-transparent bridge (NTB) technologies. OpenSHMEM is a programming model for distributed shared memory that supports a partitioned global address space (PGAS). Currently, little work has been done to develop the OpenSHMEM library for PCIe-interconnected HPC systems. This paper introduces a prototype implementation of the OpenSHMEM library through a switchless interconnect network using PCIe NTB to provide a PGAS programming model. In particular, multi-interrupt, multi-thread-based data transfer over the OpenSHMEM shared memory model is applied at the implementation level to reduce the latency and increase the throughput of the switchless ring network system. The implemented OpenSHMEM programming model over the PCIe NTB switchless interconnection network provides a feasible, cost-effective HPC system with a PGAS programming model.

Keywords

Acknowledgement

This work was conducted as a subproject of Project No. K-19-SG-26-02R-1, supported by the Korea Institute of Science and Technology Information (KISTI) and supported by a National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (No. NRF-2019R1F1A1057503, NRF-2021R1F1A1048026), and the Hankuk University of Foreign Studies Research Fund.

References

  1. Top500.org, "Interconnect family statistics," 2018 [Online]. Available: http://top500.org/statistics/list.
  2. PCI-SIG, "Peripheral Component Interconnect Special Interest Group," [Online]. Available: https://pcisig.com/.
  3. Y. W. Kim, Y. Ren, and W. Choi, "Design and implementation of an alternate system interconnect based on PCI express," Journal of the Institute of Electronics and Information Engineers, vol. 52, no. 8, pp. 74-85, 2015. https://doi.org/10.5573/IEIE.2015.52.8.074
  4. L. Mohrmann, J. Tongen, M. Friedman, and M. Wetzel, "Creating multicomputer test systems using PCI and PCI Express," in Proceedings of 2009 IEEE International Automatic Testing Conference (AUTOTESTCON), Anaheim, CA, 2009, pp. 7-10.
  5. A. Richter, C. Herber, T. Wild, and A. Herkersdorf, "Resolving performance interference in SR-IOV setups with PCIe Quality-of-Service extensions," in Proceedings of 2016 Euromicro Conference on Digital System Design (DSD), Limassol, Cyprus, 2016, pp. 454-462.
  6. J. Liu, A. Mamidala, A. Vishnu, and D. K. Panda, "Evaluating infiniband performance with PCI express," IEEE Micro, vol. 25, no. 1, pp. 20-29, 2005. https://doi.org/10.1109/MM.2005.9
  7. H. Wong, PCI express multi-root switch reconfiguration during system operation," Ph.D. dissertation, Massachusetts Institute of Technology, Cambridge, MA, 2011.
  8. V. Krishnan, "Towards an integrated IO and clustering solution using PCI express," in Proceedings of 2007 IEEE International Conference on Cluster Computing, Austin, TX, 2007, pp. 259-266.
  9. K. Kong, "Non-transparent Bridging with IDT 89HPES32NT24G2 PCI Express NTB Switch," 2019 [Online]. Available: https://www.renesas.com/kr/en/document/apn/724-non-transparent-bridging-idt-pes32nt24g2-pcie-switch?language=en.
  10. W. C. C. Tu and T. C. Chiueh, "Seamless fail-over for PCIe switched networks," in Proceedings of the 11th ACM International Systems and Storage Conference, Haifa, Israel, 2018, pp. 101-111).
  11. T. Stitt, "An introduction to the Partitioned Global Address Space (PGAS) programming model," 2009 [Online]. Available: https://cnx.org/contents/gtg1AzdI@7/An-Introduction-to-the-Partitioned-Global-Address-Space-PGAS-Programming-Model.
  12. B. Chapman, T. Curtis, S. Pophale, S. Poole, J. Kuehn, C. Koelbel, and L. Smith, "Introducing OpenSHMEM: SHMEM for the PGAS community," in Proceedings of the 4th Conference on Partitioned Global Address Space Programming Model, New York, NY, 2010, pp. 1-3.
  13. J. R. Hammond, S. Ghosh, and B. M. Chapman, "Implementing OpenSHMEM using MPI-3 one-sided communication," in OpenSHMEM and Related Technologies. Cham, Switzerland: Springer, 2014, pp. 44-58.
  14. S. Pophale, R. Nanjegowda, T. Curtis, B. Chapman, H. Jin, S. Poole, and J. Kuehn, "OpenSHMEM performance and potential: a NPB experimental study," in Proceedings of the 6th Conference on Partitioned Global Address Space Programming Models (PGAS), Santa Barbara, CA, 2012,
  15. R. E. Kessler and J. L. Schwarzmeier, "CRAY T3D: a new dimension for Cray Research," in Proceedings of COMPCON Spring Digest of Papers, San Francisco, CA, 1993, pp. 176-182.
  16. Broadcom, "PEX 8749," 2011 [Online]. Available: https://www.broadcom.com/products/pcie-switches-bridges/pcie-switches/pex8749.
  17. M. Choi and J. H. Park, "Feasibility and performance analysis of RDMA transfer through PCI Express," Journal of Information Processing Systems, vol. 13, no. 1, pp. 95-103, 2017. https://doi.org/10.3745/JIPS.01.0013
  18. L. Rota, M. Caselle, S. Chilingaryan, A. Kopmann, and M. Weber, "A new DMA PCIe architecture for Gigabyte data transmission," in Proceedings of 2014 19th IEEE-NPSS Real Time Conference, Nara, Japan, 2014, pp. 1-2.
  19. S. H. Lim, K. W. Park, and K. Cha, "Developing an OpenSHMEM model over a switchless PCIe non-transparent bridge interface," in Proceedings of 2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Rio de Janeiro, Brazil, 2019, pp. 593-602.
  20. P. Onufryk, "PCIe Networked Flash Storage," 2018 [Online]. Available: https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2018/20180809_FNET-301B-1_Onufryk.pdf.
  21. M. J. Sullivan, "Intel Xeon Processor C5500/C3500 Series Non-Transparent Bridge," 2010 [Online]. Available: https://www.yumpu.com/en/document/view/34121833/xeon-processor-c5500-c3500-series-non-transparent-bridge-intel.
  22. J. M. Mellor-Crummey and M. L. Scott, "Algorithms for scalable synchronization on shared-memory multiprocessors," ACM Transactions on Computer Systems, vol. 9, no. 1, pp. 21-65, 1991. https://doi.org/10.1145/103727.103729
  23. C. Shim, K. H. Cha, and M. Choi, "Design and implementation of initial OpenSHMEM on PCIe NTB based cloud computing," Cluster Computing, vol. 22, no. 1, pp. 1815-1826, 2019. https://doi.org/10.1007/s10586-018-1707-0