Journal of KIISE:Computer Systems and Theory (한국정보과학회논문지:시스템및이론)

Korean Institute of Information Scientists and Engineers (한국정보과학회)
권호 표지

BeanFS: A Distributed File System for Large-scale E-mail Services

BeanFS: 대규모 이메일 서비스를 위한 분산 파일 시스템

  • Published : 2009.08.15
Download PDF
⟨ Previous Next ⟩

Abstract

Distributed file systems running on a cluster of inexpensive commodity hardware are being recognized as an effective solution to support the explosive growth of storage demand in large-scale Internet service companies. This paper presents the design and implementation of BeanFS, a distributed file system for large-scale e-mail services. BeanFS is adapted to e-mail services as follows. First, the volume-based replication scheme alleviates the metadata management overhead of the central metadata server in dealing with a very large number of small files. Second, BeanFS employs a light-weighted consistency maintenance protocol tailored to simple access patterns of e-mail message. Third, transient and permanent failures are treated separately and recovering from transient failures is done quickly and has less overhead.

저가의 하드웨어를 이용하는 분산 파일 시스템은 대용량의 저장 장치를 경제적으로 제공해주는 해법으로 많은 인터넷 서비스 업체에 의해 주목받고 있다. 본 논문에서는 대규모 이메일 서비스를 위한 분산 파일 시스템인 BeanFS의 설계와 구현에 대해 소개한다. BeanFS는 다음과 같이 이메일 서비스에 최적화되었다. 첫째, 이메일 서비스에서 이용되는 작고 많은 파일을 효과적으로 처리하기 위해서, 볼륨 기반의 복제 기법을 도입하여 중앙 서버의 병목현상을 완화시킨다. 둘째, 이메일 메시지의 단순한 접근 패턴을 고려하여 일관성 유지 기법을 경량화시킨다. 셋째, 재복제시에 발생하는 오버헤드를 줄이기 위해 일시적인 장애를 영구적인 장애와 분리하여 대처한다.

Keywords

References

  1. International Data Corporation (IDC), The diverse and exploding digital universe, 2007.
  2. S. Ghemawat, H. Gobioff, and S.-T. Leung, The google file system, In Proceeding of the 19th ACM Symposium on Operating Systems Principles (SOSP'03), pp.29-43, 2003.
  3. Amazon simple storage service (amazon s3), http://aws.amazon.com/s3.
  4. The hadoop opensource project, http://lucene.apache.org/hadoop/.
  5. S. A. Weil, S. A. Brandt, E. L. Miller, D. D. E. Long, and C. Maltzahn, Ceph: a scalable, highperformance distributed file system. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI'06), pp. 307-320, 2006.
  6. B. Liskov, S. Ghemawat, R. Gruber, P. Johnson, and L. Shrira, Replication in the harp file system, In Proceedings of the 13th ACM Symposium on Operating Systems Principles (SOSP'91), pp.226-238, 1991.
  7. G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels, Dynamo: amazon's highly available key-value store, In Proceedings of 21st ACMSymposium on Operating Systems Principles (SOSP'07), pp.205-220, 2007.
  8. S.-H. Kim, Y. Lee, and J.-S. Kim, Flexrpc: A flexible remote procedure call facility for modern cluster file systems, In Proceeding of 9th IEEE International Conference on Cluster Computing (CLUSTER'07), pp.257-284, 2007.
  9. J. H. Morris, M. Satyanarayanan, M. H. Conner, J. H. Howard, D. S. Rosenthal, and F. D. Smith, Andrew: a distributed personal computing environment, Communications of the ACM (CACM), 29(3):184-201, 1986. https://doi.org/10.1145/5666.5671
  10. J. H. Howard, M. L. Kazar, S. G. Menees, D. A. Nichols, M. Satyanarayanan, R. N. Sidebotham, and M. J. West, Scale and performance in a distributed file system, ACM Transactions on Computer System (TOCS), 6(1):51-81, 1988. https://doi.org/10.1145/35037.35059
  11. D. Skeen and M. Stonebraker, A formal model of crash recovery in a distributed system, pp.295-317, 1987.
  12. Mysql reference manual, http://dev.mysql.com/doc/.
  13. D. A. Patterson, G. Gibson, and R. H. Katz, A case for redundant arrays of inexpensive disks (raid), In Proceedings of the 1988 ACM SIGMOD International Conference on Management of Data (SIGMOD'88), pp.109-116, 1988.
  14. Y. Saito, B. N. Bershad, and H. M. Levy, Manageability, availability and performance in porcupine: a highly scalable, clusterbased mail service, In Proceedings of the 17th ACM Symposium on Operating Systems Principles (SOSP'99), pp.1-15, 1999.
  15. Cluster File Systems, Inc. Lustre: A Scalable, High-Performance File System, http://www.clusterfs.com.