The Korean Journal of Applied Statistics (응용통계연구)

The Korean Statistical Society (한국통계학회)
권호 표지
DOI QR코드

DOI QR Code

Current trends in high dimensional massive data analysis

고차원 대용량 자료분석의 현재 동향

  • Jang, Woncheol (Department of Statistics, Seoul National University) ;
  • Kim, Gwangsu (Data Science for Knowledge Creation Research Center, Seoul National University) ;
  • Kim, Joungyoun (Department of Information Statistics, Chungbuk National University)
  • 장원철 (서울대학교 통계학과) ;
  • 김광수 (서울대학교 데이터과학및 지식창출 연구센터) ;
  • 김정연 (충북대학교 정보통계학과)
  • Received : 2016.10.19
  • Accepted : 2016.10.19
  • Published : 2016.10.31
Download PDF
⟨ Previous Next ⟩

Abstract

The advent of big data brings the opportunity to answer many open scientic questions but also presents some interesting challenges. Main features of contemporary datasets are the high dimensionality and massive sample size. In this paper, we give an overview of major challenges caused by these two features: (1) noise accumulation and spurious correlations in high dimensional data; (ii) computational scalability for massive data. We also provide applications of big data in various fields including forecast of disasters, digital humanities and sabermetrics.

빅 데이터의 출현은 여러가지 과학적 난제에 대답 할 수 있는 기회를 제공하지만 흥미로운 도전을 또한 제공한다. 이러한 빅데이터의 주요 특징으로 "고차원"과 "대용량"을 들 수가 있다. 본 논문은 이러한 두 가지 특징에 동반되는 다음과 같은 도전문제에 대한 개요를 제시한다 : (1) 고차원 자료에서의 소음 축적과 위 상관 관계; (ii) 대용량 자료분석을 위한 계산 확장성. 또한 본 논문에서는 재난예측, 디지털 인문학과 세이버메트릭스 등 다양한 분야에서 빅 데이터의 다양한 응용사례를 제공한다.

Keywords

References

  1. Bak, J. Y. and Oh, A. (2015). Five centuries of monarchy in Korea: mining the text of the annals of the Joseon dynasty. In Proceedings of the 9th SIGHUM workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities at the 53rd Annual Meeting of the Association for Computational Linguistics, 10-14.
  2. Clauset, A., Shalizi, C. R., and Newman, M. E. J. (2009). Power-law distributions in empirical data. SIAM Review, 51, 661-703. https://doi.org/10.1137/070710111
  3. Clauset, A. and Woodard, R. (2013). Estimating the historical and future probabilities of large terrorist events. Annals of Applied Statistics, 7, 1838-1865. https://doi.org/10.1214/12-AOAS614
  4. Fan, J., Han, F. and Liu, H. (2014). Challenges of big data analysis. National Science Review, 1, 293-314. https://doi.org/10.1093/nsr/nwt032
  5. Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96 1348-1360. https://doi.org/10.1198/016214501753382273
  6. Fan, J. and Lv, J. (2008). Sure independence screening for ultrahigh dimensional feature space (with discussion). Journal of the Royal Statistical Society Series B, 70, 849-911. https://doi.org/10.1111/j.1467-9868.2008.00674.x
  7. Franke, B., Plante, J.-F., Roscher, R., Lee, E.-S. A., Smyth, C., Hatefi, A., Chen, F., Gil, E., Schwing, A., Selvitella, A., Hoffman, M. M., Grosse, R., Hendricks, D., and Reid, N. (2016). Statistical inference, learning and models in big data. International Statistical Review, To appear.
  8. Kim, D., Son, S.-W., and Jeong, H. (2014). Large-scale quantitative analysis of painting arts. Scientific Reports, 4, 7370. https://doi.org/10.1038/srep07370
  9. Ko, S. and Won, J.-H. (2016). Processing large-scale data with Apache Spark, The Korean Journal of Applied Statistics, To appear.
  10. Kulldorff, M. (1997). A spatial scan statistic. Communications in Statistics: Theory and Methods, 26, 1481-1496. https://doi.org/10.1080/03610929708831995
  11. Lee, B., Kim, D., Kim, D., and Jeong, H. (2016). N-gram web service and stylometric analysis of Korean historical documents. New Physics: Sae Mulli, 66, 502-510. https://doi.org/10.3938/NPSM.66.502
  12. Park, J.-H., Lee, S.-Y., Kang, D. H. and Won, J.-H. (2013). Hadoop and Mapreduce. Journal of the Korean Data & Information Science Society, 24, 1013-1027. https://doi.org/10.7465/jkdi.2013.24.5.1013
  13. Sawchik, T. (2015). Big Data Baseball: Math, Miracles, and the End of a 20 Year Losing Streak, Flatiron Books.
  14. Sievert, C. (2015). Tools for harnessing 'MLBAM', 'Gameday' data and visualizing 'pictchfx', http://cpsievert.github.com/pitchRx
  15. Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B, 58, 267-288.
  16. van de Geer, S. Buhlmann, P. Ritov, Y. A., and Dezeure, R. (2014). On asymptotically optimal confidence regions and test for high-dimensional models. Annals of Statistics, 42, 1166-1202. https://doi.org/10.1214/14-AOS1221
  17. Witten, R. and Candes, E. (2013). Randomized algorithms for low-rank matrix factorizations: sharp performance bounds. Algorithmica, 63, 355-363.
  18. Zhang, C.-H. (2010.) Nearly unbiased variable selection under minimax concave penalty. Annals of Statistics, 38, 894-942. https://doi.org/10.1214/09-AOS729