Screening Vital Few Variables and Development of Logistic Regression Model on a Large Data Set

대용량 자료에서 핵심적인 소수의 변수들의 선별과 로지스틱 회귀 모형의 전개

  • Lim, Yong-B. (Department of Statistics, Ewha Womans) ;
  • Cho, J. (Department of Statistics, Ewha Womans) ;
  • Um, Kyung-A (Department of Statistics, Ewha Womans) ;
  • Lee, Sun-Ah (Department of Statistics, Ewha Womans)
  • Published : 2006.06.30

Abstract

In the advance of computer technology, it is possible to keep all the related informations for monitoring equipments in control and huge amount of real time manufacturing data in a data base. Thus, the statistical analysis of large data sets with hundreds of thousands observations and hundred of independent variables whose some of values are missing at many observations is needed even though it is a formidable computational task. A tree structured approach to classification is capable of screening important independent variables and their interactions. In a Six Sigma project handling large amount of manufacturing data, one of the goals is to screen vital few variables among trivial many variables. In this paper we have reviewed and summarized CART, C4.5 and CHAID algorithms and proposed a simple method of screening vital few variables by selecting common variables screened by all the three algorithms. Also how to develop a logistics regression model on a large data set is discussed and illustrated through a large finance data set collected by a credit bureau for th purpose of predicting the bankruptcy of the company.

Keywords

References

  1. 강현철 등(1999), '데이터마이닝, 방법론 및 활용', 자유아카데미
  2. 임용빈, 오만숙(2002), '분류와 회귀나무 분석에 관한 소고', '품질경영학회지', 30권, 1호, pp. 152-161
  3. 허명회, 이용구(2003), '데이터마이닝 모델링과 사례', SPSS 아카데미
  4. Abt, M., Lim, Y. B., Sacks, J., Xie, M., and Young, S.(2001), 'A sequential approach for identifying lead compounds in large chemical databases', Statistical Science, Vol. 16, No. 2, pp. 154-168 https://doi.org/10.1214/ss/1009213288
  5. Breiman, L., Friedman, J. H., Olshen, R. A., and Stone, C. J.(1984), Classification and regression trees, Chapman and Hall, Belmont, CA, Wadsworth
  6. Kass, G.(1980), 'An exploratory technique for investigating large quantities of categorical data', Applied Statistics, Vol. 29, pp, 119-127 https://doi.org/10.2307/2986296
  7. Quinlan, J. R.(1993), C4.5 Programs for machine learning, San Mateo: Morgan Kaufmann