• The Korean Journal of Applied Statistics
• /
• v.23 no.6
• /
• pp.1169-1178
• /
• 2010

This article consider a performance model selection based on AIC under unbalanced deign in linear mixed effect models. Vaida and Balanchard (2005) proposed conditional AIC for model selection in linear mixed effect models when the prediction of random effects is of primary interest. Theoretical properties of cAIC and related criteria have been investigated by Liang et al. (2008) and Greven and Kneib (2010). However, all of the simulation studies were performed under a balanced design. Even though functional form of AIC remain same even under the unbalanced deign, it is worthwhile to investigate performance of AIC based model selection criteria under the unbalanced design. The simulation study in this article shows how unbalancedness affects model selection in linear mixed effect models.

• International Area Studies Review
• /
• v.13 no.3
• /
• pp.73-91
• /
• 2009

In this paper, we attempt to study the trend of regional disparity among Chinese provinces and examine the effects of globalization on the disparities adapting panel data approach. The panel data set utilized consists of the annual variables of 29 provinces during 18 years from 1990 to 2007. The trend of inter-provincial disparities in the 1990s with the expansive trend but the trend has started to decrease since 2000. The results of the China case study show clearly that the provincial international trade level and ratio variables perform on regional income disparities remarkably in all cases. It means that the large development of international trade do with increased among provincial disparity. While due to the large area in the provinces, there exist urban-rural disparities within provinces could be one of the main source of regional disparities. Therefore, along with western regions development policy various development policies against small cities are necessary for balanced regional economic growth in China.

• The KIPS Transactions:PartD
• /
• v.9D no.5
• /
• pp.755-768
• /
• 2002

We investigate the effect of the data skew of join attributes on the performance of a pipelined multi-way hash join method, and propose two new hash join methods with load balancing capabilities. The first proposed method allocates buckets statically by round-robin fashion, and the second one allocates buckets adaptively via a frequency distribution. Using hash-based joins, multiple joins can be pipelined so that the early results from a join, before the whole join is completed, are sent to the next join processing without staying on disks. Unless the pipelining execution of multiple hash joins includes some load balancing mechanisms, the skew effect can severely deteriorate system performance. In this paper, we derive an execution model of the pipeline segment and a cost model, and develop a simulator for the study. As shown by our simulation with a wide range of parameters, join selectivities and sizes of relations deteriorate the system performance as the degree of data skew is larger. But the proposed method using a large number of buckets and a tuning technique can offer substantial robustness against a wide range of skew conditions.

• Journal of KIISE:Databases
• /
• v.29 no.3
• /
• pp.180-192
• /
• 2002

We investigate the effect of the data skew of join attributes on the performance of a pipelined multi-way hash join method, and propose two new harsh join methods in the shared-nothing multiprocessor environment. The first proposed method allocates buckets statically by round-robin fashion, and the second one allocates buckets dynamically via a frequency distribution. Using harsh-based joins, multiple joins can be pipelined to that the early results from a join, before the whole join is completed, are sent to the next join processing without staying in disks. Shared nothing multiprocessor architecture is known to be more scalable to support very large databases. However, this hardware structure is very sensitive to the data skew. Unless the pipelining execution of multiple hash joins includes some dynamic load balancing mechanism, the skew effect can severely deteriorate the system performance. In this parer, we derive an execution model of the pipeline segment and a cost model, and develop a simulator for the study. As shown by our simulation with a wide range of parameters, join selectivities and sizes of relations deteriorate the system performance as the degree of data skew is larger. But the proposed method using a large number of buckets and a tuning technique can offer substantial robustness against a wide range of skew conditions.

• Geophysics and Geophysical Exploration
• /
• v.23 no.3
• /
• pp.131-140
• /
• 2020

In training a classification model using machine learning, the acquisition of training data is a very important stage, because the amount and quality of the training data greatly influence the model performance. However, when the cost of obtaining data is so high that it is difficult to build ideal training data, the number of samples for each class may be acquired very differently, and a serious data-imbalance problem can occur. If such a problem occurs in the training data, all classes are not trained equally, and classes containing relatively few data will have significantly lower recall values. Additionally, the reliability of evaluation indices such as accuracy and precision will be reduced. Therefore, this study sought to overcome the problem of data imbalance in two stages. First, we introduced weighted accuracy and weighted precision as new evaluation indices that can take into account a data-imbalance ratio by modifying conventional measures of accuracy and precision. Next, oversampling was performed to balance weighted precision and recall among classes. We verified the algorithm by applying it to the problem of facies classification. As a result, the imbalance between majority and minority classes was greatly mitigated, and the boundaries between classes could be more clearly identified.

• The Korean Journal of Applied Statistics
• /
• v.34 no.4
• /
• pp.599-609
• /
• 2021

Data with a large difference in the number of objects between clusters are called unbalanced data. In discriminant analysis of unbalanced data, it is more important to classify objects in minority categories than to classify objects in majority categories well. However, objects in minority categories are often misclassified into majority categories. In this study, we propose a method that combined hierarchical DBSCAN (HDBSCAN) and SMOTE to solve this problem. Using HDBSCAN, it removes noise in minority categories and majority categories. Then it applies SMOTE to create new data. Area under the roc curve (AUC) and F1 scores were used to compare performance with existing methods. As a result, in most cases, the method combining HDBSCAN and synthetic minority oversampling technique (SMOTE) showed a high performance index, and it was found to be an excellent method for classifying unbalanced data.

• The Korean Journal of Applied Statistics
• /
• v.28 no.1
• /
• pp.9-21
• /
• 2015

The support vector machine has been successfully applied to various classification areas due to its flexibility and a high level of classification accuracy. However, when analyzing imbalanced data with uneven class sizes, the classification accuracy of SVM may drop significantly in predicting minority class because the SVM classifiers are undesirably biased toward the majority class. The weighted $L_2$-norm SVM was developed for the analysis of imbalanced data; however, it cannot identify irrelevant input variables due to the characteristics of the ridge penalty. Therefore, we propose the weighted $L_1$-norm SVM, which uses lasso penalty to select important input variables and weights to differentiate the misclassification of data points between classes. We demonstrate the satisfactory performance of the proposed method through simulation studies and a real data analysis.

• The Korean Journal of Applied Statistics
• /
• v.37 no.4
• /
• pp.455 -465
• /
• 2024

SMOTE (synthetic minority over-sampling technique) has been used the most as a solution to the problem of imbalanced data. SMOTE selects the nearest neighbor based on Euclidean distance. However, Euclidean distance has the disadvantage of not considering the correlation between variables. In particular, the Mahalanobis distance has the advantage of considering the covariance of variables. But if there are outliers, they usually influence calculating the Mahalanobis distance. To solve this problem, we use the Mahalanobis distance by estimating the covariance matrix using MCD (minimum covariance determinant). Then apply Mahalanobis distance based on MCD to SMOTE to create new data. Therefore, we showed that in most cases this method provided high performance indicators for classifying imbalanced data.

• The Korean Journal of Applied Statistics
• /
• v.34 no.5
• /
• pp.711-721
• /
• 2021

In this paper, we propose the method of adjusting thresholds using impurity indices in classification analysis on imbalanced data. Suppose the minority category is Positive and the majority category is Negative for the imbalanced binomial data. When categories are determined based on the commonly used 0.5 basis, the specificity tends to be high in unbalanced data while the sensitivity is relatively low. Increasing sensitivity is important when proper classification of objects in minority categories is relatively important. We explore how to increase sensitivity through adjusting thresholds. Existing studies have adjusted thresholds based on measures such as G-Mean and F1-score, but in this paper, we propose a method to select optimal thresholds using the chi-square statistic of CHAID, the Gini index of CART, and the entropy of C4.5. We also introduce how to get a possible unique value when multiple optimal thresholds are obtained. Empirical analysis shows what improvements have been made compared to the results based on 0.5 through classification performance metrics.

• Proceedings of the Korean Statistical Society Conference
• /
• 2000.11a
• /
• pp.165-171
• /
• 2000

일원분류 모형에서 표준 F-검정을 하기 위해서는 오차항에 대한 등분산성을 가정한다. 그러나 실제로 이러한 가정은 지켜지기 힘들며, 이에 더불어 관찰치가 각 집단별로 일정하지 않고 불균형한 경우에는 F-검정의 유의수준이 지정된 값을 만족시키지 못하며, 따라서 검정력에 관한 분석은 의미가 없게 된다. 본 연구에서는 등분산성이 지켜지지 않고, 자료가 불균형한 경우, 현실적인 상황에서 일반적으로 사용되는 F-검정의 유의수준 유지라는 문제가 어 떤 변화를 겪게 되는지를 확인하고, 더 나아가 유의수준을 유지하기 위해서는 어떤 식의 조정이 가능한지를 살펴보았다.