• International Journal of Computer Science & Network Security
• /
• v.21 no.12spc
• /
• pp.549-555
• /
• 2021

Machine Learning(ML) splits data into 3 parts, which are usually 60% for training, 20% for validation, and 20% for testing. It just splits quantitatively instead of selecting each set of data by a criterion, which is very important concept for the adequacy of test data. ML measures a model's accuracy by applying a set of validation data, and revises the model until the validation accuracy reaches on a certain level. After the validation process, the complete model is tested with the set of test data, which are not seen by the model yet. If the set of test data covers the model's attributes well, the test accuracy will be close to the validation accuracy of the model. To make sure that ML's set of test data works adequately, we design an experiment and see if the test accuracy of model is always close to its validation adequacy as expected. The experiment builds 100 different SVM models for each of six data sets published in UCI ML repository. From the test accuracy and its validation accuracy of 600 cases, we find some unexpected cases, where the test accuracy is very different from its validation accuracy. Consequently, it is not always true that ML's set of test data is adequate to assure a model's quality.

• The KIPS Transactions:PartD
• /
• v.8D no.2
• /
• pp.175-180
• /
• 2001

This paper presents the optimal fraction of validation set to obtain a prediction accuracy of software failure count or failure time in the future by a neuro-fuzzy system. Given a fixed amount of training data, the most popular effective approach to avoiding underfitting and overfitting is early stopping, and hence getting optimal generalization. But there is unresolved practical issues : How many data do you assign to the training and validation set\ulcorner Rules of thumb abound, the solution is acquired by trial-and-error and we spend long time in this method. For the sake of optimal fraction of validation set, the variant specific fraction for the validation set be provided. It shows that minimal fraction of the validation data set is sufficient to achieve good next-step prediction. This result can be considered as a practical guideline in a prediction of software reliability by neuro-fuzzy system.

• Journal of Digital Contents Society
• /
• v.18 no.3
• /
• pp.525-534
• /
• 2017

The open set recognition method should be used for the cases that the classes of test data are not known completely in the training phase. So it is required to include two processes of classification and the validation test. This kind of research is very necessary for commercialization of face recognition modules, but few domestic researches results about it have been published. In this paper, we propose an open set face recognition method that includes two sequential validation phases. In the first phase, with dummy classes we perform classification based on sparse representation. Here, when the test data is classified into a dummy class, we conclude that the data is invalid. If the data is classified into one of the regular training classes, for second validation test we extract four features and apply them for the proposed decision function. In experiments, we proposed a simulation method for open set recognition and showed that the proposed validation test outperform SCI of the well-known validation method

• Korean Journal of Radiology
• /
• v.24 no.7
• /
• pp.626-639
• /
• 2023

Objective: To investigate the association of clinical, pathologic, and magnetic resonance imaging (MRI) variables with progressive disease (PD) during neoadjuvant chemotherapy (NAC) and distant metastasis-free survival (DMFS) in patients with triple-negative breast cancer (TNBC). Materials and Methods: This single-center retrospective study included 252 women with TNBC who underwent NAC between 2010 and 2019. Clinical, pathologic, and treatment data were collected. Two radiologists analyzed the pre-NAC MRI. After random allocation to the development and validation sets in a 2:1 ratio, we developed models to predict PD and DMFS using logistic regression and Cox proportional hazard regression, respectively, and validated them. Results: Among the 252 patients (age, 48.3 ± 10.7 years; 168 in the development set; 84 in the validation set), PD was occurred in 17 patients and 9 patients in the development and validation sets, respectively. In the clinical-pathologic-MRI model, the metaplastic histology (odds ratio [OR], 8.0; P = 0.032), Ki-67 index (OR, 1.02; P = 0.044), and subcutaneous edema (OR, 30.6; P = 0.004) were independently associated with PD in the development set. The clinical-pathologic-MRI model showed a higher area under the receiver-operating characteristic curve (AUC) than the clinical-pathologic model (AUC: 0.69 vs. 0.54; P = 0.017) for predicting PD in the validation set. Distant metastases occurred in 49 patients and 18 patients in the development and validation sets, respectively. Residual disease in both the breast and lymph nodes (hazard ratio [HR], 6.0; P = 0.005) and the presence of lymphovascular invasion (HR, 3.3; P < 0.001) were independently associated with DMFS. The model consisting of these pathologic variables showed a Harrell's C-index of 0.86 in the validation set. Conclusion: The clinical-pathologic-MRI model, which considered subcutaneous edema observed using MRI, performed better than the clinical-pathologic model for predicting PD. However, MRI did not independently contribute to the prediction of DMFS.

• The Transactions of the Korea Information Processing Society
• /
• v.7 no.4
• /
• pp.1112-1121
• /
• 2000

Design metrics that quantify the design phase play an important role in reducing system development cost because the problems in early phases of software development seriously affected the quality of the late products. Real-time systems such as telecommunication systems are so large that design quantification is more important in real-time system design. Although many metrics have been proposed, few of them are correctly validated. This paper revises the SDL metrics set proposed in earlier study [9] and perform an analytical validation o the metrics set. Axiomatic approach and dimensional analysis are used for metrics validation and the old metrics are revised ot satisfy the validation criteria.

• YAKHAK HOEJI
• /
• v.48 no.1
• /
• pp.60-64
• /
• 2004

Near-infrared (NIR) spectroscopy was used to determine rapidly and nondestructively the content of ambroxol in intact ambroxol tablets containing 30 mg (12.5％ m/m nominal concentration) by collecting NIR spectra in range 1100-1750 nm. The laboratory-made samples had 10.3∼15.9％ m/m nominal ambroxol concentration. The measurements were made by reflection using a fiber-optic probe and calibration was carried out by partial least square regression (PLSR) with autoscaling. Model validation was performed by randomly splitting the data set into calibration and validation data set (7 samples as a calibration data set and 5 samples as a validation data set). The developed NIR method gave results comparable to the known values of tablets in a laboratorial manufacturing Process, standard error of calibration (SEC) and standard error of prediction (SEP) being 0.49％ and 0.49％ m/m respectively. The method showed good accuracy and repeatability NIR spectroscopic determination in intact tablets allowed the potential use of real time monitoring for a running production process.

• Journal of the Korea Institute of Military Science and Technology
• /
• v.17 no.1
• /
• pp.108-114
• /
• 2014

This paper describes a method to set an intensity-based initial validation gate for tracking filter while preserves the ability of tracking a target with maximum speed. First, we collected real data set of signal versus distance of an airplane target. And at each data point, we computed maximum distance the target can move. And a function is modeled to expect the maximum moving pixels on the lateral direction based on the intensity of the detected target in IR image sequence. The initial prediction error covariance can be computed using this function to decide the size of the initial validation gate. The simulation results show the proposed method can set the appropriate initial validation gates to track the targets with the maximum speed.

• Journal of Intelligence and Information Systems
• /
• v.5 no.1
• /
• pp.95-101
• /
• 1999

In this paper, a comparison between different methods to combine predictions from neural networks will be given. These methods are bagging, bumping, and balancing. Those are based on the analysis of the ensemble generalization error into an ambiguity term and a term incorporating generalization performances of individual networks. Neural Networks and AI machine learning models are prone to overfitting. A strategy to prevent a neural network from overfitting, is to stop training in early stage of the learning process. The complete data set is spilt up into a training set and a validation set. Training is stopped when the error on the validation set starts increasing. The stability of the networks is highly dependent on the division in training and validation set, and also on the random initial weights and the chosen minimization procedure. This causes early stopped networks to be rather unstable: a small change in the data or different initial conditions can produce large changes in the prediction. Therefore, it is advisable to apply the same procedure several times starting from different initial weights. This technique is often referred to as training ensembles of neural networks. In this paper, we presented a comparison of three statistical methods to prevent overfitting of neural network.

• Proceedings of the Korean Statistical Society Conference
• /
• 2004.11a
• /
• pp.195-198
• /
• 2004

One Important problem in a cancer microarray study is to identify a set of genes from which a molecular prognostic indicator can be developed. In parallel with this problem is to validate the chosen set of genes. We develop in this note a K-fold cross validation procedure by combining a 'pre-validation' technique and a bootstrap resampling procedure in the Cox regression . The pre-validation technique predicts the microarray predictor of a case without having seen the true class level of the case. It was suggested by Tibshirani and Efron (2002) to avoid the possible over-fitting in the regression in which a microarray based predictor is employed. The bootstrap resampling procedure for the Cox regression was proposed by Sauerbrei and Schumacher (1992) as a means of overcoming the instability of a stepwise selection procedure. We apply this K-fold cross validation to the microarray data of 92 gastric cancers of which the experiment was conducted at Cancer Metastasis Research Center, Yonsei University. We also share some of our experience on the 'false positive' result due to the information leak.

• Korean Journal of Radiology
• /
• v.21 no.6
• /
• pp.670-683
• /
• 2020

Objective: The presence of coagulative necrosis (CN) in clear cell renal cell carcinoma (ccRCC) indicates a poor prognosis, while the absence of CN indicates a good prognosis. The purpose of this study was to build and validate a radiomics signature based on preoperative CT imaging data to estimate CN status in ccRCC. Materials and Methods: Altogether, 105 patients with pathologically confirmed ccRCC were retrospectively enrolled in this study and then divided into training (n = 72) and validation (n = 33) sets. Thereafter, 385 radiomics features were extracted from the three-dimensional volumes of interest of each tumor, and 10 traditional features were assessed by two experienced radiologists using triple-phase CT-enhanced images. A multivariate logistic regression algorithm was used to build the radiomics score and traditional predictors in the training set, and their performance was assessed and then tested in the validation set. The radiomics signature to distinguish CN status was then developed by incorporating the radiomics score and the selected traditional predictors. The receiver operating characteristic (ROC) curve was plotted to evaluate the predictive performance. Results: The area under the ROC curve (AUC) of the radiomics score, which consisted of 7 radiomics features, was 0.855 in the training set and 0.885 in the validation set. The AUC of the traditional predictor, which consisted of 2 traditional features, was 0.843 in the training set and 0.858 in the validation set. The radiomics signature showed the best performance with an AUC of 0.942 in the training set, which was then confirmed with an AUC of 0.969 in the validation set. Conclusion: The CT-based radiomics signature that incorporated radiomics and traditional features has the potential to be used as a non-invasive tool for preoperative prediction of CN in ccRCC.