• The Journal of the Korea institute of electronic communication sciences
• /
• v.16 no.2
• /
• pp.287-292
• /
• 2021

In this paper, we propose an efficient CNN application method by analyzing the effect of activation function on the failure diagnosis of the inductive motor stator. Generally, the main purpose of the inductive motor stator failure diagnosis is to prevent the failure by rapidly diagnosing the minute turn short. In the application of activation function, experiments show that the Sigmoid function is 23.23% more useful in accuracy of diagnosis than the ReLu function, although it is shown that ReLu has superiority in overall fixer failure in utilizing the activation function.

• Journal of the Korea Academia-Industrial cooperation Society
• /
• v.18 no.10
• /
• pp.75-80
• /
• 2017

In this paper, we describe the effect of the regularization method and the network with identity mapping on the performance of the activation functions in deep convolutional neural networks. The activation functions act as nonlinear transformation. In early convolutional neural networks, a sigmoid function was used. To overcome the problem of the existing activation functions such as gradient vanishing, various activation functions were developed such as ReLU, Leaky ReLU, parametric ReLU, and ELU. To solve the overfitting problem, regularization methods such as dropout and batch normalization were developed on the sidelines of the activation functions. Additionally, data augmentation is usually applied to deep learning to avoid overfitting. The activation functions mentioned above have different characteristics, but the new regularization method and the network with identity mapping were validated only using ReLU. Therefore, we have experimentally shown the effect of the regularization method and the network with identity mapping on the performance of the activation functions. Through this analysis, we have presented the tendency of the performance of activation functions according to regularization and identity mapping. These results will reduce the number of training trials to find the best activation function.

• Journal of Intelligence and Information Systems
• /
• v.10 no.2
• /
• pp.53-63
• /
• 2004

Recurrent-Cascade-Correlation(RCC) is a supervised teaming algorithm that automatically determines the size and topology of the network. RCC adds new hidden neurons one by one and creates a multi-layer structure in which each hidden layer has only one neuron. By second order RCC, new hidden neurons are added to only one hidden layer. These created neurons are not connected to each other. We present a generalization of the RCC Architecture by combining the standard RCC Architecture and the second order RCC Architecture. Whenever a hidden neuron has to be added, the new RCC teaming algorithm automatically determines whether the network topology grows vertically or horizontally. This new algorithm using sigmoid, tanh and new activation functions was tested with the morse-benchmark-problem. Therefore we recognized that the number of hidden neurons was decreased by the experiments of the RCC network generalization which used the activation functions.

• Proceedings of the Korean Society of Computer Information Conference
• /
• 2023.01a
• /
• pp.25-26
• /
• 2023

본 논문에서는 OpenAI Gym 환경에서 제공하는 CartPole-v1에 대해 강화학습을 통해 에이전트를 학습시키고, 학습에 적용되는 활성화함수의 성능을 비교분석하였다. 본 논문에서 적용한 활성화함수는 Sigmoid, ReLU, ReakyReLU 그리고 softplus 함수이며, 각 활성화함수를 DQN(Deep Q-Networks) 강화학습에 적용했을 때 보상 값을 비교하였다. 실험결과 ReLU 활성화함수를 적용하였을 때의 보상이 가장 높은 것을 알 수 있었다.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.17 no.10
• /
• pp.2788-2808
• /
• 2023

Brain tumors are one of the most threatening malignancies for humans. Misdiagnosis of brain tumors can result in false medical intervention, which ultimately reduces a patient's chance of survival. Manual identification and segmentation of brain tumors from Magnetic Resonance Imaging (MRI) scans can be difficult and error-prone because of the great range of tumor tissues that exist in various individuals and the similarity of normal tissues. To overcome this limitation, the Amended Convolutional Neural Network (ACNN) model has been introduced, a unique combination of three techniques that have not been previously explored for brain tumor detection. The three techniques integrated into the ACNN model are image tissue preprocessing using the Kalman Bucy Smoothing Filter to remove noisy pixels from the input, image tissue segmentation using the Isotonic Regressive Image Tissue Segmentation Process, and feature extraction using the Marr Wavelet Transformation. The extracted features are compared with the testing features using a sigmoid activation function in the output layer. The experimental findings show that the suggested model outperforms existing techniques concerning accuracy, precision, sensitivity, dice score, Jaccard index, specificity, Positive Predictive Value, Hausdorff distance, recall, and F1 score. The proposed ACNN model achieved a maximum accuracy of 98.8%, which is higher than other existing models, according to the experimental results.

• Annual Conference on Human and Language Technology
• /
• 2017.10a
• /
• pp.56-59
• /
• 2017

CNN(Convolutional Neural Network)을 이용하여 발화 주제 다중 분류 task를 multi-labeling 방법과, cluster 방법을 이용하여 수행하고, 각 방법론에 MSE(Mean Square Error), softmax cross-entropy, sigmoid cross-entropy를 적용하여 성능을 평가하였다. Network는 음절 단위로 tokenize하고, 품사정보를 각 token의 추가한 sequence와, Naver DB를 통하여 얻은 named entity 정보를 입력으로 사용한다. 실험결과 cluster 방법으로 문제를 변형하고, sigmoid를 output layer의 activation function으로 사용하고 cross entropy cost function을 이용하여 network를 학습시켰을 때 F1 0.9873으로 가장 좋은 성능을 보였다.

• Journal of Korea Water Resources Association
• /
• v.54 no.spc1
• /
• pp.1143-1154
• /
• 2021

The selection of activation function has a great influence on the groundwater level prediction performance of artificial neural network (ANN) model. In this study, five activation functions were applied to ANN model for two groundwater level observation wells in the middle mountainous area of the Pyoseon watershed in Jeju Island. The results of the prediction of the groundwater level were compared and analyzed, and the optimal activation function was derived. In addition, the results of LSTM model, which is a widely used recurrent neural network model, were compared and analyzed with the results of the ANN models with each activation function. As a result, ELU and Leaky ReLU functions were derived as the optimal activation functions for the prediction of the groundwater level for observation well with relatively large fluctuations in groundwater level and for observation well with relatively small fluctuations, respectively. On the other hand, sigmoid function had the lowest predictive performance among the five activation functions for training period, and produced inappropriate results in peak and lowest groundwater level prediction. The ANN-ELU and ANN-Leaky ReLU models showed groundwater level prediction performance comparable to that of the LSTM model, and thus had sufficient potential for application. The methods and results of this study can be usefully used in other studies.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.14 no.8
• /
• pp.3519-3533
• /
• 2020

Tuberculosis is a chronic and delayed infection which is easily experienced by young people. According to the statistics of the World Health Organization (WHO), there are nearly ten million fell ill with tuberculosis and a total of 1.5 million people died from tuberculosis in 2018 (including 251000 people with HIV). Tuberculosis is the largest single infectious pathogen that leads to death. In order to help doctors with tuberculosis diagnosis, we compare the tuberculosis classification abilities of six popular convolutional neural network (CNN) models in the same data set to find the best model. Before training, we optimize three parts of CNN to achieve better results. We employ sigmoid function to replace the step function as the activation function. What's more, we use binary cross entropy function as the cost function to replace traditional quadratic cost function. Finally, we choose stochastic gradient descent (SGD) as gradient descent algorithm. From the results of our experiments, we find that Densenet121 is most suitable for tuberculosis diagnosis and achieve a highest accuracy of 0.835. The optimization and expansion depend on the increase of data set and the improvements of Densenet121.

• Journal of the Institute of Electronics Engineers of Korea CI
• /
• v.38 no.6
• /
• pp.26-37
• /
• 2001

In order to improve the several problems of the general backpropagation, we propose a method using a fuzzy logic system for automatic tuning of the activation function gain in the backpropagation. First, we researched that the changing of the gain of sigmoid function is equivalent to changing the learning rate, the weights, and the biases. The inputs of the fuzzy logic system were the sensitivity of error respect to the last layer and the mean sensitivity of error respect to the hidden layer, and the output was the gain of the sigmoid function. In order to verify the effectiveness of the proposed method, we performed simulations on the parity problem, function approximation, and pattern recognition. The results show that the proposed method has considerably improved the performance compared to the general backpropagation.

• Journal of The Korean Society of Agricultural Engineers
• /
• v.63 no.1
• /
• pp.103-116
• /
• 2021

Analysis of runoff is substantial for effective water management in the watershed. Runoff occurs by reaction of a watershed to the rainfall and has non-linearity and uncertainty due to the complex relation of weather and watershed factors. ANN (Artificial Neural Network), which learns from the data, is one of the machine learning technique known as a proper model to interpret non-linear data. The performance of ANN is affected by the ANN's structure, the number of hidden layer nodes, learning rate, and activation function. Especially, the activation function has a role to deliver the information entered and decides the way of making output. Therefore, It is important to apply appropriate activation functions according to the problem to solve. In this paper, ANN models were constructed to estimate runoff with different activation functions and each model was compared and evaluated. Sigmoid, Hyperbolic tangent, ReLU (Rectified Linear Unit), ELU (Exponential Linear Unit) functions were applied to the hidden layer, and Identity, ReLU, Softplus functions applied to the output layer. The statistical parameters including coefficient of determination, NSE (Nash and Sutcliffe Efficiency), NSEln (modified NSE), and PBIAS (Percent BIAS) were utilized to evaluate the ANN models. From the result, applications of Hyperbolic tangent function and ELU function to the hidden layer and Identity function to the output layer show competent performance rather than other functions which demonstrated the function selection in the ANN structure can affect the performance of ANN.