• Journal of the Korea Academia-Industrial cooperation Society
• /
• v.21 no.2
• /
• pp.189-194
• /
• 2020

This paper analyzes the gradient descent method, which is the one most used for learning neural networks. Learning means updating a parameter so the loss function is at its minimum. The loss function quantifies the difference between actual and predicted values. The gradient descent method uses the slope of the loss function to update the parameter to minimize error, and is currently used in libraries that provide the best deep learning algorithms. However, these algorithms are provided in the form of a black box, making it difficult to identify the advantages and disadvantages of various gradient descent methods. This paper analyzes the characteristics of the stochastic gradient descent method, the momentum method, the AdaGrad method, and the Adadelta method, which are currently used gradient descent methods. The experimental data used a modified National Institute of Standards and Technology (MNIST) data set that is widely used to verify neural networks. The hidden layer consists of two layers: the first with 500 neurons, and the second with 300. The activation function of the output layer is the softmax function, and the rectified linear unit function is used for the remaining input and hidden layers. The loss function uses cross-entropy error.

• Journal of the Korean Data and Information Science Society
• /
• v.28 no.6
• /
• pp.1245-1255
• /
• 2017

The online learning is a process of obtaining the solution for a given objective function where the data is accumulated in real time or in batch units. The stochastic gradient descent method is one of the most widely used for the online learning. This method is not only easy to implement, but also has good properties of the solution under the assumption that the generating model of data is homogeneous. However, the stochastic gradient method could severely mislead the online-learning when the homogeneity is actually violated. We assume that there are two heterogeneous generating models in the observation, and propose the a new stochastic gradient method that mitigate the problem of the heterogeneous models. We introduce a robust mini-batch optimization method using statistical tests and investigate the convergence radius of the solution in the proposed method. Moreover, the theoretical results are confirmed by the numerical simulations.

• Journal of the Korean Institute of Intelligent Systems
• /
• v.8 no.5
• /
• pp.98-106
• /
• 1998

This paper proposes an efficient learning algorithm for improving the training performance of the neural network. The proposed method improves the training performance by applying the backpropagation algorithm of a global optimization method which is a hybrid of a stochastic approximation and a conjugate gradient method. The approximate initial point for f a ~gtl obal optimization is estimated first by applying the stochastic approximation, and then the conjugate gradient method, which is the fast gradient descent method, is applied for a high speed optimization. The proposed method has been applied to the parity checking and the pattern classification, and the simulation results show that the performance of the proposed method is superior to those of the conventional backpropagation and the backpropagation algorithm which is a hyhrid of the stochastic approximation and steepest descent method.

• 제어로봇시스템학회:학술대회논문집
• /
• 1992.10a
• /
• pp.823-828
• /
• 1992

A novel steepest descent adaptive filter algorithm, which uses the instantaneous stochastic gradient for the steepest descent direction, is derived from a newly devised performance index function. The performance function for the new algorithm is improved from that for the LMS in consideration that the stochastic steepest descent method is utilized to minimize the performance index iterativly. Through mathematical analysis and computer simulations, it is verified that there are substantial improvements in convergence and misadjustments even though the computational simplicity and the robustness of the LMS algorithm are hardly sacrificed. On the other hand, the new algorithm can be interpreted as a variable step size adaptive filter, and in this respect a heuristic method is proposed in order to reduce the noise caused by the step size fluctuation.

• Bulletin of the Korean Mathematical Society
• /
• v.55 no.4
• /
• pp.1303-1315
• /
• 2018

For principal component analysis (PCA) to efficiently analyze large scale matrices, it is crucial to find a few singular vectors in cheaper computational cost and under lower memory requirement. To compute those in a fast and robust way, we propose a new stochastic method. Especially, we adopt the stochastic variance reduced gradient (SVRG) method [11] to avoid asymptotically slow convergence in stochastic gradient descent methods. For that purpose, we reformulate the PCA problem as a unconstrained optimization problem using a quadratic penalty. In general, increasing the penalty parameter to infinity is needed for the equivalence of the two problems. However, in this case, exact penalization is guaranteed by applying the analysis in [24]. We establish the convergence rate of the proposed method to a stationary point and numerical experiments illustrate the validity and efficiency of the proposed method.

• Journal of the Korean Data and Information Science Society
• /
• v.28 no.5
• /
• pp.1125-1132
• /
• 2017

In a situation where the severity and frequency of drought events getting stronger and higher, many studies related to drought forecast have been conducted to improve the drought forecast accuracy. However it is difficult to predict drought events using a single model because of nonlinear and complicated characteristics of temporal behavior of drought events. In this study, in order to overcome the shortcomings of the single model approach, we first build various single models capable to explain the relationship between the meteorological drought index, Standardized Precipitation Index (SPI), and other independent variables such as world climate indices. Then, we developed a combined models using Stochastic Gradient Descent method among Ensemble Learnings.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.16 no.1
• /
• pp.97-115
• /
• 2022

Smart power grid is a user friendly system that transforms the traditional electric grid to the one that operates in a co-operative and reliable manner. Demand Response (DR) is one of the important components of the smart grid. The DR programs enable the end user participation by which they can communicate with the electricity service provider and shape their daily energy consumption patterns and reduce their consumption costs. The increasing demands of electricity owing to growing population stresses the need for optimal usage of electricity and also to look out alternative and cheap renewable sources of electricity. The solar and wind energy are the promising sources of alternative energy at present because of renewable nature and low cost implementation. The proposed work models a smart home with renewable energy units. The random nature of the renewable sources like wind and solar energy brings an uncertainty to the model developed. A stochastic dual descent optimization method is used to bring optimality to the developed model. The proposed work is validated using the simulation results. From the results it is concluded that proposed work brings a balanced usage of the grid power and the renewable energy units. The work also optimizes the daily consumption pattern thereby reducing the consumption cost for the end users of electricity.

• International journal of advanced smart convergence
• /
• v.12 no.2
• /
• pp.56-66
• /
• 2023

The goal of deep learning is to extract complex features from multidimensional data use the features to create models that connect input and output. Deep learning is a process of learning nonlinear features and functions from complex data, and the user data that is employed to train deep learning models has become the focus of privacy concerns. Companies that collect user's sensitive personal information, such as users' images and voices, own this data for indefinite period of times. Users cannot delete their personal information, and they cannot limit the purposes for which the data is used. The study has designed a deep learning method that employs privacy protection technology that uses distributed collaborative learning so that multiple participants can use neural network models collaboratively without sharing the input datasets. To prevent direct leaks of personal information, participants are not shown the training datasets during the model training process, unlike traditional deep learning so that the personal information in the data can be protected. The study used a method that can selectively share subsets via an optimization algorithm that is based on modified distributed stochastic gradient descent, and the result showed that it was possible to learn with improved learning accuracy while protecting personal information.

• The Journal of the Institute of Internet, Broadcasting and Communication
• /
• v.21 no.1
• /
• pp.87-92
• /
• 2021

Bias, variance, error and learning are important factors for performance in modeling a big data based recommendation system. The recommendation model in this system must reduce complexity while maintaining the explanatory diagram. In addition, the sparsity of the dataset and the prediction of the system are more likely to be inversely proportional to each other. Therefore, a product recommendation model has been proposed through learning the similarity between products by using a factorization method of the sparsity of the dataset. In this paper, the generalization ability of the model is improved by applying the max-norm regularization as an optimization method for the loss function of this model. The solution is to apply a stochastic projection gradient descent method that projects a gradient. The sparser data became, it was confirmed that the propsed regularization method was relatively effective compared to the existing method through lots of experiment.

• Journal of the Korean Institute of Telematics and Electronics B
• /
• v.31B no.4
• /
• pp.145-154
• /
• 1994

This paper proposes an efficient method for improving the training performance of the neural network by using a hybrid of a stochastic approximation and a backpropagation algorithm. The proposed method improves the performance of the training by appliying a global optimization method which is a hybrid of a stochastic approximation and a backpropagation algorithm. The approximate initial point for a stochastic approximation and a backpropagation algorihtm. The approximate initial point for fast global optimization is estimated first by applying the stochastic approximation, and then the backpropagation algorithm, which is the fast gradient descent method, is applied for a high speed global optimization. And further speed-up of training is made possible by adjusting the training parameters of each of the output and the hidden layer adaptively to the standard deviation of the neuron output of each layer. The proposed method has been applied to the parity checking and the pattern classification, and the simulation results show that the performance of the proposed method is superior to that of the backpropagation, the Baba's MROM, and the Sun's method with randomized initial point settings. The results of adaptive adjusting of the training parameters show that the proposed method further improves the convergence speed about 20% in training.