• Journal of the Korean Operations Research and Management Science Society
• /
• v.17 no.2
• /
• pp.131-144
• /
• 1992

Even though nonhomogeneous Markov Decision Processes subsume homogeneous Markov Decision Processes and are more practical in the real world, there are many results for them. In this paper we address the nonhomogeneous Markov Decision Process with objective to maximize average reward. By extending works of Ross [17] in the homogeneous case adopting the result of Bean and Smith [3] for the dicounted deterministic problem, we first transform the original problem into the discounted nonhomogeneous Markov Decision Process. Then, secondly, we transform into the discounted deterministic problem. This approach not only shows the interrelationships between various problems but also attacks the solution method of the undiscounted nohomogeneous Markov Decision Process.

• Journal of the Korean Society for Industrial and Applied Mathematics
• /
• v.25 no.4
• /
• pp.149-161
• /
• 2021

This study proposes a methodology for mission/path planning of an unmanned aerial vehicle (UAV) using an artificial potential field with the Markov Decision Process (MDP). The planning problem is formulated as an MDP. A low-resolution solution of the MDP is obtained and used to define an artificial potential field, which provides a continuous UAV mission plan. A numerical case study is conducted to demonstrate the validity of the proposed technique.

• Journal of the Korean Operations Research and Management Science Society
• /
• v.19 no.1
• /
• pp.201-206
• /
• 1994

The linear search problem is concerned with finding a hiden target on the real line R. The position of the target governed by some probability distribution. It is desired to find the target in the least expected search time. This problem has been formulated as an optimization problem by a number of authors without making use of Markov Decision Process (MDP) theory. It is the aim of the paper to give a (MDP) formulation to the search problem which we feel is both natural and easy to follow.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.12 no.5
• /
• pp.2414-2428
• /
• 2018

In order to solve the problem that the current network security situation assessment methods just focus on the attack behaviors, this paper proposes a kind of network security situation assessment method based on Markov Decision Process and Game theory. The method takes the Markov Game model as the core, and uses the 4 levels data fusion to realize the evaluation of the network security situation. In this process, the Nash equilibrium point of the game is used to determine the impact on the network security. Experiments show that the results of this method are basically consistent with the expert evaluation data. As the method takes full account of the interaction between the attackers and defenders, it is closer to reality, and can accurately assess network security situation.

• Journal of the Korean Operations Research and Management Science Society
• /
• v.27 no.2
• /
• pp.33-49
• /
• 2002

CBM (Condition-Based Maintenance) has increasingly drawn attention in industry because of its many benefits. CBM Problem Is characterized as a state-dependent scheduling model that demands simultaneous maintenance actions, each for an attribute that influences on machine condition. This problem is very hard to solve within conventional Markov decision process framework. In this paper, we present an intelligent machine maintenance scheduler, for which a new incremental decision tree learning method as evolutionary system identification model and shortest path problem as schedule generation model are developed. Although our approach does not guarantee an optimal scheduling policy in mathematical viewpoint, we verified through simulation based experiment that the intelligent scheduler is capable of providing good scheduling policy that can be used in practice.

• IE interfaces
• /
• v.24 no.4
• /
• pp.396-407
• /
• 2011

We develop a cash management model in which firms face randomly occurred investment projects and retrieve investments upon the maturity of these projects. Using the Markov Decision Problem approach, we examine a control policy which dynamically adjusts the cash balance under the discounted cost criterion. The existence of an optimal policy is shown under some conditions. The optimal solution procedure is developed to find the optimal points and the optimal sizes for adjusting the cash balance. In numerical experiment, we investigate important structural properties of the optimal cash management policy.

• IE interfaces
• /
• v.24 no.4
• /
• pp.274-280
• /
• 2011

Delayed access to surgery may lead to deterioration in the patient condition, poor clinical outcomes, increase in the probability of emergency admission, or even death. The purpose of this work is to decide the number of patients selected from a waiting list and to schedule them in accordance with the operating room capacity in the next period. We formulate the problem as an infinite horizon Markov Decision Process (MDP), which attempts to strike a balance between the patient waiting times and overtime works. Structural properties of the proposed model are investigated to facilitate the solution procedure. The proposed procedure modifies the conventional value iteration method along with the binary search technique. An example of the optimal policy is provided, and computational results are given to show that the proposed procedure improves computational efficiency.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.8 no.10
• /
• pp.3394-3408
• /
• 2014

Conventional mobile state (MS) and base station (BS) association based on average signal strength often results in imbalance of cell load which may require more powerful processor at BSs and degrades the perceived transmission rate of MSs. To deal with this problem, a Markov decision process (MDP) for load balancing in a multi-cell system with multi-carriers is formulated. To solve the problem, exploiting Sarsa algorithm of on-line learning type [12], ${\alpha}$-controllable load balancing algorithm is proposed. It is designed to control tradeoff between the cell load deviation of BSs and the perceived transmission rates of MSs. We also propose an ${\varepsilon}$-differential soft greedy policy for on-line learning which is proven to be asymptotically convergent to the optimal greedy policy under some condition. Simulation results verify that the ${\alpha}$-controllable load balancing algorithm controls the behavior of the algorithm depending on the choice of ${\alpha}$. It is shown to be very efficient in balancing cell loads of BSs with low ${\alpha}$.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.10 no.7
• /
• pp.3171-3191
• /
• 2016

Due to growing demand on wireless data traffic, a large number of different types of base stations (BSs) have been installed. However, space-time dependent wireless data traffic densities can result in a significant number of idle BSs, which implies the waste of power resources. To deal with this problem, we propose an active state control algorithm based on semi-Markov decision process (SMDP) for a heterogeneous network. A MDP in discrete time domain is formulated from continuous domain with some approximation. Suboptimal on-line learning algorithm with a random policy is proposed to solve the problem. We explicitly include coverage constraint so that active cells can provide the same signal to noise ratio (SNR) coverage with a targeted outage rate. Simulation results verify that the proposed algorithm properly controls the active state depending on traffic densities without increasing the number of handovers excessively while providing average user perceived rate (UPR) in a more power efficient way than a conventional algorithm.

• International Journal of Reliability and Applications
• /
• v.11 no.2
• /
• pp.69-87
• /
• 2010

The optimal replacement problem was investigated for a multi-state deteriorated system for which the true internal state cannot be observed directly except when the system breaks down completely. The internal state was assumed to be monitored incompletely by a monitor that gives information related to the true state of the system. The problem was formulated as a partially observable Markov decision process. The optimal procedure was found to be a monotone procedure with respect to stochastic increasing ordering of the state probability vectors under some assumptions. Limiting the optimal procedure to a monotone procedure would greatly reduce the tremendous amount of calculation time required to find the optimal procedure.