• The Korean Journal of Applied Statistics
• /
• v.30 no.1
• /
• pp.1-12
• /
• 2017

Many stochastic processes satisfy the Markov property exactly or at least approximately. An interested property in the Markov process is the first passage time. Since the sequential analysis by Wald, the approximation of the first passage time has been studied extensively. The Statistical computing technique due to the development of high-speed computers made it possible to calculate the values of the properties close to the true ones. This article introduces an exponentially weighted moving average (EWMA) control chart as an example of the Markov process, and studied how to calculate the average run length with problematic issues that should be cautioned for correct calculation. The results derived for approximation of the first passage time in this research can be applied to any of the Markov processes. Especially the approximation of the continuous time Markov process to the discrete time Markov chain is useful for the studies of the properties of the stochastic process and makes computational approaches easy.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.7 no.5
• /
• pp.1036-1057
• /
• 2013

Wireless body area network (WBAN) is a promising candidate for future health monitoring system. Nevertheless, the path to mature solutions is still facing a lot of challenges that need to be overcome. Energy efficient scheduling is one of these challenges given the scarcity of available energy of biosensors and the lack of portability. Therefore, researchers from academia, industry and health sectors are working together to realize practical solutions for these challenges. The main difficulty in WBAN is the uncertainty in the state of the monitored system. Intelligent learning approaches such as a Markov Decision Process (MDP) were proposed to tackle this issue. A Markov Decision Process (MDP) is a form of Markov Chain in which the transition matrix depends on the action taken by the decision maker (agent) at each time step. The agent receives a reward, which depends on the action and the state. The goal is to find a function, called a policy, which specifies which action to take in each state, so as to maximize some utility functions (e.g., the mean or expected discounted sum) of the sequence of rewards. A partially Observable Markov Decision Processes (POMDP) is a generalization of Markov decision processes that allows for the incomplete information regarding the state of the system. In this case, the state is not visible to the agent. This has many applications in operations research and artificial intelligence. Due to incomplete knowledge of the system, this uncertainty makes formulating and solving POMDP models mathematically complex and computationally expensive. Limited progress has been made in terms of applying POMPD to real applications. In this paper, we surveyed the existing methods and algorithms for solving POMDP in the general domain and in particular in Wireless body area network (WBAN). In addition, the papers discussed recent real implementation of POMDP on practical problems of WBAN. We believe that this work will provide valuable insights for the newcomers who would like to pursue related research in the domain of WBAN.

• Journal of the Korean Mathematical Society
• /
• v.38 no.1
• /
• pp.25-35
• /
• 2001

In this paper we consider an age dependent branching process whose particles move according to a Markov process with continuous state space. The Markov process is assumed to the stationary with independent increments and positive recurrent. We find some sufficient conditions for he Markov motion process such that the empirical distribution of the positions converges to the limiting distribution of the motion process.

• Journal of the Korean Mathematical Society
• /
• v.61 no.4
• /
• pp.693-711
• /
• 2024

A Markov-modulated Bernoulli process is a generalization of a Bernoulli process in which the success probability evolves over time according to a Markov chain. It has been widely applied in various disciplines for modeling and analysis of systems in random environments. This paper focuses on providing analytical characterizations of the Markovmodulated Bernoulli process by introducing key metrics, including success period, failure period, and cycle. We derive expressions for the distributions and the moments of these metrics in terms of the model parameters.

• Bulletin of the Korean Mathematical Society
• /
• v.49 no.1
• /
• pp.63-74
• /
• 2012

Based on the Hilbert $C^*$-module structure we study the reconstruction theorem for stationary monotone quantum Markov processes from quantum dynamical semigroups. We prove that the quantum stochastic monotone process constructed from a covariant quantum dynamical semigroup is again covariant in the strong sense.

• The Transactions of the Korean Institute of Electrical Engineers D
• /
• v.52 no.9
• /
• pp.533-539
• /
• 2003

Reinforcement learning (RL) has been widely used as a learning mechanism of an artificial life system. However, RL usually suffers from slow convergence to the optimum state-action sequence or a sequence of stimulus-response (SR) behaviors, and may not correctly work in non-Markov processes. In this paper, first, to cope with slow-convergence problem, if some state-action pairs are considered as disturbance for optimum sequence, then they no to be eliminated in long-term memory (LTM), where such disturbances are found by a shortest path-finding algorithm. This process is shown to let the system get an enhanced learning speed. Second, to partly solve a non-Markov problem, if a stimulus is frequently met in a searching-process, then the stimulus will be classified as a sequential percept for a non-Markov hidden state. And thus, a correct behavior for a non-Markov hidden state can be learned as in a Markov environment. To show the validity of our proposed learning technologies, several simulation result j will be illustrated.

• International Journal of Control, Automation, and Systems
• /
• v.1 no.3
• /
• pp.358-367
• /
• 2003

We consider discrete-time factorial Markov Decision Processes (MDPs) in multiple decision-makers environment for infinite horizon average reward criterion with a general joint reward structure but a factorial joint state transition structure. We introduce the "localization" concept that a global MDP is localized for each agent such that each agent needs to consider a local MDP defined only with its own state and action spaces. Based on that, we present a gradient-ascent like iterative distributed algorithm that converges to a local optimal solution of the global MDP. The solution is an autonomous joint policy in that each agent's decision is based on only its local state.cal state.

• IE interfaces
• /
• v.21 no.1
• /
• pp.85-95
• /
• 2008

Reliability prediction serves many purposes during the life of a system, so several methods have been developed to predict the parts and systems reliability. MIL-HDBK-217F, among the those methods, has been widely used as a requisite tool for the reliability prediction which is applied to nuclear power plants and their safety regulations. This paper presents the reliability prediction for the DSP(Digital Signal Processor) module composed of three assemblies. One of the assemblies has a monitoring and self test function which is used to enhance the module reliability. The reliability of each assembly is predicted by MIL-HDBK-217F. Based on these predicted values, Markov modelling is finally used to predict the module reliability. Relax 7.7 software of Relax software corporation is used because it has many part libraries and easily handles Markov processes modelling.

• Communications of the Korean Mathematical Society
• /
• v.12 no.4
• /
• pp.1033-1038
• /
• 1997

A functional central limit theorem for a class of nonlinear stationary Markov processes which are geometrically Harris ergodic is derived.

• Journal of the Korean Mathematical Society
• /
• v.33 no.4
• /
• pp.983-992
• /
• 1996

Let p(x, dy) be a transition probability function on $(S, \rho)$, where S is a complete separable metric space. Then a Markov process $X_n$ which has p(x, dy) as its transition probability may be generated by random iterations of the form $X_{n+1} = f(X_n, \varepsilon_{n+1})$, where $\varepsilon_n$ is a sequence of independent and identically distributed random variables (See, e.g., Kifer(1986), Bhattacharya and Waymire(1990)).