KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

Partially Observable Markov Decision Processes (POMDPs) and Wireless Body Area Networks (WBAN): A Survey

  • Mohammed, Yahaya Onimisi (King Fahd University of Petroleum and Minerals) ;
  • Baroudi, Uthman A. (King Fahd University of Petroleum and Minerals)
  • Received : 2012.12.01
  • Accepted : 2013.03.19
  • Published : 2013.05.30
Download PDF
⟨ Previous Next ⟩

Abstract

Wireless body area network (WBAN) is a promising candidate for future health monitoring system. Nevertheless, the path to mature solutions is still facing a lot of challenges that need to be overcome. Energy efficient scheduling is one of these challenges given the scarcity of available energy of biosensors and the lack of portability. Therefore, researchers from academia, industry and health sectors are working together to realize practical solutions for these challenges. The main difficulty in WBAN is the uncertainty in the state of the monitored system. Intelligent learning approaches such as a Markov Decision Process (MDP) were proposed to tackle this issue. A Markov Decision Process (MDP) is a form of Markov Chain in which the transition matrix depends on the action taken by the decision maker (agent) at each time step. The agent receives a reward, which depends on the action and the state. The goal is to find a function, called a policy, which specifies which action to take in each state, so as to maximize some utility functions (e.g., the mean or expected discounted sum) of the sequence of rewards. A partially Observable Markov Decision Processes (POMDP) is a generalization of Markov decision processes that allows for the incomplete information regarding the state of the system. In this case, the state is not visible to the agent. This has many applications in operations research and artificial intelligence. Due to incomplete knowledge of the system, this uncertainty makes formulating and solving POMDP models mathematically complex and computationally expensive. Limited progress has been made in terms of applying POMPD to real applications. In this paper, we surveyed the existing methods and algorithms for solving POMDP in the general domain and in particular in Wireless body area network (WBAN). In addition, the papers discussed recent real implementation of POMDP on practical problems of WBAN. We believe that this work will provide valuable insights for the newcomers who would like to pursue related research in the domain of WBAN.

Keywords

References

  1. M. Chen, S. Gonzalez, A. Vasilakos, H. Cao, V.C.M. Leung, "Body Area Networks: A Survey," Springer Journal of Mobile Network Applications, April 2011, Vol. 16, No. 2, pp 171-193. https://doi.org/10.1007/s11036-010-0260-8
  2. Reusens, E., Josephm, W., Latre, B., Braem, B., Vermeeren, G., Tanghe, E., Martens, L., Moerman, I. and Blondia, C., "Characterization of On-Body Communication Channel and Energy Efficient Topology Design for Wireless Body Area Networks," IEEE Transactions on Information Technology in Biomedicine, vol. 13, no. 6, Nov. 2009, pp. 933-945. https://doi.org/10.1109/TITB.2009.2033054
  3. Hovakeemian, Y Naik, K. and Nayak, A. ,"A survey on dependability in Body Area Networks," in Proc. 5th International Symposium on Medical Information & Communication Technology (ISMICT), 27-30 March 2011, pp. 10-14.
  4. Jiang Xing and Yunru Zhu, "A Survey on Body Area Network," in Proc. 5th International Conference on Wireless Communications, Networking and Mobile Computing (WiCom '09), 24-26 Sept. 2009, pp. 1-4.
  5. Jovanov, Emil, "A survey of power efficient technologies for Wireless Body Area Networks," in Proc. IEEE 30th Annual International Conference on Engineering in Medicine and Biology Society (EMBS'08), 20-25 Aug. 2008, pp. 3628.
  6. Gopalan, S.A. and Jong-Tae Park, "Energy-efficient MAC protocols for wireless body area networks: Survey," in Proc. of International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT), 18-20 Oct. 2010, pp. 739-744.
  7. Barakah, D.M. and Ammad-uddin, M., "A Survey of Challenges and Applications of Wireless Body Area Network (WBAN) and Role of a Virtual Doctor Server in Existing Architecture Intelligent Systems," in Proc. Third International Conference on Modelling and Simulation (ISMS), 8-10 Feb. 2012, pp. 214-219.
  8. Patel, M. and W. Jianfeng, "Applications, challenges, and prospective in emerging body area networking technologies," IEEE Wireless Communications Magazine, vol. 17, no. 1, Feb. 2010, pp. 80-88.
  9. M.A. Hanson, H.C. Powell Jr., A.T. Barth, K. Ringgenberg, B.H. Calhoun, J.H. Aylor, J.Lach, "Body Area Sensor Networks: Challenges and Opportunities," IEEE Computer Magazine, Vol. 42(1), Jan. 2009, pp. 58-65.
  10. R. D. Smallwood and E. J. Sondik, The optimal control of partially observable Markov decision processes, 1971, Stanford University.
  11. L. P. Kaelbling, M.L. Littman, A.R. Cassandra, "Planning and acting in partially observable stochastic domains," Journal of Artificial Intelligence, 1998. 101: p. 99-134. https://doi.org/10.1016/S0004-3702(98)00023-X
  12. G. Monahan, "A survey of partially observable Markove decision processes," J. Management Science, 1982(28): p. 1-16.
  13. Tang, L.C., kari, O., Frank, M., Xin, L. Venkatesh, A., "Markov Decision Process (MDP) framework for Optimizing Software on Mobile Phones," in Proc. of 7th ACM international conference on Embedded software (EMSOFT '09), pp. 11-20, 2009.
  14. Dejan, V.D., Vikram, K., "MIMO transmission control in fading channels-A constrained Markov Decision process Formulation with Monotone Randomized Policies," IEEE Transactions on Signal processing, 55(10), 2007.
  15. Kottapalli, V. A., et al., "Two-tiered wireless sensor network architecture for structural health Monitoring," in Proc. of 10th Annual International Symposium on Smart Structures and Materials, 2003.
  16. Montgomery, K. et al., "LifeGuard-A Personal Physiological Monitor for Extreme Environments," in Proc. of Medicine Meets Virtual Reality (MMVR04), 2004.
  17. Malan, D., et al., "Wireless sensor networks for habitat monitoring," in Proc. of International Workshop on Wearable and Implantable Body Sensor Network, 2004.
  18. Ng, J. , et al., "Ubiquitous monitoring environment for wearable and implantable sensors (UbiMon)," in Proc. of International Conference on Ubiquitous Computing (Ubicomp), 2004.
  19. Anand, P., et al., "Markov Decision Processes for Control of a Sensor Network-based Health Monitoring System" in Proc. of Seventh Conference on Innovative Applications of Artificial intelligence(IAAI), pp. 1529-1534, 2005.
  20. Goetschalck, R.; Sanner, S.; Driessens, K., "Reinforcement Learning with the Use of Costly Features," Recent Advances in Reinforcement Learning, Lecture Notes in Computer Science, vol. 5323, pp 124-135, 2008.
  21. Kwame, O.H., "Application of Markov Decision processes to a simplified Model of Robotic fighter," African Institute for Mathematical Sciences, 2009.
  22. Osais, Y., "New optimization Models for directional and Biological Wireless Sensor Networks, PhD. Dissertation, Department of System and Computer Engineering. Carleton University, Ottawa, Canada, 2010.
  23. Russell, S. and P. Norvig, eds. Artificial Intelligence: A Modern Approach, ed. P. Hall. 1995.
  24. Hauskrecht, M. and H. Fraser, Planning treatment of ischemic heart disease with partially observable Markov decision processes. Artificial Intelligence in Medicine, 2000(18): p. 221-244. https://doi.org/10.1016/S0933-3657(99)00042-1
  25. Hauskrecht, M., "Value function approximation for partially observable Markov decision processes," Journal of Artificial Intelligence research, 2000.13: p. 33-95.
  26. Nikos, V., Mathhhijs, T.J.S., "A fast point-based algorithm for POMDPs," in Proc. of Belgian-Dutch Conference on Machine Learning, Brussels, Belgium, pp. 170-176, Jan. 2004.
  27. Milos, H., Branislav, K., "Approximate Linear programming for Solving Hybrid factored MDPs," in Proc. of 9th International Symposium on Artificial Intelligence and Mathematics, 2006.
  28. Zhang, N.L. and W. Liu, "Planning in Stochastic domains: problem Characteristics and approximations," Hong Kong University of Science and Technology, 1996.
  29. Lovejoy, W., "Computationally Feasible Bounds for Partially Observable Markov Decision Processes," J. Operations Research, p. 162-175, 1991.
  30. Cheng, H.T., "Algorithms for Partially Observable Markov Decision Processes," Ph.D. Thesis University of British Columbia, Vancouver, BC (1988).
  31. Littman, M.L., "The Witness Algorithm: Solving Partially Observable Markov Decision Processes," Technical Report: CS-94-40, 1994, Brown University, department of Computer Science.
  32. Pineau, J., et al, "Point based value iteration: anytime algorithm for POMDPs," in Proc. of Int. joint conference on artificial intelligence, 2003. Mexico.
  33. Matthijs T. J., et al, "Randomized Point-based Value Iteration for POMDPs," Journal of Artificial Intelligence Research, p. 195-220, 2005.
  34. Edwin, K.P., et al., "Monte Carlo Basex Partially Observable Markov Decision Process Approximation for Adaptive sensing," in Proc. of the 9th International Workshop on Discrete Event Systems, Goteborg, Sweden, pp. 173-180, 2008.
  35. Zhengzhu, F., "Region-Based Dynamic programming for partially Observable Markov Decision Processes," 2006, Department of Computer science University of Massachusetts: Amherst.
  36. Hyeong, S.O., et al., "Parallel Rollout for Online Solution of Partially Observable Decision Process," Kluwer Academic Publishers, pp. 309-341, 2004.
  37. Berstekas, D.P. and D.A. Castanon, "Rollout Algorithms for Stochastic Scheduling Problems," Journal of Heuristics, pp. 89-108, 1999.
  38. Amato, C., et al., "Solving POMDPs Using Quadratically Constrained Linear Programs," AAMAS, Hokkaido, Japan, 2006.
  39. Ng, A., P. Ron, and D. Koller, "Policy Search via Density Estimation," Advances in Neural Information Processing Systems 12, Denver, Colorado, pp. 1022-1028, 2000.
  40. Ng, A.Y. and M. Jordan, "PEGASUS: A Policy Search Method for Large MDPs and POMDPs," in Proc. of Uncertainty in Artificial Intelligence. 2000.
  41. Satia, J.K. and R.E. Lave, "Markovian Decision Processes With Probabilistic Observation of States," Management Science, 1973.20(1).
  42. Smith, T. and R. Simmons, "Heuristic search value iteration for POMDPs," Uncertainty in Artificial Intelligence. 2004.
  43. Poupart, P. and C. Boutilier, "Value-directed compression of POMDPs," Advances in Neural Information Processing Systems 15 MIT press, 2003.
  44. Roy, N., et al., "Finding Approximate POMDP Solutions Through Belief Compression," Journal of Artificial Intelligence Research, 2005.23: p. 1-40.
  45. Roy, N. and G. Gordon, "Exponential Family PCA for Belief Compression in POMDPs," Advances in Neural Information Processing Systems 15, 2003.
  46. Xin, F., et al., "A POMDP Based k-Coverage Dynamic Scheduling Protocol for Wireless Sensor Network," in Proc. of IEEE Globecom, 2010.
  47. Puterman, M.L., Markov Decision Processes: Discrete Stochastic Dynamic Programming, Publisher: Wiley-Interscience, 1994.
  48. Cadei, M. and D.Z. Du, "Improving Wireless sensor network lifetime through power aware organization," Wireless Networks, 2005. 11(3): pp. 333-340. https://doi.org/10.1007/s11276-005-6615-6
  49. Shih, E., et al., "Physical Layer Driven Protocol And Algorithm Design For Energy Efficient Wireless Sensor Network," in Proc. of the ACM 7th annual International conference on mobile computing and networking. 2001. New York, USA.
  50. Tian, D. and N.D. Georganas, "A coverage-Preserving node scheduling scheme for large wireless sensor networks," in Proc. of 1st ACM International Workshop on Wireless Sensor Networks and Applications. 2002. New York, USA.
  51. Huang, C.F. and Y.C. Tsengm, "The Coverage Problem in A Wireless Sensor Network," in Proc. 2nd ACM International Conference on Wireless Sensor Networks and Applications," 2003. New York, USA.
  52. Bayardo, R., et al., "Infosleuth: An agent-based semantic integration of information in open and dynamic environments," in Proc. of the ACM SIGMOD International Conference on Management of Data. 1997.
  53. Yahya, O., et al., "Thermal Management of Biosensor Networks," in Proc. 7th IEEE Consumer Communications and Networking Conference (CCNC), pp. 1-5. 9-12, Jan. 2010.
  54. Tang, Q., et al., "Thermal Aware Routing Algorithm For Implanted Sensor Networks," Distributed Computing in Sensor Systems, Springer, pp. 206-217, 2005.
  55. Tang, Q., et al., "Communication Scheduling to Minimize Thermal Effects of Implanted Biosensor Networks in Homogenous Tissue," IEEE Transactions on Biomedical Engineering, 52(7): pp. 1285-1294, 2005. https://doi.org/10.1109/TBME.2005.847527
  56. Osais, Y., et al., "Optimal Management of Rechargeable Biosensors in Temperature Sensitive Environment," Proc. 72nd IEEE Vehicular Technology Conference (VTC 2010-Fall), 6-9 Sept. , pp. 1-5, 2010.
  57. White, I., "Optimal Diagnostic Questionnaires Which Allow Less Than Truthful Responses," Information and Control, Vol. 32, pp. 61-74, 1976. https://doi.org/10.1016/S0019-9958(76)90110-8
  58. White, I., "Bounds on Optimal Cost for a Replacement Problem with Partial Observations, Naval Research Logistics Quarterly, p. 415-422, 1979.
  59. Zois, D.-S., et al., "A POMDP Framework for Heterogeneous Sensor Selection in Wireless Body Area Networks," in Proc. of IEEE INFOCOM 2012, pp. 2611-2615.
  60. Mitra, U, et al, "KNOWME: A Case Study in Wireless Body Area Sensor Network Design," IEEE Communications Magazine, pp. 116-125, May 2102.
  61. Au, L., et al., "CARER: Efficient Dynamic Sensing for Continuous Activity Monitoring," in Proc. 33rd Annual International Conference of the IEEE EMBS, Boston, Massachusetts USA, August 30 - September 3, pp. 2228-2232, 2011.

Cited by

  1. A Lightweight Integrity Authentication Scheme based on Reversible Watermark for Wireless Body Area Networks vol.8, pp.12, 2013, https://doi.org/10.3837/tiis.2014.12.023
  2. A Secure Medical Information Management System for Wireless Body Area Networks vol.10, pp.1, 2013, https://doi.org/10.3837/tiis.2016.01.013