The Journal of Korea Robotics Society (로봇학회논문지)

Korea Robotics Society (한국로봇학회)
권호 표지
DOI QR코드

DOI QR Code

Infrared Visual Inertial Odometry via Gaussian Mixture Model Approximation of Thermal Image Histogram

열화상 이미지 히스토그램의 가우시안 혼합 모델 근사를 통한 열화상-관성 센서 오도메트리

  • Jaeho Shin (Mechanical Engineering, Seoul National University) ;
  • Myung-Hwan Jeon (Institute of Advanced Machines and Design, Seoul National University) ;
  • Ayoung Kim (Mechanical Engineering, Seoul National University)
  • Received : 2023.05.25
  • Accepted : 2023.07.22
  • Published : 2023.08.31
Download PDF
⟨ Previous Next ⟩

Abstract

We introduce a novel Visual Inertial Odometry (VIO) algorithm designed to improve the performance of thermal-inertial odometry. Thermal infrared image, though advantageous for feature extraction in low-light conditions, typically suffers from a high noise level and significant information loss during the 8-bit conversion. Our algorithm overcomes these limitations by approximating a 14-bit raw pixel histogram into a Gaussian mixture model. The conversion method effectively emphasizes image regions where texture for visual tracking is abundant while reduces unnecessary background information. We incorporate the robust learning-based feature extraction and matching methods, SuperPoint and SuperGlue, and zero velocity detection module to further reduce the uncertainty of visual odometry. Tested across various datasets, the proposed algorithm shows improved performance compared to other state-of-the-art VIO algorithms, paving the way for robust thermal-inertial odometry.

Keywords

Acknowledgement

This study is a part of the research project, "Development of core machinery technologies for autonomous operation and manufacturing (NK230G)", which has been supported by a grant from National Research Council of Science & Technology under the R&D Program of Ministry of Science, ICT and Future Planning

References

  1. L. Landrieu and M. Simonovsky, "Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs," 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, pp. 4558-4567, 2018, DOI: 10.1109/CVPR.2018.00479. 
  2. P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, "Superglue: Learning feature matching with graph neural networks," IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, pp. 4938-4947, 2020, DOI: 10.1109/CVPR42600.2020.00499. 
  3. E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, "ORB: An efficient alternative to SIFT or SURF," IEEE International Conference on Computer Vision (ICCV), pp. 2564-2571, 2011. 
  4. D. G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, vol. 60, pp. 91-110, Nov., 2004, DOI: 10.1023/B:VISI.0000029664.99615.94. 
  5. H. Bay, A. Ess, T. Tuytelaars, and L. V. Gool, "Speeded-up robust features (SURF)," Computer Vision and Image Understanding, vol. 110, no. 3, pp. 346-359, Jun., 2008, DOI: 10.1016/j.cviu.2007.09.014. 
  6. P. V. K. Borges and S. Vidas, "Practical infrared visual odometry," IEEE Transactions on Intelligent Transportation Systems, vol. 17, no. 8, pp. 2205-2213, Aug., 2016, DOI: 10.1109/TITS.2016.2515625. 
  7. S. Vidas and S. Sridharan, "Hand-held monocular slam in thermal-infrared," 2012 12th International Conference on Control Automation Robotics & Vision (ICARCV), Guangzhou, China, pp. 859-864, 2012, DOI: 10.1109/ICARCV.2012.6485270. 
  8. E. Nilsson, "An optimization based approach to visual odometry using infrared images," 2010, [Online] https://www.diva-portal.org/smash/record.jsf?pid=diva2%3A351934&dswid=4224, Accessed: Sept. 24, 2010. 
  9. S. Khattak, C. Papachristos, and K. Alexis, "Keyframe-based thermal-inertial odometry," Journal of Field Robotics, vol. 37, no. 4, pp. 552-579, Jun., 2020, DOI: 10.1002/rob.21932. 
  10. Y.-S. Shin and A. Kim, "Sparse depth enhanced direct thermal-infrared SLAM beyond the visible spectrum," IEEE Robotics and Automation Letters, vol. 4, no. 3, pp. 2918-2925, Jul., 2019, DOI: 10.1109/LRA.2019.2923381. 
  11. S. Zhao, P. Wang, H. Zhang, Z. Fang, and S. Scherer, "TP-TIO: A robust thermal-inertial odometry with deep thermalpoint," 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA, pp. 4505-4512, 2020, DOI: 10.1109/IROS45743.2020.9341716. 
  12. J. Poujol, C. A. Aguilera, E. Danos, B. X. Vintimilla, R. Toledo, and A. D. Sappa, "A visible-thermal fusion based monocular visual odometry," Robot 2015: Second Iberian Robotics Conference: Advances in Robotics, vol. 417, pp. 517-528, 2015, DOI: 10.1007/978-3-319-27146-0_40. 
  13. T. Mouats, N. Aouf, A. D. Sappa, C. Aguilera, and R. Toledo, "Multispectral Stereo Odometry," IEEE Transactions on Intelligent Transportation Systems, vol. 16, no. 3, pp. 1210-1224, Jun., 2015, DOI: 10.1109/TITS.2014.2354731. 
  14. L. Chen, L. Sun, T. Yang, L. Fan, K. Huang, and Z. Xuanyuan, "Rgb-t slam: A flexible slam framework by combining appearance and thermal information," 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, pp. 5682-5687, 2017, DOI: 10.1109/ICRA.2017.7989668. 
  15. S. Khattak, C. Papachristos, and K. Alexis, "Visual-thermal landmarks and inertial fusion for navigation in degraded visual environments," 2019 IEEE Aerospace Conference, Big Sky, MT, USA, pp. 1-9, 2019, DOI: 10.1109/AERO.2019.8741787. 
  16. T. Mouats, N. Aouf, L. Chermak, and M. A. Richardson, "Thermal stereo odometry for UAVs," IEEE Sensors Journal, vol. 15, no. 11, pp. 6335-6347, Nov., 2015, DOI: 10.1109/JSEN.2015.2456337. 
  17. S. Wang, R. Clark, H. Wen, and N. Trigoni, "DeepVO: Towards end-to-end visual odometry with deep Recurrent Convolutional Neural Networks," 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, pp. 2043-2050, 2017, DOI: 10.1109/ICRA.2017.7989236. 
  18. A. Kendall, M. Grimes, and R. Cipolla, "PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization," 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, pp. 2938-2946, 2015, DOI: 10.1109/ICCV.2015.336. 
  19. I. Melekhov, J. Ylioinas, J. Kannala, and E. Rahtu, "Relative Camera Pose Estimation Using Convolutional Neural Networks," Advanced Concepts for Intelligent Vision Systems, pp. 675-687, 2017, DOI: 10.1007/978-3-319-70353-4_57. 
  20. D. DeTone, T. Malisiewicz, and A. Rabinovich, "Deep Image Homography Estimation," arXiv:1606.03798, 2016, DOI: 10.48550/arXiv.1606.03798. 
  21. E. Simo-Serra, E. Trulls, L. Ferraz, I. Kokkinos, P. Fua, and F. Moreno-Noguer, "Discriminative Learning of Deep Convolutional Feature Point Descriptors," 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, pp. 118-126, 2015, DOI: 10.1109/ICCV.2015.22. 
  22. K. M. Yi, E. Trulls, V. Lepetit, and P. Fua, "Lift: Learned invariant feature transform," European Conference on Computer Vision (ECCV), pp. 467-483, Sept., 2016, DOI: 10.1007/978-3-319-46466-4_28. 
  23. H. Noh, A. Araujo, J. Sim, T. Weyand, and B. Han, "Large-scale image retrieval with attentive deep local features," 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, pp. 3456-3465, 2017, DOI: 10.1109/ICCV.2017.374. 
  24. D. Li, X. Shi, Q. Long, S. Liu, W. Yang, F. Wang, Q. Wei, and F. Qiao, "DXSLAM: A robust and efficient visual SLAM system with deep features," 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA, pp. 4958-4965, 2020, DOI: 10.1109/IROS45743.2020.9340907. 
  25. J. Tang, L. Ericson, J. Folkesson, and P. Jensfelt, "GCNv 2: Efficient correspondence prediction for real-time SLAM," IEEE Robotics and Automation Letters, vol. 4, no. 4, pp. 3505-3512, Oct., 2019, DOI: 10.1109/LRA.2019.2927954. 
  26. R. Mur-Artal and J. D. Tardos, "ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras," IEEE Transactions on Robotics, vol. 33, no. 5, pp. 1255-1262, Oct., 2017, DOI: 10.1109/TRO.2017.2705103. 
  27. H.-J. Liang, N. J. Sanket, C. Fermuller, and Y. Aloimonos, "SalientDSO: Bringing Attention to Direct Sparse Odometry," IEEE Transactions on Automation Science and Engineering, vol. 16, no. 4, pp. 1619-1626, Oct., 2019, DOI: 10.1109/TASE.2019.2900980. 
  28. J. Engel, V. Koltun, and D. Cremers, "Direct Sparse Odometry," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 3, pp. 611-625, Mar., 2018, DOI: 10.1109/TPAMI.2017.2658577. 
  29. A. I. Mourikis and S. I. Roumeliotis, "A Multi-State Constraint Kalman Filter for Vision-aided Inertial Navigation," 2007 IEEE International Conference on Robotics and Automation, Rome, Italy, pp. 3565-3572, 2007, DOI: 10.1109/ROBOT.2007.364024. 
  30. N. Trawny and S. I. Roumeliotis, "Indirect Kalman Filter for 3D Attitude Estimation," University of Minnesota, Minneapolis, USA, Rep. 2005-002, Mar., 2005, [Online] https://www.semanticscholar.org/paper/Indirect-Kalman-Filter-for-3-D-Attitude-Estimation-Trawny-Roumeliotis/56ea520b4a9a7a718c490f392f275f5f8feb2886. 
  31. P. Geneva, K. Eckenhoff, W. Lee, Y. Yang, and G. Huang, "OpenVINS: A Research Platform for Visual-Inertial Estimation," 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France, pp. 4666-4672, 2020, DOI: 10.1109/ICRA40945.2020.9196524. 
  32. T. Qin, P. Li, and S. Shen, "Vins-mono: A robust and versatile monocular visual-inertial state estimator," IEEE Transactions on Robotics, vol. 34, no. 4, pp. 1004-1020, Aug., 2018, DOI: 10.1109/TRO.2018.2853729.