Journal of the Korea Institute of Military Science and Technology (한국군사과학기술학회지)

The Korea Institute of Military Science and Technology (한국군사과학기술학회)
권호 표지
DOI QR코드

DOI QR Code

FAST Design for Large-Scale Satellite Image Processing

대용량 위성영상 처리를 위한 FAST 시스템 설계

  • Lee, Youngrim (The Defense AI Technology Center, Agency for Defense Development) ;
  • Park, Wanyong (The Defense AI Technology Center, Agency for Defense Development) ;
  • Park, Hyunchun (The Defense AI Technology Center, Agency for Defense Development) ;
  • Shin, Daesik (The Defense AI Technology Center, Agency for Defense Development)
  • 이영림 (국방과학연구소 국방인공지능기술센터) ;
  • 박완용 (국방과학연구소 국방인공지능기술센터) ;
  • 박현춘 (국방과학연구소 국방인공지능기술센터) ;
  • 신대식 (국방과학연구소 국방인공지능기술센터)
  • Received : 2022.03.07
  • Accepted : 2022.07.29
  • Published : 2022.08.05
Download PDF
⟨ Previous Next ⟩

Abstract

This study proposes a distributed parallel processing system, called the Fast Analysis System for remote sensing daTa(FAST), for large-scale satellite image processing and analysis. FAST is a system that designs jobs in vertices and sequences, and distributes and processes them simultaneously. FAST manages data based on the Hadoop Distributed File System, controls entire jobs based on Apache Spark, and performs tasks in parallel in multiple slave nodes based on a docker container design. FAST enables the high-performance processing of progressively accumulated large-volume satellite images. Because the unit task is performed based on Docker, it is possible to reuse existing source codes for designing and implementing unit tasks. Additionally, the system is robust against software/hardware faults. To prove the capability of the proposed system, we performed an experiment to generate the original satellite images as ortho-images, which is a pre-processing step for all image analyses. In the experiment, when FAST was configured with eight slave nodes, it was found that the processing of a satellite image took less than 30 sec. Through these results, we proved the suitability and practical applicability of the FAST design.

Keywords

References

  1. Almeer MH, Cloud Hadoop Map Reduce for Remote Sensing Image Analysis, Journal of Emerging Trends in Computing and Information Sciences, Vol. 3, No. 4, pp. 637-644, 2012.
  2. The Apache Software Foundation, "Apache Hadoop," Apache Hadoop, 2006, https://hadoop.apache.org/, accessed 22 Apr 2022.
  3. Chen Z, Chen N and Yang C, Di L, Cloud Computing Enabled Web Processing Service for Earth Observation Data Processing, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 5, No. 6, pp. 1637-1649, 2012. https://doi.org/10.1109/JSTARS.2012.2205372
  4. Dean J and Ghemawat S, MapReduce: Simplified Data Processing on Large Clusters, Communications of the ACM, Vol. 51, No. 1, pp. 107-113, 2008. https://doi.org/10.1145/1327452.1327492
  5. Ghemawat S, Gobioff H and Leung ST, The Google File System, SIGOPS Operating Systems Review, Vol. 37, No. 5, pp. 29-43, 2003. https://doi.org/10.1145/1165389.945450
  6. Hee-Jeong H., Hyun Y., Jae-Moo H. and Young-Je P., "Systemic Design and Development of the Second Geostationary Ocean Color Ground Segment," KIISE Transactions on Computing Practices, Vol. 25, No. 10, pp. 477-484, 2019. https://doi.org/10.5626/KTCP.2019.25.10.477
  7. Huang W, Meng L, Zhang D and Zhang W, In-Memory Parallel Processing of Massive Remotely Sensed Data Using an Apache Spark on Hadoop YARN Model, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 10, No. 1, pp. 3-19, 2016. https://doi.org/10.1109/JSTARS.2016.2547020
  8. Hunt P, Konar M, Junqueira FP and Reed B, ZooKeeper: Wait-Free Coordination for Internet-Scale Systems, USENIX Annual Technical Conference, pp. 145-158, 2020.
  9. Huo Y, Blaber J, Damon SM, Boyd BD, Bao S, Parvathaneni P, Noguera CB, Chaganti S, Nath V, Greer JM, Lyu I, French WR, Newton AT, Rogers BP and Landman BA, Towards Portable LargeScale Image Processing with High-Performance Computing, Journal of Digital Imaging, Vol. 31, No. 3, pp. 304-314, 2018. https://doi.org/10.1007/s10278-018-0080-0
  10. Knoth C and Nust D, Reproducibility and Practical Adoption of Geobia with Open-Source Software in Docker Containers, Remote Sensing, Vol. 9, No. 3, p. 290, 2017.
  11. Statistics Korea, "The land area(G20)," Korean Statistical Information Service, 30 Sep 2021, https://kosis.kr/statHtml/statHtml.do?orgId=101&tblId=DT_2KAA101_G20&conn_path=I2, accessed 21 July 2022.
  12. Kreps J, Narkhede N and Rao J, Kafka: A Distributed Messaging System for Log Processing, Proceedings of the NetDB, Vol. 11, pp. 1-7, 2011.
  13. Liu X, Iftikhar N and Xie X, Survey of Real-Time Processing Systems for Big Data, Proceedings of the 18th International Database Engineering & Applications Symposium, pp. 356-361, 2014.
  14. Merkel D, Docker: Lightweight Linux Containers for Consistent Development and Deployment, Linux Journal, LLC 239, Article 2, 2014.
  15. Nibedita Mohanta, "How Many Satellites are Orbiting the Earch in 2021?," Geospatial World, 28 May 2021, https://www.geospatialworld.net/blogs/howmany-satellites-are-orbiting-the-earth-in-2021/, accessed 30 Jun 2022.
  16. OGC, OpenGIS Web Processing Service Specification (Version 1.0.0) OGC Standard No. 05-007r7, OGC, 2007.
  17. Planet Labs PBC, "Planet," Planet, http://planet.com, accessed 22 Apr 2022
  18. Rathore MM, Son H, Ahmad A, Paul A and Jeon G, Real-Time Big Data Stream Processing Using GPU with Spark Over Hadoop Ecosystem, International Journal of Parallel Programming, Vol. 46, No. 3, pp. 630-646, 2018. https://doi.org/10.1007/s10766-017-0513-2
  19. S. N. K. B. Amit, S. Shiraishi, T. Inoshita and Y. Aoki, "Analysis of Satellite Images for Disaster Detection," 2016 IEEE International Geoscience and Remote Sensing Symposium(IGARSS), pp. 5189-5192, 2016, doi: 10.1109/IGARSS.2016.7730352.
  20. S. P. R and R. U, "A Survey on Agriculture Monitoring with Satellite and Its Benefits," 2022 8th International Conference on Advanced Computing and Communication Systems(ICACCS), pp. 1854-1858, 2022, doi: 10.1109/ICACCS54159.2022.9785323.
  21. SERGEY S., TAMARA D., ANNA K. and EIDELSTEINC L., "Engaging the Public to Fight the Consequences of Terrorism and Disasters," IOS Press, 120, pp. 91-103, 2015.
  22. Sharma T, Shokeen V and Mathur S, Distributed Processing of Satellite Images on Hadoop to Generate Normalized Difference Vegetation Index Images, Proceedings of 2017 International Conference on Computing, Communication, pp. 1-5, 2017.
  23. Vavilapalli VK, et. al., Apache Hadoop Yarn: Yet Another Resource Negotiator, Proceedings of the 4th Annual Symposium on Cloud Computing(SOCC '13), Article 5, pp. 1-16, 2013.
  24. Zaharia M, et. al., Spark: Cluster Computing with Working Sets. Proceedings of 2nd USENIX Conf. Hot Topics Cloud Comput., pp. 10-10, 2010.
  25. Zhang Y, Gao Q, Gao L and Wang C, iMapReduce: A Distributed Computing Framework for Iterative Computation, Journal of Grid Computing, Vol. 10, No. 1, pp. 47-68, 2012. https://doi.org/10.1007/s10723-012-9204-9