• Proceedings of the Korea Information Processing Society Conference
• /
• 2017.11a
• /
• pp.52-54
• /
• 2017

실시간으로 빠르게 발생하는 대용량 데이터를 다루기 위해 Apache Storm, Apache Spark 등 실시간 데이터 스트림 처리 기술에 대한 연구가 활발하다. 대부분의 실시간 처리 기술들은 단독으로 사용하기에 어려움이 있으며, 데이터 스트림의 입출력을 위해 메시징 시스템과 함께 사용하는 것이 일반적이다. Apache Kafka는 대표적인 분산 메시징 시스템으로써, 실시간으로 발생하는 대용량의 로그 데이터를 전달하는데 특화된 시스템이다. 현재 Kafka를 위한 다양한 성능 모니터링 도구들이 존재한다. 이러한 모니터링 도구들은 Kafka에서 처리되는 데이터의 양 이외에도 유입 데이터의 크기, 수집 속도, 처리 속도 등 다양한 데이터들을 관찰할 수 있다. 본 논문은 Kafka에서 제공하는 도구와 오픈 소스로 제공되는 여러 개의 도구들을 비교하여, 향후 Kafka의 로드 쉐딩에 대한 연구에 적용할 수 있는 최적의 모니터링 도구를 선별하고자 한다.

• Proceedings of the Korean Institute of Information and Commucation Sciences Conference
• /
• 2021.05a
• /
• pp.574-576
• /
• 2021

It is necessary to introduce micro-service techniques when constructing large-scale operating systems or systems that take into account scalability. Kafka is a message queue with the pub/sub model, which has features that are well applied to distributed environments and is also suitable for microservices in that it can utilize various data sources. In this paper, we propose a data sharing method for educational video sharing services using Apache's Kafka. The proposed system builds a Kafka cluster for the educational video sharing service to share various data, and also uses a spark cluster to link with recommendation systems based on similarities in educational videos. We also present a way to share various data sources, such as files, various DBMS, etc.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2021.11a
• /
• pp.1144-1147
• /
• 2021

코로나19 대유행으로 인해 전 세계가 원격으로 일상을 옮겨가면서 인터넷 트래픽량이 증가하고 보안 위협 또한 높아졌다. 높은 보안성이 요구되는 현 상황에 대응하기 위해 본 논문에서는 Suricata와 Elastic Stack, Kafka를 이용해 보안관제 로그 분석시스템을 구축하였다. 실시간으로 공격을 탐지하고 로그를 수집해 유의미한 데이터를 도출하여 시각화한다. 또한 시각화 한 대시보드를 제공함으로써 사용자는 공격의 위험도를 파악할 수 있고 앞으로의 공격을 대비할 수 있다.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2024.05a
• /
• pp.556-558
• /
• 2024

오늘날 클라우드 컴퓨팅은 FIFA, WTA, F1, MLB등과 같은 비디오 및 실시간 스포츠 이벤트에 널리 사용된다. DataM에 따르면 비디오 스트리밍 플랫폼 시장은 545억 달러에서 2,523달러에 달할 것으로 예측된다. 기존 실시간 스트리밍 방법은 스트리밍 비디오의 개수가 증가하고나 스트리밍 이용자가 증가할 경우 성능 저하 문제가 발생한다. 본 논문에서는 Apache Kafka Server를 활용한 대규모 비디오 스트리밍 기법을 제안한다. Apache Kafka Server를 사용하여 네트워크를 수집하면 대규모 데이터를 처리할 수 있으며, 데이터의 안정성과 실시간 처리를 할 수 있어 온라인 비디오 스트리밍에 적합하다. 이에 비디오 품질을 선택할 때 적합한 비디오 품질을 선택할 수 있다. 향후 제안하는 기법은 많은 데이터와 실험으로 실질적인 검증을 할 예정이다.

• Journal of English Language & Literature
• /
• v.64 no.2
• /
• pp.151-170
• /
• 2018

This paper examines the relationship and ideas of Hannah Arendt and Philip Roth including how they met, their correspondence and intellectual parallels, particularly in their shared criticism of Jewish ideals and culture in Europe and North America. It analyzes similarities in their careers and texts, especially between Eichmann in Jerusalem and Operation Shylock, as well as The Ghost Writer, while measuring their reception as social commentators and writers. Kafka was an important figure for both writers, Arendt's earliest writing engaged with the significance of Kafka in understanding and criticizing twentieth century political and cultural values in Europe. For Roth, Kafka offered a similar critique of moral principles he found corroded in North American Jewish life. Arendt connected with other writers, notably Isak Dinesen, W. H. Auden, Randall Jarrell and William Styron who further linked the two: he knew both Arendt and Roth and cited, incorrectly, a work by Arendt as the source for the key incident in his 1979 novel Sophie's Choice. He claimed it was Eichmann in Jerusalem; it was Arendt's The Origins of Totalitarianism. Arendt's reaction to Roth's fiction, however, remains a mystery: she died in 1975, before Roth began to seriously and consistently engage with Holocaust issues in works like The Ghost Writer (1979) and Operation Shylock (1993). Yet even in death they are joined. Their graves are only steps apart at the Bard College Cemetery in Annandale-on-Hudson, New York.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.16 no.5
• /
• pp.1446-1465
• /
• 2022

Diversified SaaS applications allow users more choices to use, according to their own preferences. However, the diversification of SaaS applications also makes it impossible for users to choose the best one. Furthermore, users can't take advantage of the functionality between SaaS applications. In this paper, we propose a platform that provides an SaaS mashup service, by extracting interoperable service functions from SaaS-based applications that independent vendors deploy and supporting a customized service recommendation function through log data binding in the cloud environment. The proposed SaaS mashup service platform consists of a SaaS aggregation framework and a log data binding framework. Each framework was concreted by using Apache Kafka and rule matrix-based recommendation techniques. We present the theoretical basis of implementing the high-performance message-processing function using Kafka. The SaaS mashup service platform, which provides a new type of mashup service by linking SaaS functions based on the above technology described, allows users to combine the required service functions freely and access the results of a rich service-utilization experience, using the SaaS mashup function. The platform developed through SaaS mashup service technology research will enable various flexible SaaS services, expected to contribute to the development of the smart-contents industry and the open market.

• The Journal of the Convergence on Culture Technology
• /
• v.9 no.6
• /
• pp.245-250
• /
• 2023

Digital twin technology, which copies information from real space into virtual space, is being used in a variety of fields.Interest in digital twins is increasing, especially in advanced manufacturing fields such as Industry 4.0-based smart manufacturing. Operating a digital twin system generates a large amount of data, and the data generated has different characteristics depending on the technology field, so it is necessary to efficiently manage resources and use an optimized digital twin platform technology. Research on digital twin pipelines has continued, mainly in the advanced manufacturing field, but research on high-speed pipelines suitable for data in the plant field is still lacking. Therefore, in this paper, we propose a pipeline design method that is specialized for digital twin data in the plant field that is rapidly poured through Apache Kafka. The proposed model applies plant information on a Revit basis. and collect plant-specific data through Apache Kafka. Equipped with a lightweight CFD engine, it is possible to create a digital twin model that is more suitable for the plant field than existing digital twin technology for the manufacturing field.

• International Journal of Contents
• /
• v.14 no.1
• /
• pp.28-33
• /
• 2018

In this paper, we design and implement a distributed, moving objects management system for processing locations and sensor data from smart black boxes. The proposed system is designed and implemented based on Apache Kafka, Apache Spark & Spark Streaming, Hbase, HDFS. Apache Kafka is used to collect the data from smart black boxes and queries from users. Received location data from smart black boxes and queries from users becomes input of Apache Spark Streaming. Apache Spark Streaming preprocesses the input data for indexing. Recent location data and indexes are stored in-memory managed by Apache Spark. Old data and indexes are flushed into HBase later. We perform experiments to show the throughput of the index manager. Finally, we describe the implementation detail in Scala function level.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.16 no.4
• /
• pp.1146-1165
• /
• 2022

A huge amount of data in the form of videos and images is being produced owning to advancements in sensor technology. Use of low performance commodity hardware coupled with resource heavy image processing and analyzing approaches to infer and extract actionable insights from this data poses a bottleneck for timely decision making. Current approach of GPU assisted and cloud-based architecture video analysis techniques give significant performance gain, but its usage is constrained by financial considerations and extremely complex architecture level details. In this paper we propose a data pipeline system that uses open-source tools such as Apache Spark, Kafka and OpenCV running over commodity hardware for video stream processing and image processing in a distributed environment. Experimental results show that our proposed approach eliminates the need of GPU based hardware and cloud computing infrastructure to achieve efficient video steam processing for face detection with increased throughput, scalability and better performance.

• Journal of the Korea Institute of Information and Communication Engineering
• /
• v.21 no.8
• /
• pp.1531-1539
• /
• 2017

The existing SSD storage tester processed logs in a 1-N structure between server and client using TCP and network file system. This method causes some problems for example, an increase in CPU usage and difficulty in exception handling, etc. In this paper, we implement a log processing message layer that can deal with asynchronous distributed processing using open source messaging system such as kafka, RabbitMQ and compare this layer with existing log transmission method. A log simulator was implemented to compare the transmission bandwidth and CPU usage. Test results show that the transmission using the message layer has higher performance than the transmission using the message layer, and the CPU usage does not show any significant difference The message layer can be implemented more easily than the conventional method and the efficiency is higher than that of the conventional method.