• Annual Conference of KIPS
• /
• 2023.05a
• /
• pp.396-398
• /
• 2023

Statista 통계 조사에 따르면 데이터의 규모는 매년 증가할 것으로 예상하고 빅데이터 처리 프레임워크의 관심이 높아지고 있다. 빅데이터 처리 프레임워크 Spark는 Shuffle 과정에서 노드 간 데이터 전송이 일어난다. 이때 분산 처리한 데이터를 네트워크로 전송하기 위해 객체를 바이트 스트림으로 변환하여 메모리 buffer에 담는 직렬화 작업이 필요하다. 그러나 바이트 스트림을 buffer에 담는 과정에서 바이트 스트림의 크기가 메모리 buffer보다 클 경우, 메모리 할당 과정이 추가로 발생하여 전체적이 Spark의 성능 저하로 이어질 수 있다. 이에 본 논문에서는 Spark 환경에서 분산 처리 성능 향상을 위한 직렬화 buffer 최적화 시스템을 제안한다. 제안하는 방법은 Spark Driver가 Executor에게 작업을 할당하기 전 직렬화된 데이터 크기 측정과 직렬화 옵션 설정을 통해 Executor에게 적절한 buffer를 할당할 수 있다. 향후 제안하는 방법의 검증을 위해 실제 Spark 클러스터 환경에서 성능 평가가 필요하다.

• Annual Conference of KIPS
• /
• 2023.11a
• /
• pp.2-4
• /
• 2023

KISTI의 GPU 클러스터 시스템인 뉴론은 NVIDIA의 A100과 V100 GPU가 총 260개 탑재되어 있는 클러스터 시스템이다. 뉴론의 계산노드들은 고성능의 인터커넥트인 Infiniband(IB) 케이블로 연결되어 있어 멀티 노드 작업 수행 시에 고대역 병렬통신이 가능하다. 본 논문에서는 NVIDIA사에서 제공하는 NCCL의 벤치마크 코드를 이용하여 인터커넥트 네트워크의 통신 성능을 비교분석하는 방안에 대해서 소개한다.

• Annual Conference of KIPS
• /
• 2023.11a
• /
• pp.382-385
• /
• 2023

Spark는 대용량의 데이터를 처리를 위해 분산된 데이터를 네트워크로 모은 다음, 데이터를 분할하는 작업인 Shuffle을 진행한다. 이때 Spark 클러스터의 어느 한 노드의 네트워크 전송 속도가 느릴 경우 병목 현상으로 인한 전체 처리 성능이 저하된다. 이에 본 논문에서는 네트워크 병목 현상을 예방하기 위한 클러스터 구성 방법을 제안한다. 본 논문에서 제안하는 노드 선택 시스템은 iperf 도구를 이용해 노드들의 대역폭을 측정하고 이에 따라 노드 선택 알고리즘을 통해 클러스터를 구성한다. 기존 Spark 클러스터와 본 논문이 제안하는 시스템으로 구성한 클러스터를 비교했을 때, 250MB 로그 파일을 제외하고 750MB 로그 파일부터는 네트워크 전송 속도가 낮은 노드를 가지고 있는 클러스터의 성능이 병목 현상으로 인해 느려졌다. 본 논문의 제안에 따라 노드들의 네트워크 전송 속도를 고려하여 클러스터를 구성하면 네트워크 전송 속도로 발생하는 병목 현상을 예방할 수 있다.

• Journal of KIISE
• /
• v.43 no.10
• /
• pp.1115-1123
• /
• 2016

Assumption-based truth maintenance system (ATMS) is a tool that maintains the reasoning process of inference engine. It also supports non-monotonic reasoning based on dependency-directed backtracking. Bookkeeping all the reasoning processes allows it to quickly check and retract beliefs and efficiently provide solutions for problems with large search space. However, the amount of data has been exponentially grown recently, making it impossible to use a single machine for solving large-scale problems. The maintaining process for solving such problems can lead to high computation cost due to large memory overhead. To overcome this drawback, this paper presents an approach towards incrementally maintaining the reasoning process of inference engine on cluster using Spark. It maintains data dependencies such as assumption, label, environment and justification on a cluster of machines in parallel and efficiently updates changes in a large amount of inferred datasets. We deployed the proposed ATMS on a cluster with 5 machines, conducted OWL/RDFS reasoning over University benchmark data (LUBM) and evaluated our system in terms of its performance and functionalities such as assertion, explanation and retraction. In our experiments, the proposed system performed the operations in a reasonably short period of time for over 80GB inferred LUBM2000 dataset.

• Journal of KIISE:Computing Practices and Letters
• /
• v.10 no.2
• /
• pp.134-145
• /
• 2004

With rapid growth in the amount of data transferred on the Internet, traditional storage systems have reached the limits of their capacity and performance. SAN (Storage Area Network), which connects hosts to disk with the Fibre Channel switches, provides one of the powerful solutions to scale the data storage and servers. In this environment, the maintenance of data consistency among hosts is an important issue because multiple hosts share the files on disks attached to the SAN. To preserve data consistency, each host can execute the disk I/O whenever disk read and write operations are requested. However, frequent disk I/O requests cause the deterioration of the overall performance of a SAN cluster. In this paper, we introduce a SANtopia global buffer manager to improve the performance of a SAN cluster reducing the number of disk I/Os. We describe the design and algorithms of the SANtopia global buffer manager, which provides a buffer cache sharing mechanism among the hosts in the SAN cluster. Micro-benchmark results to measure the performance of block I/O operations show that the global buffer manager achieves speed-up by the factor of 1.8-12.8 compared with the existing method using disk I/O operations. Also, File system micro-benchmark results show that SANtopia file system with the global buffer manager improves performance by the factor of 1.06 in case of directories and 1.14 in case of files compared with the file system without a global buffer manager.

• Journal of the Korea Institute of Information Security & Cryptology
• /
• v.22 no.1
• /
• pp.43-56
• /
• 2012

Wireless sensor networks consist of numerous small-sized nodes equipped with limited computing power and storage as well as energy-limited disposable batteries. In this networks, nodes are deployed in a large given area and communicate with each other in short distances via wireless links. For energy efficient networks, dynamic clustering protocol is an effective technique to achieve prolonged network lifetime, scalability, and load balancing which are known as important requirements. this technique has a characteristic that sensing data which gathered by many nodes are aggregated by cluster head node. In the case of cluster head node is exposed by attacker, there is no guarantee of safe and stable network. Therefore, for secure communications in such a sensor network, it is important to be able to encrypt the messages transmitted by sensor nodes. Especially, cluster based sensor networks that are designed for energy efficient, strongly recommended suitable key management and authentication methods to guarantee optimal stability. To achieve secured network, we propose a key management scheme which is appropriate for hierarchical sensor networks. Proposed scheme is based on polynomial key pool pre-distribution scheme, and sustain a stable network through key authentication process.

• The Journal of the Convergence on Culture Technology
• /
• v.9 no.3
• /
• pp.715-720
• /
• 2023

The military is driving innovative changes such as AI, cloud computing, and drone operation through the Fourth Industrial Revolution. It is expected that such changes will lead to a rapid increase in the demand for information exchange requirements, reaching all lower-ranking soldiers, as networking based on IoT occurs. The flow of such information must ensure efficient information distribution through various infrastructures such as ground networks, stationary satellites, and low-earth orbit small communication satellites, and the demand for information exchange that is distributed through them must be appropriately dispersed. In this study, we redefined the DSCP, which is closely related to QoS (Quality of Service) in information dissemination, into 11 categories and performed research to map each cluster group identified by cluster analysis to the defense "information exchange requirement list" on a one-to-one basis. The purpose of the research is to ensure efficient information dissemination within a multi-layer integrated network (ground network, stationary satellite network, low-earth orbit small communication satellite network) with limited bandwidth by re-establishing QoS policies that prioritize important information exchange requirements so that they are routed in priority. In this paper, we evaluated how well the information exchange requirement lists classified by cluster analysis were assigned to DSCP through M&S, and confirmed that reclassifying DSCP can lead to more efficient information distribution in a network environment with limited bandwidth.

• KIISE Transactions on Computing Practices
• /
• v.21 no.12
• /
• pp.774-779
• /
• 2015

In this paper, we present the design and implementation of a large-scale qualitative spatial reasoner using Apache Spark, an in-memory high speed cluster computing environment, which is effective for sequencing and iterating component reasoning jobs. The proposed reasoner can not only check the integrity of a large-scale spatial knowledge base representing topological and directional relationships between spatial objects, but also expand the given knowledge base by deriving new facts in highly efficient ways. In general, qualitative reasoning on topological and directional relationships between spatial objects includes a number of composition operations on every possible pair of disjunctive relations. The proposed reasoner enhances computational efficiency by determining the minimal set of disjunctive relations for spatial reasoning and then reducing the size of the composition table to include only that set. Additionally, in order to improve performance, the proposed reasoner is designed to minimize disk I/Os during distributed reasoning jobs, which are performed on a Hadoop cluster system. In experiments with both artificial and real spatial knowledge bases, the proposed Spark-based spatial reasoner showed higher performance than the existing MapReduce-based one.

• Journal of KIISE:Computer Systems and Theory
• /
• v.34 no.9
• /
• pp.407-415
• /
• 2007

In parallel cluster computing systems, the efficiency of communication between computing nodes is one of important factors that decide overall system performance. Accordingly, many researchers have studied on high-performance inter-node communication. The recently launched multi-core processor, however. increases the importance of intra-node communication as well because the more the number of cores in a node, the more the number of parallel processes running in the same node. Though there have been studies on intra-node communications, these have limited considerations on the state-of-the-art systems. In this paper, we propose a Linux kernel module that minimizes the number of data copy by exploiting the memory mapping mechanism for high-performance intra-node communication. The proposed kernel module supports the Linux kernel version 2.6. The performance measurements over a multi-core system present that the proposed kernel module can achieve lower latency up to 62% and higher throughput up to 144% than an existing kernel module approach. In addition, the measurements reveal that the performance of intra-node communication can vary significantly based on whether the cores that run the communication processes are belong to the same processor package (i.e., sharing the L2 cache).

• Journal of Korean Society for Geospatial Information Science
• /
• v.7 no.2 s.14
• /
• pp.123-132
• /
• 1999

With the rapid increases of the amount of data and computing, the parallelization of the computing algorithm becomes necessary more than ever. However the parallelization had been conducted mostly in a super-computer until the rod 1990s, it was not for the general users due to the high price, the complexity of usage, and etc. A new concept for the parallel processing has been emerged in the form of K-clustering form the late 1990s, it becomes an excellent alternative for the applications need high computer power with a relative low cost although the installation and the usage are still difficult to the general users. The mapping algorithms (cut, join, resizing, warping, conversion from raster to vector and vice versa, etc) in GIS are well suited for the parallelization due to the characteristics of the data structure. If those algorithms are manipulated using PC-clustering, the result will be satisfiable in terms of cost and performance since they are processed in real flu with a low cos4 In this paper the tools and the libraries for the parallel processing and PC-clustering we introduced and how those tools and libraries are applied to mapping algorithms in GIS are showed. Parallel programs are developed for the mapping algorithms and the result of the experiments shows that the performance in most algorithms increases almost linearly according to the number of node.