• The Journal of the Convergence on Culture Technology
• /
• v.8 no.2
• /
• pp.361-366
• /
• 2022

There are several ways to process large amounts of data. Depending on the processing method, there is a big difference in processing speed to create a large data list. Typically, to make a large data list, large data is converted into a normalized query, and the result of the query is stored in a List Map and converted into a printable form. This process occurs as a cause of lowering the processing speed step by step. In the process of storing the results of the created query as a List Map, the processing speed differs because the data is stored in a different format for each type of data. Through the simultaneous processing of GO language, we want to solve the problem of the existing difference in processing speed. In other words, it compares the results of GO language concurrency processing by providing how different and how it proceeds between the format contained in the existing List Map and the method of processing using concurrency in large data lists for faster processing. do.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2014.04a
• /
• pp.301-304
• /
• 2014

비정형 게임 Big Data에서 고효율 정보를 추출하고, 신뢰 할 수 있는 클러스터 게임서버 환경을 위한 병렬 처리를 위해 MapReduce를 사용한다. 본 논문에서는 빈번하게 입력되는 신규 게임데이터 처리를 위해 함수 Demap을 사용하는 Incremental MapReduce를 적용하여 불필요한 중간 값 저장과 재계산 없이 점차적으로 MapReduce 함수를 실행한다.

• Journal of the Korea Convergence Society
• /
• v.9 no.10
• /
• pp.13-19
• /
• 2018

MapReduce, the Hadoop's essential core technology, is most commonly used to process big data based on the Hadoop distributed file system. However, the existing MapReduce-based big data processing techniques have a feature of dividing and storing files in blocks predefined in the Hadoop distributed file system, thus wasting huge infrastructure resources. Therefore, in this paper, we propose an efficient MapReduce-based big data processing scheme. The proposed method enhances the storage efficiency of a big data infrastructure environment by converting and compressing the data to be processed into a data format in advance suitable for processing by MapReduce. In addition, the proposed method solves the problem of the data processing time delay arising from when implementing with focus on the storage efficiency.

• Journal of Digital Contents Society
• /
• v.14 no.3
• /
• pp.313-321
• /
• 2013

In recent, proliferation of mobile smart devices have led to big-data era, the importance of location-based services is increasing due to the exponential growth of trajectory related data. In order to process trajectory data, parallel processing platforms such as cloud computing and MapReduce are necessary. Currently, the researches based on MapReduce are on progress, but due to the MapReduce's properties in using batch processing and simple key-value structure, applying MapReduce framework for real time LBS is difficult. Therefore, in this research we propose a suitable system design on efficient indexing and search techniques for real time service based on detailed analysis on the properties of MapReduce.

• IEMEK Journal of Embedded Systems and Applications
• /
• v.9 no.2
• /
• pp.109-115
• /
• 2014

Analyzing large scale data has become an important activity for many organizations. Since MapReduce is a promising tool for processing the massive data sets, there are increasing studies to evaluate the performance of various algorithms related to MapReduce. In this paper, we first develop a simulation framework that includes MapReduce workload model, data center model, and the model of data access pattern. Then we propose two algorithms that can reduce the energy consumption of MapReduce systems. Using the simulation framework, we evaluate the performance of the proposed algorithms under different application characteristics and configurations of data centers.

• Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography
• /
• v.27 no.2
• /
• pp.235-247
• /
• 2009

Map is information media for human life. Recently, innovative map making technology has been developed and most updated various data are provided to map users. In consequence, use of maps is rapidly increasing in various applications. One of the examples is car navigation map because navigation maps provide reliable and necessary information with convenient way. Therefore, maps are to be changed to user-oriented from map making-oriented design. The main goal of spatial data infrastructure is to allow various users to access the real world information easily for decision making. However, most of the digital maps are focused on map making aspect such as providing up-to-date data, acquisition of accurate data with state-of-the-art technology, and addition of layers (For example, ortho-images and DEMs). If map data processing tools are to be provided with the digital maps, users could utilize maps for decision making without professional software or knowledge. Therefore, objective of this paper is to propose scheme of the digital map data processing service with application examples.

• KIPS Transactions on Software and Data Engineering
• /
• v.2 no.9
• /
• pp.589-594
• /
• 2013

As the concept of web characteristics represented by openness and mind sharing grows more and more popular, device log data generated by both users and developers have become increasingly complicated. For such reasons, a log data processing mechanism that automatically produces meaningful data set from large amount of log records have become necessary for mobile device UX(User eXperience) analysis. In this paper, we define the attributes of to-be-analyzed log data that reflect the characteristics of a mobile device and collect real log data from mobile device users. Along with the MapReduce programming paradigm in Hadoop platform, we have performed a mobile device User eXperience analysis in a distributed processing environment using the collected real log data. We have then demonstrated the effectiveness of the proposed analysis mechanism by applying the various combinations of Map and Reduce steps to produce a simple data schema from the large amount of complex log records.

• KIPS Transactions on Software and Data Engineering
• /
• v.3 no.11
• /
• pp.479-486
• /
• 2014

MapReduce is a programing model used for parallelly processing a large amount of data. To analyze a large amount data, the data cube is widely used, which is an operator that computes group-bys for all possible combinations of given dimension attributes. When the number of dimension attributes is n, the data cube computes $2^n$ group-bys. In this paper, we propose an efficient method for computing data cubes using MapReduce. The proposed method partitions $2^n$ group-bys into $_nC_{{\lceil}n/2{\rceil}}$ batches, and computes those batches in stages using ${\lceil}n/2{\rceil}$ MapReduce jobs. Compared to the existing methods, the proposed method significantly reduces the amount of intermediate data generated by mappers, so that the cost of sorting and transferring those intermediate data is reduced significantly. Consequently, the total processing time for computing a data cube is reduced. Through experiments, we show the efficiency of the proposed method over the existing methods.

• Journal of Electrical Engineering and Technology
• /
• v.6 no.4
• /
• pp.573-579
• /
• 2011

To meet the scalability and performance requirements of data analyses, which often involve voluminous data, efficient parallel or concurrent algorithms and frameworks are essential. We present a high-performance Korean morphological analyzer which employs the MapReduce framework on the graphics processing unit (GPU). MapReduce is a programming framework introduced by Google to aid the development of web search applications on a large number of central processing units (CPUs). GPUs are designed as a special-purpose co-processor. Their programming interfaces are typically formulated for graphics applications. Compared to CPUs, GPUs have greater computation power and memory bandwidth; however, GPUs are more difficult to program because of the design of their architectures. The performance of the Korean morphological analyzer using the MapReduce framework on the GPU is evaluated in comparison with the CPU-based model. The proposed Korean Morphological analyzer shows promising scalable performance on distributed computing with the GPU.

• Journal of Information Processing Systems
• /
• v.13 no.4
• /
• pp.668-676
• /
• 2017

Recently, the importance of big data has been emphasized with the development of smartphone, web/SNS. As a result, MapReduce, which can efficiently process big data, is receiving worldwide attention because of its excellent scalability and stability. Since big data has a large amount, fast creation speed, and various properties, it is more efficient to process big data summary information than big data itself. Wavelet histogram, which is a typical data summary information generation technique, can generate optimal data summary information that does not cause loss of information of original data. Therefore, a system applying a wavelet histogram generation technique based on MapReduce has been actively studied. However, existing research has a disadvantage in that the generation speed is slow because the wavelet histogram is generated through one or more MapReduce Jobs. And there is a high possibility that the error of the data restored by the wavelet histogram becomes large. However, since the wavelet histogram generation system based on the MapReduce developed in this paper generates the wavelet histogram through one MapReduce Job, the generation speed can be greatly increased. In addition, since the wavelet histogram is generated by adjusting the error boundary specified by the user, the error of the restored data can be adjusted from the wavelet histogram. Finally, we verified the efficiency of the wavelet histogram generation system developed in this paper through performance evaluation.