• KIPS Transactions on Software and Data Engineering
• /
• v.5 no.10
• /
• pp.477-482
• /
• 2016

As being the advanced analytics indispensable on big data for agile decision-making and tactical planning in enterprises, distributed processing platforms, such as Hadoop and Spark which distribute and handle the large volume of data on multiple nodes, receive great attention in the field. In Spark platform stack, Spark SQL unveiled recently to make Spark able to support distributed processing framework based on SQL. However, Spark SQL cannot effectively handle advanced analytics that involves machine learning and graph processing in terms of iterative tasks and task allocations. Motivated by these issues, this paper proposes the design of SQL-based big data optimal processing engine and processing framework to support advanced analytics in Spark environments. Big data optimal processing engines copes with complex SQL queries that involves multiple parameters and join, aggregation and sorting operations in distributed/parallel manner and the proposing framework optimizes machine learning process in terms of relational operations.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2013.05a
• /
• pp.476-477
• /
• 2013

Gadget-2 is a scientific simulation code has been used for many different types of simulations like, Colliding Galaxies, Cluster Formation and the popular Millennium Simulation. The code is parallelized with Message Passing Interface (MPI) and is written in C language. There is also a Java adaptation of the original code written using MPJ Express called Java Gadget. Java Gadget writes a lot of checkpoint data which may or may not use the HDF-5 file format. Since, HDF-5 is MPI-IO compliant, we can use our MPJ-IO library to perform parallel reading and writing of the checkpoint files and improve I/O performance. Additionally, to add reliability to the code execution, we propose the usage of Hadoop Distributed File System (HDFS) for writing the intermediate (checkpoint files) and final data (output files). The current code writes and reads the input, output and checkpoint files sequentially which can easily become bottleneck for large scale simulations. In this paper, we propose Sim-Hadoop, a framework to leverage HDFS and MPJ-IO for improving the I/O performance of Java Gadget code.

• The Transactions of the Korea Information Processing Society
• /
• v.6 no.9
• /
• pp.2311-2319
• /
• 1999

We present two heuristic algorithms for the task allocation problem (NP-complete problem) in parallel computing. The problem is to find an optimal mapping of multiple communicating tasks of a parallel program onto the multiple processing nodes of a distributed-memory multicomputer. The purpose of mapping these tasks into the nodes of the target architecture is the minimization of parallel execution time without sacrificing solution quality. Many heuristic approaches have been employed to obtain satisfactory mapping. Our heuristics are based on genetic algorithms and simulated annealing. We formulate an objective function as a total computational cost for a mapping configuration, and evaluate the performance of our heuristic algorithms. We compare the quality of solutions and times derived by the random, greedy, genetic, and annealing algorithms. Our experimental findings from a simulation study of the allocation algorithms are presented.

• 한국전산유체공학회:학술대회논문집
• /
• 2005.10a
• /
• pp.157-162
• /
• 2005

Parallel implementation and performance assessment of the grid assembly in a structured chimera grid approach is studied. The grid assembly process, involving hole cutting and searching donor, is parallelized on the PC cluster. A message passing programming model based on the MPI library is implemented using the single program multiple data(SPMD) paradigm. The coarse-grained communication is optimized with the minimized memory allocation because that the parallel grid assembly can access the decomposed geometry data in other processors by only message passing in the distributed memory system such as a PC cluster. The grid assembly workload is based on the static load balancing tied to flow solver. A goal of this work is a development of parallelized grid assembly that is suited for handling multiple moving body problems with large grid size.

• IEMEK Journal of Embedded Systems and Applications
• /
• v.16 no.4
• /
• pp.153-161
• /
• 2021

Real-time access is required to handle continuous and unstructured data and should be flexible in management under dynamic state. Platform can be built to allow data collection, storage, and processing from local-server or multi-server. Although the former centralize method is easy to control, it creates an overload problem because it proceeds all the processing in one unit, and the latter distributed method performs parallel processing, so it is fast to respond and can easily scale system capacity, but the design is complex. This paper provides data collection and processing on one platform to derive significant insights from various data held by an enterprise or agency in the latter manner, which is intuitively available on dashboards and utilizes Spark to improve distributed processing performance. All service utilize dockers to distribute and management. The data used in this study was 100% collected from Kafka, showing that when the file size is 4.4 gigabytes, the data processing speed in spark cluster mode is 2 minute 15 seconds, about 3 minutes 19 seconds faster than the local mode.

• The Transactions of the Korea Information Processing Society
• /
• v.3 no.6
• /
• pp.1386-1396
• /
• 1996

In the distributed multimedia application, high throughput and low delay is one of the most important QoS(Quality of Service) requirement. Emerging high speed communication offers transmission rate above Gbps, but it can not be utilized efficiently by the performance bottlenecks of communication protocols. To overcom the problem, we propose a horizontal processing architecture that processes data as soon as it arrive from the network. Unfortunately, the OSI protocol stack often imposes ordering constraints that prevent concurrent processing of the protocol layers. By using a fixed packet format, the network layer through the presentation layer are processed in parallel. Our prototype shows that the proposed model has performance improvement up to 61% more advantage than the conventional approach.

• The Journal of Korean Institute of Communications and Information Sciences
• /
• v.41 no.11
• /
• pp.1490-1501
• /
• 2016

In this paper, we propose an image-based indoor localization system using parallel distributed computing. In order to reduce computation time for indoor localization, an scale invariant feature transform (SIFT) algorithm is performed in parallel by using Apache Spark. Toward this goal, we propose a novel image processing interface of Apache Spark. The experimental results show that the speed of the proposed system is about 3.6 times better than that of the conventional system.

• Journal of the Korean School Mathematics Society
• /
• v.11 no.3
• /
• pp.363-376
• /
• 2008

This study is to understand intuition that is the tool of invention and the one factor of the creative thinking in mathematical education. For this, I examine the nature of intuition and the history of research about intuition. And I study the result of research about intuition in cognitive psychological perspectives. This study brings to a focus in informational processing model. Informational processing model is similar to the mathematical problem solving process that is expressed linear process. Recently, parallel distributed processing models try to understand the nature of intuition. But any models cannot adequately explain the nature and the phenomena of illumination of intuition. Some scholars try to examine the intuition in mathematical education. But systematic and practical research is rare. So, I suggest the mathematical educational implications about intuition. Conclusively, it is necessary to systematic concern in intuition and the methods of improvement of intuition in mathematical education.

• The KIPS Transactions:PartA
• /
• v.8A no.4
• /
• pp.439-446
• /
• 2001

The main stream of parallel programming today is using synchronous algorithms, where processor synchronization for correct computation and workload balance are essential. Overall performance of the whole system is dependent upon the performance of the slowest processor, if workload is not well-balanced or heterogeneous clusters are used. Asynchronous iteration is a way to mitigate such problems, but most of the works done so far are for shared memory systems. In this paper, we suggest and implement a parallel large sparse linear system solver that improves performance on distributed memory systems like clusters by reducing processor idle times as much as possible by asynchronous iterations.

• Journal of the Korean Data and Information Science Society
• /
• v.24 no.5
• /
• pp.989-998
• /
• 2013

We cannot analyze big data, which attracts recent attentions in industry and academy, by batch processing algorithms developed in data mining because big data, by definition, cannot be uploaded and processed in the memory of a single system. So an imminent issue is to develop various leaning algorithms so that they can be applied to big data. In this paper, we review various algorithms for support vector machines in the literature. Particularly, we introduce online type and parallel processing algorithms that are expected to be useful in big data classifications and compare the strengths, the weaknesses and the performances of those algorithms through simulations for linear classification.