• Title/Summary/Keyword: Map-Reduce

Search Result 849, Processing Time 0.029 seconds

Big Data Platform Based on Hadoop and Application to Weight Estimation of FPSO Topside

  • Kim, Seong-Hoon;Roh, Myung-Il;Kim, Ki-Su;Oh, Min-Jae
    • Journal of Advanced Research in Ocean Engineering
    • /
    • v.3 no.1
    • /
    • pp.32-40
    • /
    • 2017
  • Recently, the amount of data to be processed and the complexity thereof have been increasing due to the development of information and communication technology, and industry's interest in such big data is increasing day by day. In the shipbuilding and offshore industry also, there is growing interest in the effective utilization of data, since various and vast amounts of data are being generated in the process of design, production, and operation. In order to effectively utilize big data in the shipbuilding and offshore industry, it is necessary to store and process large amounts of data. In this study, it was considered efficient to apply Hadoop and R, which are mostly used in big data related research. Hadoop is a framework for storing and processing big data. It provides the Hadoop Distributed File System (HDFS) for storing big data, and the MapReduce function for processing. Meanwhile, R provides various data analysis techniques through the language and environment for statistical calculation and graphics. While Hadoop makes it is easy to handle big data, it is difficult to finely process data; and although R has advanced analysis capability, it is difficult to use to process large data. This study proposes a big data platform based on Hadoop for applications in the shipbuilding and offshore industry. The proposed platform includes the existing data of the shipyard, and makes it possible to manage and process the data. To check the applicability of the platform, it is applied to estimate the weights of offshore structure topsides. In this study, we store data of existing FPSOs in Hadoop-based Hortonworks Data Platform (HDP), and perform regression analysis using RHadoop. We evaluate the effectiveness of large data processing by RHadoop by comparing the results of regression analysis and the processing time, with the results of using the conventional weight estimation program.

Noise Source Identification of Electric Parking Brake by Using Noise Contribution Analysis and Identifying Resonance of Vehicle System (차량 시스템의 소음 기여도분석 및 공진 규명을 통한 전자식 주차 브레이크 소음원 규명)

  • Park, Goon-Dong;Seo, Bum-June;Yang, In-Hyung;Jeong, Jae-Eun;Oh, Jae-Eung;Lee, Jung-Youn
    • Transactions of the Korean Society of Automotive Engineers
    • /
    • v.20 no.3
    • /
    • pp.119-125
    • /
    • 2012
  • Caliper intergrated Electric Parking Brake (EPB) is an automatic parking brake system, attached to rear caliper. Because EPB uses luxury vehicles recently, the drivers of vehicles are sensitive to the EPB noise. EPB is operated by the motor and gear, so noise is generated by motor and gear. In order to reduce noise, One of EPB manufacturers uses helical gear and changes the shape of EPB housing. But these methods are not optimized for reduction of interior noise. There are many noise transfer paths into vehicle interior and it is difficult to identify the noise sources. Therefore, in this study, we performed contribution analysis and modal testing in the vehicle system. It is possible to distinguish between air-borne noise and structure-borne noise in the vehicle interior noise by comparing interior noise peak with resonance mode map.

A Study on Fabric Color Mapping for 2D Virtual Wearing System (2D 가상 착의 시스템의 직물 컬러 매핑에 관한 연구)

  • Kwak, No-Yoon
    • Journal of Digital Contents Society
    • /
    • v.7 no.4
    • /
    • pp.287-294
    • /
    • 2006
  • Mass-customization is fast growing a segment of the apparel market. 2D Virtual wearing system is one of visual support tools that make possible to sell apparel before producing and reduce the time and costs related to product development and manufacturing in the world of apparel mass-customization. This paper is related to fabric color mapping method for 2D image-based virtual wearing system. In proposed method, clothing shape section of interest is segmented from a clothes model image using a region growing method, and then mapping a new fabric color selected by user into it based on its intensity difference map is processed. With the proposed method in 2D virtual wearing system, regardless of color or intensity of model clothes, it is possible to virtually change the fabric color with holding the illumination and shading properties of the selected clothing shape section, and also to quickly and easily simulate, compare, and select multiple fabric color combinations for individual styles or entire outfits.

  • PDF

Automatic Segmentation of Renal Parenchyma using Graph-cuts with Shape Constraint based on Multi-probabilistic Atlas in Abdominal CT Images (복부 컴퓨터 단층촬영영상에서 다중 확률 아틀라스 기반 형상제한 그래프-컷을 사용한 신실질 자동 분할)

  • Lee, Jaeseon;Hong, Helen;Rha, Koon Ho
    • Journal of the Korea Computer Graphics Society
    • /
    • v.22 no.4
    • /
    • pp.11-19
    • /
    • 2016
  • In this paper, we propose an automatic segmentation method of renal parenchyma on abdominal CT image using graph-cuts with shape constraint based on multi-probabilistic atlas. The proposed method consists of following three steps. First, to use the various shape information of renal parenchyma, multi-probabilistic atlas is generated by cortex-based similarity registration. Second, initial seeds for graph-cuts are extracted by maximum a posteriori (MAP) estimation and renal parenchyma is segmented by graph-cuts with shape constraint. Third, to reduce alignment error of probabilistic atlas and increase segmentation accuracy, registration and segmentation are iteratively performed. To evaluate the performance of proposed method, qualitative and quantitative evaluation are performed. Experimental results show that the proposed method avoids a leakage into neighbor regions with similar intensity of renal parenchyma and shows improved segmentation accuracy.

The Bigdata Processing Environment Building for the Learning System (학습 시스템을 위한 빅데이터 처리 환경 구축)

  • Kim, Young-Geun;Kim, Seung-Hyun;Jo, Min-Hui;Kim, Won-Jung
    • The Journal of the Korea institute of electronic communication sciences
    • /
    • v.9 no.7
    • /
    • pp.791-797
    • /
    • 2014
  • In order to create an environment for Apache Hadoop for parallel distributed processing system of Bigdata, by connecting a plurality of computers, or to configure the node, using the configuration of the virtual nodes on a single computer it is necessary to build a cloud fading environment. However, be constructed in practice for education in these systems, there are many constraints in terms of cost and complex system configuration. Therefore, it is possible to be used as training for educational institutions and beginners in the field of Bigdata processing, development of learning systems and inexpensive practical is urgent. Based on the Raspberry Pi board, training and analysis of Big data processing, such as Hadoop and NoSQL is now the design and implementation of a learning system of parallel distributed processing of possible Bigdata in this study. It is expected that Bigdata parallel distributed processing system that has been implemented, and be a useful system for beginners who want to start a Bigdata and education.

Large Scale Incremental Reasoning using SWRL Rules in a Distributed Framework (분산 처리 환경에서 SWRL 규칙을 이용한 대용량 점증적 추론 방법)

  • Lee, Wan-Gon;Bang, Sung-Hyuk;Park, Young-Tack
    • Journal of KIISE
    • /
    • v.44 no.4
    • /
    • pp.383-391
    • /
    • 2017
  • As we enter a new era of Big Data, the amount of semantic data has rapidly increased. In order to derive meaningful information from this large semantic data, studies that utilize the SWRL(Semantic Web Rule Language) are being actively conducted. SWRL rules are based on data extracted from a user's empirical knowledge. However, conventional reasoning systems developed on single machines cannot process large scale data. Similarly, multi-node based reasoning systems have performance degradation problems due to network shuffling. Therefore, this paper overcomes the limitations of existing systems and proposes more efficient distributed inference methods. It also introduces data partitioning strategies to minimize network shuffling. In addition, it describes a method for optimizing the incremental reasoning process through data selection and determining the rule order. In order to evaluate the proposed methods, the experiments were conducted using WiseKB consisting of 200 million triples with 83 user defined rules and the overall reasoning task was completed in 32.7 minutes. Also, the experiment results using LUBM bench datasets showed that our approach could perform reasoning twice as fast as MapReduce based reasoning systems.

An Analysis of Utilization on Virtualized Computing Resource for Hadoop and HBase based Big Data Processing Applications (Hadoop과 HBase 기반의 빅 데이터 처리 응용을 위한 가상 컴퓨팅 자원 이용률 분석)

  • Cho, Nayun;Ku, Mino;Kim, Baul;Xuhua, Rui;Min, Dugki
    • Journal of Information Technology and Architecture
    • /
    • v.11 no.4
    • /
    • pp.449-462
    • /
    • 2014
  • In big data era, there are a number of considerable parts in processing systems for capturing, storing, and analyzing stored or streaming data. Unlike traditional data handling systems, a big data processing system needs to concern the characteristics (format, velocity, and volume) of being handled data in the system. In this situation, virtualized computing platform is an emerging platform for handling big data effectively, since virtualization technology enables to manage computing resources dynamically and elastically with minimum efforts. In this paper, we analyze resource utilization of virtualized computing resources to discover suitable deployment models in Apache Hadoop and HBase-based big data processing environment. Consequently, Task Tracker service shows high CPU utilization and high Disk I/O overhead during MapReduce phases. Moreover, HRegion service indicates high network resource consumption for transfer the traffic data from DataNode to Task Tracker. DataNode shows high memory resource utilization and Disk I/O overhead for reading stored data.

Fast Coding Mode Decision for Temporal Scalability in H.264/AVC Scalable Extension (시간적 계층에서의 스케일러블 부호화 고속 모드 결정 방법)

  • Jeon, Byeungwoo
    • The Journal of Korea Institute of Information, Electronics, and Communication Technology
    • /
    • v.6 no.2
    • /
    • pp.71-75
    • /
    • 2013
  • Recently proliferating heterogeneous multimedia service environments should be able to deal with many different transmission speeds, image sizes, or qualities of video. However, not many existing video compression standards satisfy those necessities. To satisfy the functional requirements, the standardization of the H.264/AVC Scalable Extension (SE) technique has been recently completed. It is an extension of the H.264/AVC which can encode several image sizes and qualities at the same time as a single bitstream. To perform optimum mode decision, motion estimation is performed for all MB modes, and the RD costs are compared to identify an MB mode with the smallest RD cost. This increases computational complexity of H.264/AVC SE encoding. In this paper, we propose an early skip mode detection scheme to reduce candidate modes and suggest an algorithm of fast mode decision utilizing reference modes according to the mode history.

Dosimetric Analysis on the Effect of Target Motion in the Delivery of Conventional IMRT, RapidArc and Tomotherapy

  • Song, Ju-Young
    • Progress in Medical Physics
    • /
    • v.28 no.4
    • /
    • pp.164-170
    • /
    • 2017
  • One of the methods to consider the effect of respiratory motion of a tumor target in radiotherapy is to establish a treatment plan with the internal target volume (ITV) created based on an accurate analysis of the target motion displacement. When this method is applied to intensity modulated radiotherapy (IMRT), it is expected to yield a different treatment dose distribution under the motion condition according to the IMRT method. In this study, we prepared ITV-based IMRT plans with conventional IMRT using fixed gantry angle beams, RapidArc using volumetric modulated arc therapy, and tomotherapy using helical therapy. Then, the variation in dose distribution caused by the target motion was analyzed by the dose measurement in the actual motion condition. A delivery quality assurance plan was prepared for the established IMRT plan and the dose distribution in the actual motion condition was measured and analyzed using a two-dimensional diode detector placed on a moving phantom capable of simulating breathing movements. The dose measurement was performed considering only a uniform target shape and motion in the superior-inferior (SI) direction. In this condition, it was confirmed that the error of the dose distribution due to the target motion is minimum in tomotherapy. This is thought to be due to the characteristic of tomotherapy that treats the target sequentially by dividing it into several slices. When the target shape is uniform and the main target motion direction is SI, it is considered that tomotherapy for the ITV-based IMRT method has a characteristic which can reduce the dose difference compared with the plan dose under the target motion condition.

A Hadoop-based Multimedia Transcoding System for Processing Social Media in the PaaS Platform of SMCCSE

  • Kim, Myoungjin;Han, Seungho;Cui, Yun;Lee, Hanku;Jeong, Changsung
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.6 no.11
    • /
    • pp.2827-2848
    • /
    • 2012
  • Previously, we described a social media cloud computing service environment (SMCCSE). This SMCCSE supports the development of social networking services (SNSs) that include audio, image, and video formats. A social media cloud computing PaaS platform, a core component in a SMCCSE, processes large amounts of social media in a parallel and distributed manner for supporting a reliable SNS. Here, we propose a Hadoop-based multimedia system for image and video transcoding processing, necessary functions of our PaaS platform. Our system consists of two modules, including an image transcoding module and a video transcoding module. We also design and implement the system by using a MapReduce framework running on a Hadoop Distributed File System (HDFS) and the media processing libraries Xuggler and JAI. In this way, our system exponentially reduces the encoding time for transcoding large amounts of image and video files into specific formats depending on user-requested options (such as resolution, bit rate, and frame rate). In order to evaluate system performance, we measure the total image and video transcoding time for image and video data sets, respectively, under various experimental conditions. In addition, we compare the video transcoding performance of our cloud-based approach with that of the traditional frame-level parallel processing-based approach. Based on experiments performed on a 28-node cluster, the proposed Hadoop-based multimedia transcoding system delivers excellent speed and quality.