• Title/Summary/Keyword: File splitting

Search Result 10, Processing Time 0.022 seconds

Design and Implementation of Kernel-Level Split and Merge Operations for Efficient File Transfer in Cyber-Physical System (사이버 물리 시스템에서 효율적인 파일 전송을 위한 커널 레벨 분할 및 결합 연산의 설계와 구현)

  • Park, Hyunchan;Jang, Jun-Hee;Lee, Junseok
    • IEMEK Journal of Embedded Systems and Applications
    • /
    • v.14 no.5
    • /
    • pp.249-258
    • /
    • 2019
  • In the cyber-physical system, big data collected from numerous sensors and IoT devices is transferred to the Cloud for processing and analysis. When transferring data to the Cloud, merging data into one single file is more efficient than using the data in the form of split files. However, current merging and splitting operations are performed at the user-level and require many I / O requests to memory and storage devices, which is very inefficient and time-consuming. To solve this problem, this paper proposes kernel-level partitioning and combining operations. At the kernel level, splitting and merging files can be done with very little overhead by modifying the file system metadata. We have designed the proposed algorithm in detail and implemented it in the Linux Ext4 file system. In our experiments with the real Cloud storage system, our technique has achieved a transfer time of up to only 17% compared to the case of transferring split files. It also confirmed that the time required can be reduced by up to 0.5% compared to the existing user-level method.

The File Splitting Distribution Scheme Using the P2P Networks with The Mesh topology (그물망 위상의 P2P 네트워크를 활용한 파일 분리 분산 방안)

  • Lee Myoung-Hoon;Park Jung-Su;Kim Jin-Hong;Jo In-June
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.9 no.8
    • /
    • pp.1669-1675
    • /
    • 2005
  • Recently, the small sized wireless terminals have problems of processing of large sized file because of the trends of a small sized terminals and a large sized files. Moreover, the web servers or the file servers have problems of the overload because of the concentration with many number of files to the them. Also, There is a security vulnerability of the data processing caused by the processing with a unit of the independent file. To resolve the problems, this paper proposes a new scheme of fat splining distribution using the P2P networks with the mesh topology. The proposed scheme is to distribute blocks of file into any peer of P2P networks. It can do that the small sized wireless terminals can process the large size file, the overload problems of a web or file servers can solve because of the decentralized files, and, the security vulnerability of the data processing is mitigated because of the distributed processing with a unit of the blocks to the peers.

Reduction of processing time for concept printer with model splitting (모델 분할을 이용한 컨셉 프린터의 가공시간 단축)

  • Lee, In-Tak;Kim, Ho-Chan;Guen, Min-Hyung;Lee, Suk;Lee, Seok-Hee
    • Proceedings of the Korean Society of Precision Engineering Conference
    • /
    • 1997.10a
    • /
    • pp.105-108
    • /
    • 1997
  • To reduce design period, demands for rapid prototyping system is increasing. Among the rapid prototyping system, as a concept modeler, 3D Printer appears before the footlights. In spite of short production time and convenience, 3D Printer causes post processing problem. And that is too slow to adapt to the demand of workshop. In this research, we developed a system for reducing build time and roughness using splitting operation. Also, propose BFS(B-rep For STL) data structure to maintain model data, split STL model and implement Euler Operation for input and output.

  • PDF

Data Deduplication Method using Locality-based Chunking policy for SSD-based Server Storages (SSD 기반 서버급 스토리지를 위한 지역성 기반 청킹 정책을 이용한 데이터 중복 제거 기법)

  • Lee, Seung-Kyu;Kim, Ju-Kyeong;Kim, Deok-Hwan
    • Journal of the Institute of Electronics and Information Engineers
    • /
    • v.50 no.2
    • /
    • pp.143-151
    • /
    • 2013
  • NAND flash-based SSDs (Solid State Drive) have advantages of fast input/output performance and low power consumption so that they could be widely used as storages on tablet, desktop PC, smart-phone, and server. But, SSD has the disadvantage of wear-leveling due to increase of the number of writes. In order to improve the lifespan of the SSD, a variety of data deduplication techniques have been introduced. General fixed-size splitting method allocates fixed size of chunk without considering locality of data so that it may execute unnecessary chunking and hash key generation, and variable-size splitting method occurs excessive operation since it compares data byte-by-byte for deduplication. This paper proposes adaptive chunking method based on application locality and file name locality of written data in SSD-based server storage. The proposed method split data into 4KB or 64KB chunks adaptively according to application locality and file name locality of duplicated data so that it can reduce the overhead of chunking and hash key generation and prevent duplicated data writing. The experimental results show that the proposed method can enhance write performance, reduce power consumption and operation time compared to existing variable-size splitting method and fixed size splitting method using 4KB.

The Implementation Performance Evaluation of PR-File Based on Circular ar Domain (순환도메인을 기반으로 하는 PR-화일의 구현 및 성능 평가)

  • Kim, Hong-Ki;Hwang, Bu-Hyun
    • The Transactions of the Korea Information Processing Society
    • /
    • v.3 no.1
    • /
    • pp.63-76
    • /
    • 1996
  • In this paper, we propose a new dynamic spatial index structure, called PR -file, for handling spatial objects and the modified hierarchical variance which measures the degree of spatial locality at each level. Under the assumption that a multidimensional search space has a circular domain, PR-file uses the modified hierarchical variance for clustering spatially adjacent objects. The insertion and splitting algorithms of PR_file preserve and index which has a low hierarchical variance regardless of object distributions. The simulation result shows that PR- file has a high hit ratio during a retrieval of objects by using an index with low hierarchical variance. And it shows a characteristic that the larger the bucket capacity, the higher the bucket utilization.

  • PDF

Secure file distribution method using distribution P2P system (분산 P2P 시스템을 활용한 안전한 파일 분산 방안)

  • Kim, Jin-Hong;Kim, Seon-Young;Lee, Yoon-Jin;Jo, In-June
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • v.9 no.2
    • /
    • pp.545-548
    • /
    • 2005
  • Recently, the computing environment of P2P come out to solve the excessive load of the server in the computing environment of the client/server. Currently, operated computing environment of P2P is mainly spreading out P2P system of read name or anonymity base about a publisher, sender and receiver of the file. But, to the current file protection there is three problem. The first problem is to a host become attack target. The second is to received file loose attack of Brute Force. The third is to define target of attack of DOS. To solve the this problem. it divide file into block unit. Each block is safely scattered peers. This paper propose the distribution P2P system of file division. Both proposing system solve the this problem and promote efficiency of file application.

  • PDF

The Study on improve method of GNUnet for structured P2P Model of encording block (제안 인코딩 블록을 구조적 P2P 모델에 적용한 GNUnet 성능 개선방안)

  • Lee, Myoung-Hoon;Park, Byung-Yeon;Jo, In-June
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • v.9 no.2
    • /
    • pp.75-81
    • /
    • 2005
  • The GNUnet in P2P system have solved an anonymous(publisher, storer, demander); a service of condition equal for peer. The GNUnet for an anonymous has separated a file and dispersed to the network. But, the 1Kbyte block size of the GNUnet is a creation of many additional block. I and R block has created with indirect point of D block The waste of I and R block appeared to 4% of the original file and the additional network traffic for the block transmission. To resolve the problems, this paper proposes an new scheme of file splitting distribution using P2P networks with the new GNUnet protocol, The GNUnet support minimization of the additional block and for a network traffic. It proposed an efficiency improvement of encording block and routing algorithm.

  • PDF

Development of User-friendly Modeling Interface for Process-based Total System Performance Assessment Framework (APro) for Geological Disposal System of High-level Radioactive Waste (고준위폐기물 심층처분시스템에 대한 프로세스 기반 종합성능평가 체계(APro)의 사용자 친화적 모델링 인터페이스 개발)

  • Kim, Jung-Woo;Lee, Jaewon;Cho, Dong-Keun
    • Journal of Nuclear Fuel Cycle and Waste Technology(JNFCWT)
    • /
    • v.17 no.2
    • /
    • pp.227-234
    • /
    • 2019
  • A user-friendly modeling interface is developed for a process-based total system performance assessment framework (APro) specialized for a generic geological disposal system for high-level radioactive waste. The APro modeling interface is constructed using MATLAB, and the operator splitting scheme is used to combine COMSOL for simulation of multiphysics and PHREEQC for the calculation of geochemical reactions. As APro limits the modeling domain to the generic disposal system, the degree of freedom of the model is low. In contrast, the user-friendliness of the model is improved. Thermal, hydraulic, mechanical and chemical processes considered in the disposal system are modularized, and users can select one of multiple modules: "Default process" and multi "Alternative process". APro mainly consists of an input data part and calculation execution part. The input data are prepared in a single EXCEL file with a given format, and the calculation part is coded using MATLAB. The final results of the calculation are created as an independent COMSOL file for further analysis.

A Study on the Implement of Image Recognition the Road Traffic Safety Information Board using Nearest Neighborhood Decision Making Algorithm (최근접 이웃 결정방법 알고리즘을 이용한 도로교통안전표지판 영상인식의 구현)

  • Jung Jin-Yong;Kim Dong-Hyun;Lee So-Haeng
    • Management & Information Systems Review
    • /
    • v.4
    • /
    • pp.257-284
    • /
    • 2000
  • According as the drivers increase who have their cars, the comprehensive studies on the automobile for the traffic safety have been raised as the important problems. Visual Recognition System for radio-controled driving is a part of the sensor processor of Unmanned Autonomous Vehicle System. When a driver drives his car on an unknown highway or general road, it produces a model from the successively inputted road traffic information. The suggested Recognition System of the Road Traffic Safety Information Board is to recognize and distinguish automatically a Road Traffic Safety Information Board as one of road traffic information. The whole processes of Recognition System of the Road Traffic Safety Information Board suggested in this study are as follows. We took the photographs of Road Traffic Safety Information Board with a digital camera in order to get an image and normalize bitmap image file with a size of $200{\times}200$ byte with Photo Shop 5.0. The existing True Color is made up the color data of sixteen million kinds. We changed it with 256 Color, because it has large capacity, and spend much time on calculating. We have practiced works of 30 times with erosion and dilation algorithm to remove unnecessary images. We drawing out original image with the Region Splitting Technique as a kind of segmentation. We made three kinds of grouping(Attention Information Board, Prohibit Information Board, and Introduction Information Board) by RYB( Red, Yellow, Blue) color segmentation. We minimized the image size of board, direction, and the influence of rounding. We also minimized the Influence according to position. and the brightness of light and darkness with Eigen Vector and Eigen Value. The data sampling this feature value appeared after building the learning Code Book Database. The suggested Recognition System of the Road Traffic Safety Information Board firstly distinguished three kinds of groups in the database of learning Code Book, and suggested in order to recognize after comparing and judging the board want to recognize within the same group with Nearest Neighborhood Decision Making.

  • PDF

An Analysis of Big Video Data with Cloud Computing in Ubiquitous City (클라우드 컴퓨팅을 이용한 유시티 비디오 빅데이터 분석)

  • Lee, Hak Geon;Yun, Chang Ho;Park, Jong Won;Lee, Yong Woo
    • Journal of Internet Computing and Services
    • /
    • v.15 no.3
    • /
    • pp.45-52
    • /
    • 2014
  • The Ubiquitous-City (U-City) is a smart or intelligent city to satisfy human beings' desire to enjoy IT services with any device, anytime, anywhere. It is a future city model based on Internet of everything or things (IoE or IoT). It includes a lot of video cameras which are networked together. The networked video cameras support a lot of U-City services as one of the main input data together with sensors. They generate huge amount of video information, real big data for the U-City all the time. It is usually required that the U-City manipulates the big data in real-time. And it is not easy at all. Also, many times, it is required that the accumulated video data are analyzed to detect an event or find a figure among them. It requires a lot of computational power and usually takes a lot of time. Currently we can find researches which try to reduce the processing time of the big video data. Cloud computing can be a good solution to address this matter. There are many cloud computing methodologies which can be used to address the matter. MapReduce is an interesting and attractive methodology for it. It has many advantages and is getting popularity in many areas. Video cameras evolve day by day so that the resolution improves sharply. It leads to the exponential growth of the produced data by the networked video cameras. We are coping with real big data when we have to deal with video image data which are produced by the good quality video cameras. A video surveillance system was not useful until we find the cloud computing. But it is now being widely spread in U-Cities since we find some useful methodologies. Video data are unstructured data thus it is not easy to find a good research result of analyzing the data with MapReduce. This paper presents an analyzing system for the video surveillance system, which is a cloud-computing based video data management system. It is easy to deploy, flexible and reliable. It consists of the video manager, the video monitors, the storage for the video images, the storage client and streaming IN component. The "video monitor" for the video images consists of "video translater" and "protocol manager". The "storage" contains MapReduce analyzer. All components were designed according to the functional requirement of video surveillance system. The "streaming IN" component receives the video data from the networked video cameras and delivers them to the "storage client". It also manages the bottleneck of the network to smooth the data stream. The "storage client" receives the video data from the "streaming IN" component and stores them to the storage. It also helps other components to access the storage. The "video monitor" component transfers the video data by smoothly streaming and manages the protocol. The "video translator" sub-component enables users to manage the resolution, the codec and the frame rate of the video image. The "protocol" sub-component manages the Real Time Streaming Protocol (RTSP) and Real Time Messaging Protocol (RTMP). We use Hadoop Distributed File System(HDFS) for the storage of cloud computing. Hadoop stores the data in HDFS and provides the platform that can process data with simple MapReduce programming model. We suggest our own methodology to analyze the video images using MapReduce in this paper. That is, the workflow of video analysis is presented and detailed explanation is given in this paper. The performance evaluation was experiment and we found that our proposed system worked well. The performance evaluation results are presented in this paper with analysis. With our cluster system, we used compressed $1920{\times}1080(FHD)$ resolution video data, H.264 codec and HDFS as video storage. We measured the processing time according to the number of frame per mapper. Tracing the optimal splitting size of input data and the processing time according to the number of node, we found the linearity of the system performance.