• Title/Summary/Keyword: 데이터 레이크

Search Result 22, Processing Time 0.01 seconds

Apache NiFi-based ETL Process for Building Data Lakes (데이터 레이크 구축을 위한 Apache NiFi기반 ETL 프로세스)

  • Lee, Kyoung Min;Lee, Kyung-Hee;Cho, Wan-Sup
    • The Journal of Bigdata
    • /
    • v.6 no.1
    • /
    • pp.145-151
    • /
    • 2021
  • In recent years, digital data has been generated in all areas of human activity, and there are many attempts to safely store and process the data to develop useful services. A data lake refers to a data repository that is independent of the source of the data and the analytical framework that leverages the data. In this paper, we designed a tool to safely store various big data generated by smart cities in a data lake and ETL it so that it can be used in services, and a web-based tool necessary to use it effectively. Implement. A series of processes (ETLs) that quality-check and refine source data, store it safely in a data lake, and manage it according to data life cycle policies are often significant for costly infrastructure and development and maintenance. It is a labor-intensive technology. The mounting technology makes it possible to set and execute ETL work monitoring and data life cycle management visually and efficiently without specialized knowledge in the IT field. Separately, a data quality checklist guide is needed to store and use reliable data in the data lake. In addition, it is necessary to set and reserve data migration and deletion cycles using the data life cycle management tool to reduce data management costs.

Turbomachinery Inlet Flow Measurement without the Effect of Instrumentation (입구 Instrumentation의 영향을 최소화하는 터보기계 성능측정방법)

  • Kang, Jeong-Seek;Ahn, Iee-Ki
    • Aerospace Engineering and Technology
    • /
    • v.8 no.2
    • /
    • pp.8-12
    • /
    • 2009
  • It is absolutely necessary to measure the inlet pressure and temperature of a turbine or a compressor to evaluate the performance of it. And to measure the representative-averaged pressure and temperature of turbine inlet flow, rake is normally used. Rake has several elements for temperature and pressure and several rakes are installed at the inlet to average the radial and circumferential distribution of inlet flow. However the rakes cause unexpected losses and flow distortion at the turbine inlet which make the measured rake data different from true inlet value. So the evaluation of a turbine or a compressor performance becomes not accurate. This study suggest a correlation method which measure the loss by inlet rake and incorporates it in evaluating the performance of turbomachinery.

  • PDF

Optimum Rake Processing for Multipath Fading in Direct-Sequence Spread-Spectrum Communication Systems (주파수대역 직접확산 통신시스템에서 다중경로 페이딩 보상을 위한 최적 레이크 신호처리에 관한 연구)

  • 장원석;이재천
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.28 no.10C
    • /
    • pp.995-1006
    • /
    • 2003
  • It is well know that in the wireless communication systems the transmitted signals can suffer from multipath fading due to the wave propagation characteristics and the obstacles over the paths, resulting in serious reduction in the power of the received signals. However, it is possible to take advantage of the inherent diversity imposed in the multipath reception if the underlying channel can be properly estimated. One of the diversity reception methods in this case is Rake processing. In this paper we study the Rake receivers for the direct-sequence spread-spectrum communication systems utilizing PN (pseudo noise) sequences to achieve spread spectrum. A conventional Rake receiver can use the finite-duration impulse (FIR) filter followed by the PN sequence demodulator, where the FIR filter coefficients are the reverse-ordered complex conjugate values of the fading channel impulse response estimates. Here, we propose a new Rake processing method by replacing the aforementioned PN code sequence with a new set of optimum demodulator coefficients. More specifically, the concept of the new optimum Rake processing is first introduced and then the optimum demodulator coefficients are theoretically derived. The performance obtained using the new optimum Rake processing is also calculated. The analytical results are verified by computer simulation. As a result, it is shown that the new optimum Rake processing method improves the MSE performance more than 10 dB over the conventional one using the fixed PN sequence demodulator. It is also shown that the new optimum Rake processing method improves the MSE performance about 10 dB over the Adaptive Correlator that performs the combining of the multipath components and PN demodulation concurrently. And finally, the MSE performance of the optimum Rake demodulator is very close to the MSE performance of OPSK demodulator under the AWGN channel.

A Study on Data Resource Management Comparing Big Data Environments with Traditional Environments (전통적 환경과 빅데이터 환경의 데이터 자원 관리 비교 연구)

  • Park, Jooseok;Kim, Inhyun
    • The Journal of Bigdata
    • /
    • v.1 no.2
    • /
    • pp.91-102
    • /
    • 2016
  • In traditional environments we have called the data life cycle DIKW, which represents data-information-knowledge-wisdom. In big data environments, on the other hand, we call it DIA, which represents data-insight-action. The difference between the two data life cycles results in new architecture of data resource management. In this paper, we study data resource management architecture for big data environments. Especially main components of the architecture are proposed in this paper.

  • PDF

A Study on the Rake Finger System Design for the System Performance Improvement in the Mobile Communications (시스템 효율향상을 위한 이동통신망 Rake Finger 시스템 설계에 관한 연구)

  • Lee Seon-Keun;Lim Soon-Ja
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.29 no.1A
    • /
    • pp.31-36
    • /
    • 2004
  • In this paper, we proposed the new structure of the Rake Finger using Walsh Switch, the shared accumulator, and the pipeline-FWHT algorithm for reducing the signal processing complexity resulting from the increase of the number of data correlator. The function simulation of the proposed architecture is performed by Synopsys tool and the timing simulation is performed by Compass tool. The number of computational operation in the proposed data correlators is 160 additions and the conventional ones is 512 additions when the number of walsh code N=4. As a result, it is reduced about 3.2 times other than the number of computational operation of the conventional ones. Also, the result shows that the data processing time of the proposed Rake Finger architecture is 90,496[ns] and the conventional ones is 110,696[ns]. It is $18.3\%$ faster than the data processing time of the conventional Rake Finger architecture.

On the error rate of multicode-CDMA system in frequency selective fading channel (주파수 선택적 페이딩 채널에서 멀티코드 CDMA 시스템의 성능 분석)

  • 김연진;김남수;김민택
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.23 no.4
    • /
    • pp.932-939
    • /
    • 1998
  • In this paper, we analyze the performance of a multicode-CDMA system which have been proposed for the multimedia communications. The performance of a multicode-CDMA system, providing good spectrum efficiency as well as serving various bit rates, is analyzed with multipath, frequency selective, slowly fading Rayleigh channel. Also the proposed scheme adopting RAKE receiver with MRC(Maximal Ratio Combine) is advantageous to multipath channel. For a practical channel modeling, the JTC(Joint Technical Committee) recommended channel model(JTC(AIR) 23-065R6) is applied to simulation. The proposed schemehas serial-to-parallel convertor which splits input data stream of 2 Mits/s into 20 branches o 100 kbits/s. From the result of simulation, the case of RAKE receiver with 3 fingers to reduce the system complexity required the relatively large $E_{b}/N_O$ of 0 dB~1.5 dB, compared to the case of RAKE receiver with the number of path finger to keep the average error rate to be $1{\times}10^{-3}$ in channel A.

  • PDF

Symbol Timing Alignment and Combining Technique in Rake Receiver for cdma2000 Systems (cdma2000 시스템용 레이크 수신기에서의 심볼 정렬 및 컴바이닝 기법)

  • Lee, Seong-Ju;Kim, Jae-Seok;Eo, Ik-Su;Kim, Gyeong-Su
    • Journal of the Institute of Electronics Engineers of Korea TC
    • /
    • v.39 no.1
    • /
    • pp.34-41
    • /
    • 2002
  • In the conventional rake receiver structure for the IS-95 CDMA system, each finger has its own time-deskew buffer or FIFO that aligns the multipath signals to the same timing reference in order to combine symbols. This architecture is not a burden to the rake receiver design mainly because of the small number and size of the buffers. However, the number and size of the buffers are significantly increased in the cdma2000 system which adopts multiple carriers and the small spreading gain for a higher rate in data services. In order to decrease the number of buffers, we propose a new model of the time-deskew buffers, which combines the symbols as well as realigns them at the same time. Our architecture reduces the hardware complexity of the buffers by about more than 60% and 70% compared with the conventional one when we consider each rake receiver has three and four independent fingers, respectively. Moreover, the proposed algorithm is very useful not only to the cdma2000 rake receiver but also to the receiver with many fingers in order to increase the BER performance.

A Leading Study of Data Lake Platform based on Big Data to support Business Intelligence (Business Intelligence를 지원하기 위한 Big Data 기반 Data Lake 플랫폼의 선행 연구)

  • Lee, Sang-Beom
    • Proceedings of the Korean Society of Computer Information Conference
    • /
    • 2018.01a
    • /
    • pp.31-34
    • /
    • 2018
  • We live in the digital era, and the characteristics of our customers in the digital era are constantly changing. That's why understanding business requirements and converting them to technical requirements is essential, and you have to understand the data model behind the business layout. Moreover, BI(Business Intelligence) is at the crux of revolutionizing enterprise to minimize losses and maximize profits. In this paper, we have described a leading study about the situation of desk-top BI(software product & programming language) in aspect of front-end side and the Data Lake platform based on Big Data by data modeling in aspect of back-end side to support the business intelligence.

  • PDF

Fishery R&D Big Data Platform and Metadata Management Strategy (수산과학 빅데이터 플랫폼 구축과 메타 데이터 관리방안)

  • Kim, Jae-Sung;Choi, Youngjin;Han, Myeong-Soo;Hwang, Jae-Dong;Cho, Wan-Sup
    • The Journal of Bigdata
    • /
    • v.4 no.2
    • /
    • pp.93-103
    • /
    • 2019
  • In this paper, we introduce a big data platform and a metadata management technique for fishery science R & D information. The big data platform collects and integrates various types of fisheries science R & D information and suggests how to build it in the form of a data lake. In addition to existing data collected and accumulated in the field of fisheries science, we also propose to build a big data platform that supports diverse analysis by collecting unstructured big data such as satellite image data, research reports, and research data. Next, by collecting and managing metadata during data extraction, preprocessing and storage, systematic management of fisheries science big data is possible. By establishing metadata in a standard form along with the construction of a big data platform, it is meaningful to suggest a systematic and continuous big data management method throughout the data lifecycle such as data collection, storage, utilization and distribution.

  • PDF