• Title/Summary/Keyword: Big data Processing

Search Result 1,063, Processing Time 0.035 seconds

Multi-channel data connection and Real-time processing system designed for Big Data collection (빅데이터 수집을 위한 다채널 데이터 연계와 실시간 처리 시스템 설계)

  • Paik, Kyoung-Seok;Oh, Jae-Chel;Yang, Jae-Hyek
    • Proceedings of the Korea Contents Association Conference
    • /
    • 2016.05a
    • /
    • pp.269-270
    • /
    • 2016
  • 빅데이터 분석을 통한 여러 산업 군과 융합으로 시너지를 발생시키기 위해서, 다양한 유형의 데이터 수집을 통해 빅데이터를 구성하는 것이 첫 번째 단계이며 기상, 교통, 인터넷 활동, 상권 등의 다양한 출처로부터 데이터 연계를 수행하고 사물인터넷과 같은 실시간으로 발생하는 로그 성 데이터 수집을 고려한 실시간 처리 시스템을 설계 하였다. 이를 통해 서로 다른 유형의 데이터가 빅데이터로 수집 되면 여러 산업 군에서 요구되는 인사이트 기반의 빅데이터 분석을 통해 B2B 또는 B2C 서비스에 응용 될 수 있다.

  • PDF

Survey on Publish/Subscribe Communication Technologies based on Information Centric Networking (정보중심네트워크 기반의 Pub/Sub 통신 연구동향)

  • Jung, H.Y.;Kim, S.M.
    • Electronics and Telecommunications Trends
    • /
    • v.33 no.5
    • /
    • pp.86-94
    • /
    • 2018
  • Information-Centric Networking (ICN) has been recognized as a new networking technology for the upcoming data-centric 4th industrial revolution based society. In addition, it has noted that Pub/Sub-style communication is rapidly growing in areas including big data processing and microservice as well as the existing Internet of Things and social networking technologies. Therefore, ICN is highly needed to efficiently support Pub/Sub-style communication for successful deployment as a next-generation network infrastructure technology. This paper summarizes the recent research trends of Pub/Sub communication technologies over ICN, and discusses future research issues.

Green Employee Empowerment? Driving and Inhibiting Factors for Green Employee Performance

  • ADI, Nyoman Rasmen;MULYADI, Made;SETINI, Made;ASTAWA, Nengah Dasi
    • The Journal of Asian Finance, Economics and Business
    • /
    • v.8 no.5
    • /
    • pp.293-302
    • /
    • 2021
  • Being able to survive during the Covid19 pandemic is a big task for a company, as such, empowerment of employees is a must. The sample in this study was 300 employees who worked in Spas throughout Bali. The sampling technique was purposive sampling. Data analysis was using SEM and SMARTS as data processing. The results showed that green communication, namely communication that occurs between employees, superiors, and the environment, has a very good influence on the sustainability of employee performance. To become green management, a green organizational commitment that cares about the safety of employee health and the environment is an important factor as motivation in green dedication or positive employee productivity, but communication between work actors and justice is also a motivating factor. Work safety and job security for employees empower employees (which is a green line), especially for freelance work organizations so that further research in subsequent studies can make samples in a more varied industrial sector.

Machine Learning based Bandwidth Prediction for Dynamic Adaptive Streaming over HTTP

  • Yoo, Soyoung;Kim, Gyeongryeong;Kim, Minji;Kim, Yeonjin;Park, Soeun;Kim, Dongho
    • Journal of Advanced Information Technology and Convergence
    • /
    • v.10 no.2
    • /
    • pp.33-48
    • /
    • 2020
  • By Digital Transformation, new technologies like ML (Machine Learning), Big Data, Cloud, VR/AR are being used to video streaming technology. We choose ML to provide optimal QoE (Quality of Experience) in various network conditions. In other words, ML helps DASH in providing non-stopping video streaming. In DASH, the source video is segmented into short duration chunks of 2-10 seconds, each of which is encoded at several different bitrate levels and resolutions. We built and compared the performances of five prototypes after applying five different machine learning algorithms to DASH. The prototype consists of a dash.js, a video processing server, web servers, data sets, and five machine learning models.

Development of a displacement measurement system for architectural structures using artificial intelligence techniques (인공지능 기법을 활용한 건축 구조물 변위측정시스템 개발)

  • Kang, Ye-Jin;Kim, Dae-Geon;Woo, Jong-Yeol;Lee, Dong-Oun
    • Proceedings of the Korean Institute of Building Construction Conference
    • /
    • 2022.04a
    • /
    • pp.135-136
    • /
    • 2022
  • As a recent technology, it is possible to partially grasp the occurrence of displacement of the entire building through artificial intelligence technology for big data through scanning. However, scanning and data processing take a lot of time, so there is a limit to constant monitoring, so constant monitoring technology of building behavior that combines wireless remote sensors and 3D shape scanning is required. Therefore, in this study, artificial intelligence program coding technology is linked. In addition, a technology capable of real-time wireless remote measurement of structure displacement will be developed through technology development in response to safety management that combines existing building technologies such as sensors. Through this, it is possible to establish an integrated management system for safety inspection and diagnosis.

  • PDF

Novel Kernel Design for Implementing Volume Rendering in the PyCUDA Framework (PyCUDA 프레임워크에서 볼륨 렌더링을 구현하기 위한 새로운 커널 디자인)

  • Lee, SooHo;Kim, Jong-Hyun
    • Proceedings of the Korean Society of Computer Information Conference
    • /
    • 2022.01a
    • /
    • pp.349-351
    • /
    • 2022
  • 본 논문에서는 계산양이 큰 볼륨 렌더링을 구현할 수 있는 파이썬 기반의 CUDA(Computed Unified Device Architecture) 커널(Kernel) 디자인에 대해서 소개한다. 최근에 파이썬은 인공지능뿐만 아니라 서버, 보안, GUI, 데이터 시각화, 빅 데이터 처리 등 다양한 분야에서 활용이 되고 있기 때문에 인터페이스만을 위한 언어라는 색을 탈피한지 오래이다. 본 논문에서는 대용량 병렬처리 기법인 NVIDIA의 CUDA를 이용하여 파이썬 환경에서 커널을 디자인하고, 계산양이 큰 볼륨 렌더링이 빠르게 계산되는 결과를 보여준다. 결과적으로 C언어 기반의 CUDA뿐만 아니라, 상대적으로 개발이 효율적인 파이썬 환경에서도 GPU(Graphic Processing Unit)기반 애플리케이션 개발이 가능하다는 것을 볼륨 렌더링을 통해 보여준다.

  • PDF

A Study of Data Collection Method for Efficient Sharing in IoT Environment (사물인터넷(IoT) 환경에서 효율적 공유를 위한 데이터 수집 기법에 대한 연구)

  • Hwang, Chi-Gon;Yoon, Chang-Pyo
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2015.10a
    • /
    • pp.268-269
    • /
    • 2015
  • The current Internet environment, it is accessible by a computer, but also transferred to the IoT(Internet of Things). These data become large. If the data are provided to the application without any adjustment, it is difficult to exert the original performance. In this paper, we propose a method for filtering the data using the MapReduce of big data processing techniques to refine the collected data. We want to address the heterogeneity of the data generated by the sensor by adding a knowledge identification step in MapReduce. We use XMDR for this purpose.

  • PDF

BDSS: Blockchain-based Data Sharing Scheme With Fine-grained Access Control And Permission Revocation In Medical Environment

  • Zhang, Lejun;Zou, Yanfei;Yousuf, Muhammad Hassam;Wang, Weizheng;Jin, Zilong;Su, Yansen;Kim, Seokhoon
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.16 no.5
    • /
    • pp.1634-1652
    • /
    • 2022
  • Due to the increasing need for data sharing in the age of big data, how to achieve data access control and implement user permission revocation in the blockchain environment becomes an urgent problem. To solve the above problems, we propose a novel blockchain-based data sharing scheme (BDSS) with fine-grained access control and permission revocation in this paper, which regards the medical environment as the application scenario. In this scheme, we separate the public part and private part of the electronic medical record (EMR). Then, we use symmetric searchable encryption (SSE) technology to encrypt these two parts separately, and use attribute-based encryption (ABE) technology to encrypt symmetric keys which used in SSE technology separately. This guarantees better fine-grained access control and makes patients to share data at ease. In addition, we design a mechanism for EMR permission grant and revocation so that hospital can verify attribute set to determine whether to grant and revoke access permission through blockchain, so it is no longer necessary for ciphertext re-encryption and key update. Finally, security analysis, security proof and performance evaluation demonstrate that the proposed scheme is safe and effective in practical applications.

Manhole Cover Detection from Natural Scene Based on Imaging Environment Perception

  • Liu, Haoting;Yan, Beibei;Wang, Wei;Li, Xin;Guo, Zhenhui
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.13 no.10
    • /
    • pp.5095-5111
    • /
    • 2019
  • A multi-rotor Unmanned Aerial Vehicle (UAV) system is developed to solve the manhole cover detection problem for the infrastructure maintenance in the suburbs of big city. The visible light sensor is employed to collect the ground image data and a series of image processing and machine learning methods are used to detect the manhole cover. First, the image enhancement technique is employed to improve the imaging effect of visible light camera. An imaging environment perception method is used to increase the computation robustness: the blind Image Quality Evaluation Metrics (IQEMs) are used to percept the imaging environment and select the images which have a high imaging definition for the following computation. Because of its excellent processing effect the adaptive Multiple Scale Retinex (MSR) is used to enhance the imaging quality. Second, the Single Shot multi-box Detector (SSD) method is utilized to identify the manhole cover for its stable processing effect. Third, the spatial coordinate of manhole cover is also estimated from the ground image. The practical applications have verified the outdoor environment adaptability of proposed algorithm and the target detection correctness of proposed system. The detection accuracy can reach 99% and the positioning accuracy is about 0.7 meters.

Dynamical Polynomial Regression Prefetcher for DRAM-PCM Hybrid Main Memory (DRAM-PCM 하이브리드 메인 메모리에 대한 동적 다항식 회귀 프리페처)

  • Zhang, Mengzhao;Kim, Jung-Geun;Kim, Shin-Dug
    • Annual Conference of KIPS
    • /
    • 2020.11a
    • /
    • pp.20-23
    • /
    • 2020
  • This research is to design an effective prefetching method required for DRAM-PCM hybrid main memory systems especially used for big data applications and massive-scale computing environment. Conventional prefetchers perform well with regular memory access patterns. However, workloads such as graph processing show extremely irregular memory access characteristics and thus could not be prefetched accurately. Therefore, this research proposes an efficient dynamical prefetching algorithm based on the regression method. We have designed an intelligent prefetch engine that can identify the characteristics of the memory access sequences. It can perform regular, linear regression or polynomial regression predictive analysis based on the memory access sequences' characteristics, and dynamically determine the number of pages required for prefetching. Besides, we also present a DRAM-PCM hybrid memory structure, which can reduce the energy cost and solve the conventional DRAM memory system's thermal problem. Experiment result shows that the performance has increased by 40%, compared with the conventional DRAM memory structure.