• Title/Summary/Keyword: Big data Processing

Search Result 1,063, Processing Time 0.035 seconds

The Analysis of the GPS Data Processing of the NGII CORS by Bernese and TGO (Bernese와 TGO에 의한 국내 GPS 상시관측소 자료처리 결과 분석)

  • Kim, Ji-Woon;Kwon, Jay-Hyoun;Lee, Ji-Sun
    • Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography
    • /
    • v.26 no.6
    • /
    • pp.549-559
    • /
    • 2008
  • This study verified the limitations of commercial GPS data processing software and the applicability on precise positioning through comparing the processing results between Bernese and TGO under various conditions. To achieve the goal, we selected three nationwide station data and two smaller local data to constitute networks. By using Bernese and TGO, those networks are processed through the baseline analysis and the network adjustment. The comparative analysis was carried out, in terms of software, baseline length and network scale, observation duration, and number of fixed points. In the comparison between softwares, the scientific software was excellent in accuracy. It was confirmed that, as GPS-related technology is developed, the performance of the receiver was enhanced. And, in parallel with this, even the functionalities of the commercial software were tremendously enhanced. The difference, however, in result between the scientific and commercial software are still exist even if it is not big. Therefore, this study confirms that the scientific software should be used when the most precise position is necessary to be computed, especially if baseline vectors are big.

A System Design for Real-Time Monitoring of Patient Waiting Time based on Open-Source Platform (오픈소스 플랫폼 기반의 실시간 환자 대기시간 모니터링 시스템 설계)

  • Ryu, Wooseok
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.22 no.4
    • /
    • pp.575-580
    • /
    • 2018
  • This paper discusses system for real-time monitoring of patient waiting time in hospitals based on open-source platform. It is necessary to make use of open-source projects to develop a high-performance stream processing system, which analyzes and processes stream data in real time, with less cost. The Hadoop ecosystem is a well-known big data processing platform consisting of numerous open-source subprojects. This paper first defines several requirements for the monitoring system, and selects a few projects from the Hadoop ecosystem that are suited to meet the requirements. Then, the paper proposes system architecture and a detailed module design using Apache Spark, Apache Kafka, and so on. The proposed system can reduce development costs by using open-source projects and by acquiring data from legacy hospital information system. High-performance and fault-tolerance of the system can also be achieved through distributed processing.

Development of Internet of Things Sensor-based Information System Robust to Security Attack (보안 공격에 강인한 사물인터넷 센서 기반 정보 시스템 개발)

  • Yun, Junhyeok;Kim, Mihui
    • Journal of Internet Computing and Services
    • /
    • v.23 no.4
    • /
    • pp.95-107
    • /
    • 2022
  • With the rapid development of Internet of Things sensor devices and big data processing techniques, Internet of Things sensor-based information systems have been applied in various industries. Depending on the industry in which the information systems are applied, the accuracy of the information derived can affect the industry's efficiency and safety. Therefore, security techniques that protect sensing data from security attacks and enable information systems to derive accurate information are essential. In this paper, we examine security threats targeting each processing step of an Internet of Things sensor-based information system and propose security mechanisms for each security threat. Furthermore, we present an Internet of Things sensor-based information system structure that is robust to security attacks by integrating the proposed security mechanisms. In the proposed system, by applying lightweight security techniques such as a lightweight encryption algorithm and obfuscation-based data validation, security can be secured with minimal processing delay even in low-power and low-performance IoT sensor devices. Finally, we demonstrate the feasibility of the proposed system by implementing and performance evaluating each security mechanism.

A Study on Ontology Generation by Machine Learning in Big Data (빅 데이터에서 기계학습을 통한 온톨로지 생성에 관한 연구)

  • Hwang, Chi-Gon;Yoon, Chang-Pyo
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2018.10a
    • /
    • pp.645-646
    • /
    • 2018
  • Recently, the concept of machine learning has been introduced as a decision making method through data processing. Machine learning uses the results of running based on existing data as a means of decision making. The data generated by the development of technology is vast. This data is called big data. It is important to extract the necessary data from these data. In this paper, we propose a method for extracting related data for constructing an ontology through machine learning. The results of machine learning can be given a relationship from a semantic perspective. it can be added to the ontology to support relationships depending on the needs of the application.

  • PDF

Anomaly Detection of Hadoop Log Data Using Moving Average and 3-Sigma (이동 평균과 3-시그마를 이용한 하둡 로그 데이터의 이상 탐지)

  • Son, Siwoon;Gil, Myeong-Seon;Moon, Yang-Sae;Won, Hee-Sun
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.5 no.6
    • /
    • pp.283-288
    • /
    • 2016
  • In recent years, there have been many research efforts on Big Data, and many companies developed a variety of relevant products. Accordingly, we are able to store and analyze a large volume of log data, which have been difficult to be handled in the traditional computing environment. To handle a large volume of log data, which rapidly occur in multiple servers, in this paper we design a new data storage architecture to efficiently analyze those big log data through Apache Hive. We then design and implement anomaly detection methods, which identify abnormal status of servers from log data, based on moving average and 3-sigma techniques. We also show effectiveness of the proposed detection methods by demonstrating that our methods identifies anomalies correctly. These results show that our anomaly detection is an excellent approach for properly detecting anomalies from Hadoop log data.

Comparison of Student Churning Prediction Models based on Deep Learning Algorithms (딥러닝 알고리즘에 기반한 퇴원 학생 예측모델 비교)

  • Ko, Young-Sang;Lim, Heui-Seok
    • Annual Conference of KIPS
    • /
    • 2019.10a
    • /
    • pp.833-835
    • /
    • 2019
  • 교육열이 강한 우리나라에서는 사교육은 언제나 뜨거운 감자이다. 교육대상 연령층의 인구수가 1990 년부터 빠르게 감소하기 시작했으며, 2005 년을 전후로 초등학생 수의 감소가 더욱 빨라지고 있다. 통계청 데이터에 따르면 2016 년 출생아 수는 40 만 6 천여명에서 2017 년은 35 만 7 천여명으로 향후에도 지속적으로 줄어들 추세이다. 이렇듯 매년 학생수가 감소함에도 불구하고 2018 년 사교육비 총액은 19 조 5 천억수준으로 2017 년 18 조 7 천억보다 8 천억원이 늘어 났다. 학생수는 전년보다 2.5% 줄었지만 사교육비는 반대로 4.4% 늘어났다. 이렇듯 사교육 시장이 심화 되게 되면 경쟁은 더욱 치열해 질 수 밖에 없으며 이 경쟁에서 살아 남기 위해서는 다양한 비즈니스 전략이 필요하며 특히 학생들의 이탈을 줄이는 것은 사업의 가장 중요한 포인트라고 볼 수 있을 것이다. 학원에서의 학생이 퇴원을 하는 이유에 대한 영향도를 분석하고 그 영향도 분석을 통해 학원 학생들의 퇴원 방지에 활용하고자 한다. 본 논문의 주요 연구 내용은 사교육을 대표하는 국내 사설 학원에서의 성적, 출결사항 및 학원 상담 내역 등의 다양한 학원 데이터들을 최적의 딥러닝 알고리즘 분석을 통한 퇴원 학생을 사전 예측하기 위한 논문임을 밝힌다.

A Study on the Web Building Assistant System Using GUI Object Detection and Large Language Model (웹 구축 보조 시스템에 대한 GUI 객체 감지 및 대규모 언어 모델 활용 연구)

  • Hyun-Cheol Jang;Hyungkuk Jang
    • Annual Conference of KIPS
    • /
    • 2024.05a
    • /
    • pp.830-833
    • /
    • 2024
  • As Large Language Models (LLM) like OpenAI's ChatGPT[1] continue to grow in popularity, new applications and services are expected to emerge. This paper introduces an experimental study on a smart web-builder application assistance system that combines Computer Vision with GUI object recognition and the ChatGPT (LLM). First of all, the research strategy employed computer vision technology in conjunction with Microsoft's "ChatGPT for Robotics: Design Principles and Model Abilities"[2] design strategy. Additionally, this research explores the capabilities of Large Language Model like ChatGPT in various application design tasks, specifically in assisting with web-builder tasks. The study examines the ability of ChatGPT to synthesize code through both directed prompts and free-form conversation strategies. The researchers also explored ChatGPT's ability to perform various tasks within the builder domain, including functions and closure loop inferences, basic logical and mathematical reasoning. Overall, this research proposes an efficient way to perform various application system tasks by combining natural language commands with computer vision technology and LLM (ChatGPT). This approach allows for user interaction through natural language commands while building applications.

Comparison of similarity measures and community detection algorithms using collaboration filtering (협업 필터링을 사용한 유사도 기법 및 커뮤니티 검출 알고리즘 비교)

  • Ugli, Sadriddinov Ilkhomjon Rovshan;Hong, Minpyo;Park, Doo-Soon
    • Annual Conference of KIPS
    • /
    • 2022.05a
    • /
    • pp.366-369
    • /
    • 2022
  • The glut of information aggravated the process of data analysis and other procedures including data mining. Many algorithms were devised in Big Data and Data Mining to solve such an intricate problem. In this paper, we conducted research about the comparison of several similarity measures and community detection algorithms in collaborative filtering for movie recommendation systems. Movielense data set was used to do an empirical experiment. We applied three different similarity measures: Cosine, Euclidean, and Pearson. Moreover, betweenness and eigenvector centrality were used to detect communities from the network. As a result, we elucidated which algorithm is more suitable than its counterpart in terms of recommendation accuracy.

Personalized Clothing Recommendation Service Using Weather Information and Big Data (날씨 정보와 빅데이터를 활용한 개인 맞춤 의류추천서비스 설계 및 구현)

  • Choi, Byeol-Kyu;Kim, Yu-Sung;Kim, Sun-Yeol;Hong, Ki-Hyun
    • Annual Conference of KIPS
    • /
    • 2020.11a
    • /
    • pp.37-40
    • /
    • 2020
  • 날씨에 대한 인류의 관심은 인류 역사가 시작되면서 지금까지 예측하며 관심 영역인 만큼 인류에게 끼치는 영향이 크다. 초기 인류에게 있어서 의류는 생존을 위한 생존 도구에서 현재는 패션의 영역으로 자기를 표출하거나 자신에게 가장 어울리는 옷을 찾기 위한 욕구로 발전해 왔다. 따라서 본 논문에서는 날씨에 따른 개인의 체감온도와 해당 날씨에 가장 선호하는 의상을 분석하고, 예측하며 추천해주는 시스템을 제안한다. 제안하는 시스템은 지속적인 유지 관리를 통해 보완해 나간다면 날씨와 패션 분야에서 다양한 접목을 하는 등 기술발전을 할 것으로 기대된다.

Eco-System: REC Price Prediction Simulation in Cloud Computing Environment (Eco-System: 클라우드 컴퓨팅환경에서 REC 가격예측 시뮬레이션)

  • Cho, Kyucheol
    • Journal of the Korea Society for Simulation
    • /
    • v.23 no.4
    • /
    • pp.1-8
    • /
    • 2014
  • Cloud computing helps big data processing to make various information using IT resources. The government has to start the RPS(Renewable Portfolio Standard) and induce the production of electricity using renewable energy equipment. And the government manages system to gather big data that is distributed geographically. The companies can purchase the REC(Renewable Energy Certificate) to other electricity generation companies to fill shortage among their duty from the system. Because of the RPS use voluntary competitive market in REC trade and the prices have the large variation, RPS is necessary to predict the equitable REC price using RPS big data. This paper proposed REC price prediction method base on fuzzy logic using the price trend and trading condition infra in REC market, that is modeled in cloud computing environment. Cloud computing helps to analyze correlation and variables that act on REC price within RPS big data and the analysis can be predict REC price by simulation. Fuzzy logic presents balanced REC average trading prices using the trading quantity and price. The model presents REC average trading price using the trading quantity and price and the method helps induce well-converged price in the long run in cloud computing environment.