• Title/Summary/Keyword: Big data Processing

Search Result 1,063, Processing Time 0.029 seconds

A Performance Study on CPU-GPU Data Transfers of Unified Memory Device (통합메모리 장치에서 CPU-GPU 데이터 전송성능 연구)

  • Kwon, Oh-Kyoung;Gu, Gibeom
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.11 no.5
    • /
    • pp.133-138
    • /
    • 2022
  • Recently, as GPU performance has improved in HPC and artificial intelligence, its use is becoming more common, but GPU programming is still a big obstacle in terms of productivity. In particular, due to the difficulty of managing host memory and GPU memory separately, research is being actively conducted in terms of convenience and performance, and various CPU-GPU memory transfer programming methods are suggested. Meanwhile, recently many SoC (System on a Chip) products such as Apple M1 and NVIDIA Tegra that bundle CPU, GPU, and integrated memory into one large silicon package are emerging. In this study, data between CPU and GPU devices are used in such an integrated memory device and performance-related research is conducted during transmission. It shows different characteristics from the existing environment in which the host memory and GPU memory in the CPU are separated. Here, we want to compare performance by CPU-GPU data transmission method in NVIDIA SoC chips, which are integrated memory devices, and NVIDIA SMX-based V100 GPU devices. For the experimental workload for performance comparison, a two-dimensional matrix transposition example frequently used in HPC applications was used. We analyzed the following performance factors: the difference in GPU kernel performance according to the CPU-GPU memory transfer method for each GPU device, the transfer performance difference between page-locked memory and pageable memory, overall performance comparison, and performance comparison by workload size. Through this experiment, it was confirmed that the NVIDIA Xavier can maximize the benefits of integrated memory in the SoC chip by supporting I/O cache consistency.

A preliminary study for development of an automatic incident detection system on CCTV in tunnels based on a machine learning algorithm (기계학습(machine learning) 기반 터널 영상유고 자동 감지 시스템 개발을 위한 사전검토 연구)

  • Shin, Hyu-Soung;Kim, Dong-Gyou;Yim, Min-Jin;Lee, Kyu-Beom;Oh, Young-Sup
    • Journal of Korean Tunnelling and Underground Space Association
    • /
    • v.19 no.1
    • /
    • pp.95-107
    • /
    • 2017
  • In this study, a preliminary study was undertaken for development of a tunnel incident automatic detection system based on a machine learning algorithm which is to detect a number of incidents taking place in tunnel in real time and also to be able to identify the type of incident. Two road sites where CCTVs are operating have been selected and a part of CCTV images are treated to produce sets of training data. The data sets are composed of position and time information of moving objects on CCTV screen which are extracted by initially detecting and tracking of incoming objects into CCTV screen by using a conventional image processing technique available in this study. And the data sets are matched with 6 categories of events such as lane change, stoping, etc which are also involved in the training data sets. The training data are learnt by a resilience neural network where two hidden layers are applied and 9 architectural models are set up for parametric studies, from which the architectural model, 300(first hidden layer)-150(second hidden layer) is found to be optimum in highest accuracy with respect to training data as well as testing data not used for training. From this study, it was shown that the highly variable and complex traffic and incident features could be well identified without any definition of feature regulation by using a concept of machine learning. In addition, detection capability and accuracy of the machine learning based system will be automatically enhanced as much as big data of CCTV images in tunnel becomes rich.

Analysis of Twitter for 2012 South Korea Presidential Election by Text Mining Techniques (텍스트 마이닝을 이용한 2012년 한국대선 관련 트위터 분석)

  • Bae, Jung-Hwan;Son, Ji-Eun;Song, Min
    • Journal of Intelligence and Information Systems
    • /
    • v.19 no.3
    • /
    • pp.141-156
    • /
    • 2013
  • Social media is a representative form of the Web 2.0 that shapes the change of a user's information behavior by allowing users to produce their own contents without any expert skills. In particular, as a new communication medium, it has a profound impact on the social change by enabling users to communicate with the masses and acquaintances their opinions and thoughts. Social media data plays a significant role in an emerging Big Data arena. A variety of research areas such as social network analysis, opinion mining, and so on, therefore, have paid attention to discover meaningful information from vast amounts of data buried in social media. Social media has recently become main foci to the field of Information Retrieval and Text Mining because not only it produces massive unstructured textual data in real-time but also it serves as an influential channel for opinion leading. But most of the previous studies have adopted broad-brush and limited approaches. These approaches have made it difficult to find and analyze new information. To overcome these limitations, we developed a real-time Twitter trend mining system to capture the trend in real-time processing big stream datasets of Twitter. The system offers the functions of term co-occurrence retrieval, visualization of Twitter users by query, similarity calculation between two users, topic modeling to keep track of changes of topical trend, and mention-based user network analysis. In addition, we conducted a case study on the 2012 Korean presidential election. We collected 1,737,969 tweets which contain candidates' name and election on Twitter in Korea (http://www.twitter.com/) for one month in 2012 (October 1 to October 31). The case study shows that the system provides useful information and detects the trend of society effectively. The system also retrieves the list of terms co-occurred by given query terms. We compare the results of term co-occurrence retrieval by giving influential candidates' name, 'Geun Hae Park', 'Jae In Moon', and 'Chul Su Ahn' as query terms. General terms which are related to presidential election such as 'Presidential Election', 'Proclamation in Support', Public opinion poll' appear frequently. Also the results show specific terms that differentiate each candidate's feature such as 'Park Jung Hee' and 'Yuk Young Su' from the query 'Guen Hae Park', 'a single candidacy agreement' and 'Time of voting extension' from the query 'Jae In Moon' and 'a single candidacy agreement' and 'down contract' from the query 'Chul Su Ahn'. Our system not only extracts 10 topics along with related terms but also shows topics' dynamic changes over time by employing the multinomial Latent Dirichlet Allocation technique. Each topic can show one of two types of patterns-Rising tendency and Falling tendencydepending on the change of the probability distribution. To determine the relationship between topic trends in Twitter and social issues in the real world, we compare topic trends with related news articles. We are able to identify that Twitter can track the issue faster than the other media, newspapers. The user network in Twitter is different from those of other social media because of distinctive characteristics of making relationships in Twitter. Twitter users can make their relationships by exchanging mentions. We visualize and analyze mention based networks of 136,754 users. We put three candidates' name as query terms-Geun Hae Park', 'Jae In Moon', and 'Chul Su Ahn'. The results show that Twitter users mention all candidates' name regardless of their political tendencies. This case study discloses that Twitter could be an effective tool to detect and predict dynamic changes of social issues, and mention-based user networks could show different aspects of user behavior as a unique network that is uniquely found in Twitter.

Professional Baseball Viewing Culture Survey According to Corona 19 using Social Network Big Data (소셜네트워크 빅데이터를 활용한 코로나 19에 따른 프로야구 관람문화조사)

  • Kim, Gi-Tak
    • Journal of Korea Entertainment Industry Association
    • /
    • v.14 no.6
    • /
    • pp.139-150
    • /
    • 2020
  • The data processing of this study focuses on the textom and social media words about three areas: 'Corona 19 and professional baseball', 'Corona 19 and professional baseball', and 'Corona 19 and professional sports' The data was collected and refined in a web environment and then processed in batch, and the Ucinet6 program was used to visualize it. Specifically, the web environment was collected using Naver, Daum, and Google's channels, and was summarized into 30 words through expert meetings among the extracted words and used in the final study. 30 extracted words were visualized through a matrix, and a CONCOR analysis was performed to identify clusters of similarity and commonality of words. As a result of analysis, the clusters related to Corona 19 and Pro Baseball were composed of one central cluster and five peripheral clusters, and it was found that the contents related to the opening of professional baseball according to the corona 19 wave were mainly searched. The cluster related to Corona 19 and unrelated to professional baseball consisted of one central cluster and five peripheral clusters, and it was found that the keyword of the position of professional baseball related to the professional baseball game according to Corona 19 was mainly searched. Corona 19 and the cluster related to professional sports consisted of one central cluster and five peripheral clusters, and it was found that the keywords related to the start of professional sports according to the aftermath of Corona 19 were mainly searched.

A System for Automatic Classification of Traditional Culture Texts (전통문화 콘텐츠 표준체계를 활용한 자동 텍스트 분류 시스템)

  • Hur, YunA;Lee, DongYub;Kim, Kuekyeng;Yu, Wonhee;Lim, HeuiSeok
    • Journal of the Korea Convergence Society
    • /
    • v.8 no.12
    • /
    • pp.39-47
    • /
    • 2017
  • The Internet have increased the number of digital web documents related to the history and traditions of Korean Culture. However, users who search for creators or materials related to traditional cultures are not able to get the information they want and the results are not enough. Document classification is required to access this effective information. In the past, document classification has been difficult to manually and manually classify documents, but it has recently been difficult to spend a lot of time and money. Therefore, this paper develops an automatic text classification model of traditional cultural contents based on the data of the Korean information culture field composed of systematic classifications of traditional cultural contents. This study applied TF-IDF model, Bag-of-Words model, and TF-IDF/Bag-of-Words combined model to extract word frequencies for 'Korea Traditional Culture' data. And we developed the automatic text classification model of traditional cultural contents using Support Vector Machine classification algorithm.

An Empirical Study on Defense Future Technology in Artificial Intelligence (인공지능 분야 국방 미래기술에 관한 실증연구)

  • Ahn, Jin-Woo;Noh, Sang-Woo;Kim, Tae-Hwan;Yun, Il-Woong
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.21 no.5
    • /
    • pp.409-416
    • /
    • 2020
  • Artificial intelligence, which is in the spotlight as the core driving force of the 4th industrial revolution, is expanding its scope to various industrial fields such as smart factories and autonomous driving with the development of high-performance hardware, big data, data processing technology, learning methods and algorithms. In the field of defense, as the security environment has changed due to decreasing defense budget, reducing military service resources, and universalizing unmanned combat systems, advanced countries are also conducting technical and policy research to incorporate artificial intelligence into their work by including recognition systems, decision support, simplification of the work processes, and efficient resource utilization. For this reason, the importance of technology-driven planning and investigation is also increasing to discover and research potential defense future technologies. In this study, based on the research data that was collected to derive future defense technologies, we analyzed the characteristic evaluation indicators for future technologies in the field of artificial intelligence and conducted empirical studies. The study results confirmed that in the future technologies of the defense AI field, the applicability of the weapon system and the economic ripple effect will show a significant relationship with the prospect.

Arrival Time Estimation for Bus Information System Using Hidden Markov Model (은닉 마르코프 모델을 이용한 버스 정보 시스템의 도착 시간 예측)

  • Park, Chul Young;Kim, Hong Geun;Shin, Chang Sun;Cho, Yong Yun;Park, Jang Woo
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.6 no.4
    • /
    • pp.189-196
    • /
    • 2017
  • BIS(Bus Information System) provides the different information related to buses including predictions of arriving times at stations. BIS have been deployed almost all cities in our country and played active roles to improve the convenience of public transportation systems. Moving average filters, Kalman filter and regression models have been representative in forecasting the arriving times of buses in current BIS. The accuracy in prediction of arriving times depends largely on the forecasting algorithms and traffic conditions considered when forecasting in BIS. In present BIS, the simple prediction algorithms are used only considering the passage times and distances between stations. The forecasting of arrivals, however, have been influenced by the traffic conditions such as traffic signals, traffic accidents and pedestrians ets., and missing data. To improve the accuracy of bus arriving estimates, there are big troubles in building models including the above problems. Hidden Markov Models have been effective algorithms considering various restrictions above. So, we have built the HMM forecasting models for bus arriving times in the current BIS. When building models, the data collected from Sunchean City at 2015 have been utilized. There are about 2298 stations and 217 routes in Suncheon city. The models are developed differently week days and weekend. And then the models are conformed with the data from different districts and times. We find that our HMM models can provide more accurate forecasting than other existing methods like moving average filters, Kalmam filters, or regression models. In this paper, we propose Hidden Markov Model to obtain more precise and accurate model better than Moving Average Filter, Kalman Filter and regression model. With the help of Hidden Markov Model, two different sections were used to find the pattern and verified using Bootstrap process.

An Exploratory Study on Construction of Electronic Government as Platform with Customized Public Services : to Improve Administrative Aspects of Administrative Processes and Information Systems (맞춤형 공공서비스제공을 위한 플랫폼 전자정부 구축방안에 대한 탐색적 연구: 행정프로세스와 행정정보시스템 개선측면에서)

  • Lee, Sang-Yun;Chung, Myungju
    • Journal of Digital Convergence
    • /
    • v.14 no.1
    • /
    • pp.1-11
    • /
    • 2016
  • Currently Korean government is rushing the new electronic government system introduced as 'platform e-government' with big data and cloud computing technologies and systems, ultimately intending to provide the public institution services customized from the integrated counter or window for the heterogeneous resident services. In this regard, this study suggested how to design the new metadata information system in which mutual integration of information systems can take place, where heterogeneous services can be shared efficiently at the application and data unit, as a separate application that can provide a single one- stop service for residents' petition at the integrated level in the back-office based on the public data in possession of each of government ministries and related organizations. If this proposed system is implemented, the achievement of customized public service can be advanced one step forward in processing the petitions of the residents by organically connected link between 'Demand Chain' and 'Supply Chain' in the integrated window. In other words, it could be made possible through the unification of both the 'Supply Chain' performed in the office space of the officials at the back-office level and the 'Demand Chain' performed in the living space of the residents at the front-office level.

Multi-blockchain model ensures scalability and reliability based on intelligent Internet of Things (지능형 사물인터넷 기반의 확장성과 신뢰성을 보장하는 다중 블록체인 모델)

  • Jeong, Yoon-Su;Kim, Yong-Tae
    • Journal of Convergence for Information Technology
    • /
    • v.11 no.3
    • /
    • pp.140-146
    • /
    • 2021
  • As the environment using intelligent IoT devices increases, various studies are underway to ensure the integrity of information sent and received from intelligent IoT devices. However, all IoT information generated in heterogeneous environments is not fully provided with reliable protocols and services. In this paper, we propose an intelligent-based multi-blockchain model that can extract only critical information among various information processed by intelligent IoT devices. In the proposed model, blockchain is used to ensure the integrity of IoT information sent and received from IoT devices. The proposed model uses the correlation index of the collected information to trust a large number of IoT information to extract only the information with a high correlation index and bind it with blockchain. This is because the collected information can be extended to the n-tier structure as well as guaranteed reliability. Furthermore, since the proposed model can give weight information to the collection information based on blockchain, similar information can be selected (or bound) according to priority. The proposed model is able to extend the collection information to the n-layer structure while maintaining the data processing cost processed in real time regardless of the number of IoT devices.

Research of Water-related Disaster Monitoring Using Satellite Bigdata Based on Google Earth Engine Cloud Computing Platform (구글어스엔진 클라우드 컴퓨팅 플랫폼 기반 위성 빅데이터를 활용한 수재해 모니터링 연구)

  • Park, Jongsoo;Kang, Ki-mook
    • Korean Journal of Remote Sensing
    • /
    • v.38 no.6_3
    • /
    • pp.1761-1775
    • /
    • 2022
  • Due to unpredictable climate change, the frequency of occurrence of water-related disasters and the scale of damage are also continuously increasing. In terms of disaster management, it is essential to identify the damaged area in a wide area and monitor for mid-term and long-term forecasting. In the field of water disasters, research on remote sensing technology using Synthetic Aperture Radar (SAR) satellite images for wide-area monitoring is being actively conducted. Time-series analysis for monitoring requires a complex preprocessing process that collects a large amount of images and considers the noisy radar characteristics, and for this, a considerable amount of time is required. With the recent development of cloud computing technology, many platforms capable of performing spatiotemporal analysis using satellite big data have been proposed. Google Earth Engine (GEE)is a representative platform that provides about 600 satellite data for free and enables semi real time space time analysis based on the analysis preparation data of satellite images. Therefore, in this study, immediate water disaster damage detection and mid to long term time series observation studies were conducted using GEE. Through the Otsu technique, which is mainly used for change detection, changes in river width and flood area due to river flooding were confirmed, centered on the torrential rains that occurred in 2020. In addition, in terms of disaster management, the change trend of the time series waterbody from 2018 to 2022 was confirmed. The short processing time through javascript based coding, and the strength of spatiotemporal analysis and result expression, are expected to enable use in the field of water disasters. In addition, it is expected that the field of application will be expanded through connection with various satellite bigdata in the future.