• Title/Summary/Keyword: data crawling

Search Result 195, Processing Time 0.025 seconds

Development of Hazardous Food Notification Application Using CNN Model (CNN 모델을 이용한 위해 식품 알림 애플리케이션의 개발)

  • Yoon, Dong Eon;Lee, Hyo Sang;Oh, Am Suk
    • Journal of Korea Multimedia Society
    • /
    • v.25 no.3
    • /
    • pp.461-467
    • /
    • 2022
  • This research is to raise awareness of food safety by designing and supporting a hazard food information notification platform for consumers. To this end, the design was carried out by dividing the process into a data extraction process, an application screen design process, and a CNN-based food inference process. Data was collected through public data APIs and crawling, and it was sent to each activity screen designed for Android studios so that it could be output. As a result, when the platform is executed, information on hazardous food names, registration dates, food classification, manufacturing dates, recovery grades, recovery reasons, recovery methods, company names, barcode numbers, and packaging units can be intuitively and conveniently checked. In addition, CNN-based food inference processes allowed mobile cameras to infer harmful food and applied various quantization techniques such as Dynamic Range, Integer, and Float16 to compare the degree of improvement in inference performance. As a result, the group that applied basic quantization and treated device resources with GPU showed the greatest improvement in inference performance. Through this platform, it is expected that the reliability of food safety will be improved by making it more convenient for consumers to recognize food risks.

Seasonal Weather Factors and Sensibility Change Relationship via Textmining

  • Yeo, Hyun-Jin
    • Journal of the Korea Society of Computer and Information
    • /
    • v.27 no.8
    • /
    • pp.219-224
    • /
    • 2022
  • The Korea Meteorological Administration(KMA) has been released life-related indexes such as 'Life industrial weather information' and 'Safety weather information' while other countries' meteorological administrations have been made 'Human-biometeorology' and 'Health meteorology' indexes that concern human sensibility effections to diverse criteria. Although human sensibility changes have been studied in psychological research criteria with diverse and innumerous application areas, there are not enough studies that make data mining based validation of sensibility change factors. In this research I made models to estimate sensibility change caused by weather factors such as temperature and humidity, and validated by collecting sensibility data from SNS text crawling and weather data from KMA public dataset. By Logistic Regression, I clarify factors affecting sensibility changes.

Word Frequency-Based Big Data Analysis for the Annals of the Joseon Dynasty (조선왕조실록 분석을 위한 단어 빈도수 기반 빅 데이터 분석)

  • Bong, Young-Il;Lee, Choong-Ho
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2022.05a
    • /
    • pp.707-709
    • /
    • 2022
  • Annals of the Joseon Dynasty is a librarian that compiled the history of the Joseon Dynasty for 472 years, from Taejo to Cheoljong. The Annals of the Joseon Dynasty, National Treasure No. 151, are important documented heritages, but they are difficult to analyze due to their vast content. Therefore, rather than analyzing all the contents of the Annals of the Joseon Dynasty, it is necessary to extract and analyze important words. In this paper, we propose a method of extracting words from the main body of the Annals of the Joseon Dynasty through web crawling and analyzing the translated texts of the Annals of the Joseon Dynasty based on the data sorted according to the frequency of words. In this study, only the part of King Sejong of the Annals of the Joseon Dynasty was extracted and the importance was analyzed according to the frequency of words.

  • PDF

Conflict Analysis in Construction Project with Unstructured Data: A Case Study of Jeju Naval Base Project in South Korea

  • Baek, Seungwon;Han, Seung Heon;Lee, Changjun;Jang, Woosik;Ock, Jong Ho
    • International conference on construction engineering and project management
    • /
    • 2017.10a
    • /
    • pp.291-296
    • /
    • 2017
  • Infrastructure development as national project suffers from social conflict which is one of main risk to be managed. Social conflicts have a negative impact on not only the social integration but also the national economy as they require enormous social costs to be solved. Against this backdrop, this study analyzes social conflict using articles published by online news media based on web-crawling and natural language processing (NLP) techniques. As an illustrative case, the Jeju Naval Base (JNB) project which is one of representative conflict case in South Korea is analyzed. Total of 21,788 articles and representative keywords are identified annually. Additionally, comparative analysis is conducted between the extracted keywords and actual events occurred during the project. The authors explain actual events in the JNB project based on the extracted words by the year. This study contributes to analyze social conflict and to extract meaningful information from unstructured data.

  • PDF

Development of technology to improve information accessibility of information vulnerable class using crawling & clipping

  • Jeong, Seong-Bae;Kim, Kyung-Shin
    • Journal of the Korea Society of Computer and Information
    • /
    • v.23 no.2
    • /
    • pp.99-107
    • /
    • 2018
  • This study started from the public interest purpose to help accessibility for the information acquisition of the vulnerable groups due to visual difficulties such as the elderly and the visually impaired. In this study, the server resources are minimized and implemented in most of the user smart phones. In addition, we implement a method to gather necessary information by collecting only pattern information by utilizing crawl & clipping without having to visit the site of the information of the various sites having the data necessary for the user, and to have it in the server. Especially, we applied the TTS(Text-To-Speech) service composed of smart phone apps and tried to develop a unified customized information collection service based on voice-based information collection method.

Development of a method for urban flooding detection using unstructured data and deep learing (비정형 데이터와 딥러닝을 활용한 내수침수 탐지기술 개발)

  • Lee, Haneul;Kim, Hung Soo;Kim, Soojun;Kim, Donghyun;Kim, Jongsung
    • Journal of Korea Water Resources Association
    • /
    • v.54 no.12
    • /
    • pp.1233-1242
    • /
    • 2021
  • In this study, a model was developed to determine whether flooding occurred using image data, which is unstructured data. CNN-based VGG16 and VGG19 were used to develop the flood classification model. In order to develop a model, images of flooded and non-flooded images were collected using web crawling method. Since the data collected using the web crawling method contains noise data, data irrelevant to this study was primarily deleted, and secondly, the image size was changed to 224×224 for model application. In addition, image augmentation was performed by changing the angle of the image for diversity of image. Finally, learning was performed using 2,500 images of flooding and 2,500 images of non-flooding. As a result of model evaluation, the average classification performance of the model was found to be 97%. In the future, if the model developed through the results of this study is mounted on the CCTV control center system, it is judged that the respons against flood damage can be done quickly.

Analysis of drama viewership related words through unstructured data collection (비정형데이터 수집을 통한 드라마 시청률 연관어 분석)

  • Kang, Sun-Kyoung;Lee, Hyun-Chang;Shin, Seong-Yoon
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.21 no.8
    • /
    • pp.1567-1574
    • /
    • 2017
  • In this paper, we analyzed the stereotyped and non - stereotyped data in order to analyze the drama 's ratings. The formalized data collection collected 19 items from the four areas of drama information, person information, broadcasting information, and audience rating information of each broadcasting company. Atypical data were collected from bulletin boards, pre - broadcast blogs and post - broadcast blogs operated by each broadcasting company using a crawling technique. As a result of comparing the differences according to the four areas for each broadcaster from the collected regular data, the results were similar to each other. And we derived seven related words by analyzing the correlation of occurrence frequencies from unstructured data collected from bulletin boards and blogs of each broadcasting company. The derived associations were obtained through reliability analysis.

A Study on the Document Topic Extraction System for LDA-based User Sentiment Analysis (LDA 기반 사용자 감정분석을 위한 문서 토픽 추출 시스템에 대한 연구)

  • An, Yoon-Bin;Kim, Hak-Young;Moon, Yong-Hyun;Hwang, Seung-Yeon;Kim, Jeong-Joon
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.21 no.2
    • /
    • pp.195-203
    • /
    • 2021
  • Recently, big data, a major technology in the IT field, has been expanding into various industrial sectors and research on how to utilize it is actively underway. In most Internet industries, user reviews help users make decisions about purchasing products. However, the process of screening positive, negative and helpful reviews from vast product reviews requires a lot of time in determining product purchases. Therefore, this paper designs and implements a system that analyzes and aggregates keywords using LDA, a big data analysis technology, to provide meaningful information to users. For the extraction of document topics, in this study, the domestic book industry is crawling data into domains, and big data analysis is conducted. This helps buyers by providing comprehensive information on products based on user review topics and appraisal words, and furthermore, the product's outlook can be identified through the review status analysis.

Docker and Kubernetes Based Approaches for PM Data Collection (도커와 쿠버네티스 기반 미세먼지 데이터 수집 방안)

  • Hyo Hyun Choi;Yeon Wook Kim
    • Proceedings of the Korean Society of Computer Information Conference
    • /
    • 2024.01a
    • /
    • pp.305-306
    • /
    • 2024
  • 본 논문에서는 도커와 쿠버네티스를 활용하여 미세먼지 데이터를 수집할 때 다량으로 늘어나는 데이터를 효율적으로 수집하고 관리하기 위한 방안을 제시한다. 도커 이미지는 작성된 Dockerfile을 통해 생성되며, 필요한 의존성과 설정이 반영되어 있다. 쿠버네티스를 이용하여 생성된 도커 이미지를 기반으로 컨테이너를 생성하고, 컨테이너들을 파드 내에서 실행함으로써 데이터를 효율적으로 수집하고 관리한다.

  • PDF

An Analysis on Anti-Drone Technology Trends of Domestic Companies Using News Crawling on the Web (뉴스 기사의 크롤링을 통한 국내 기업의 안티 드론에 사용되는 기술 현황 분석)

  • Kim, Kyuseok
    • Journal of Advanced Navigation Technology
    • /
    • v.24 no.6
    • /
    • pp.458-464
    • /
    • 2020
  • Drones are being spreaded for the purposes such as construction, logistics, scientific research, recording, toy and so on. However, anti-drone related technologies which make the opposite drones neutralized are also widely being researched and developed because some drones are being used for crime or terror. The range of anti-drone related technologies can be divided into detection, identification and neutralization. The drone neutralization methods are divided into Soft-kill one which blocks the detected drones using jamming and Hard-kill one which destroys the detected ones physically. In this paper, Google and Naver domestic news articles related to anti-drone were gathered. Analyzing the domestic news articles, 8 of related technologies using RF, GNSS, Radar and so on were found. Regarding as this, the general features and usage status of those technologies were described and those on anti-drone for each company and agency were gathered and analyzed.