• Title/Summary/Keyword: web crawler

Search Result 102, Processing Time 0.028 seconds

Differences across countries in the impact of developers' collaboration characteristics on performance : Focused on weak tie theory (국가별 오픈소스 소프트웨어 개발자의 네트워크 특성이 개방형 협업 성과에 미치는 영향 : 약한 연결 이론을 중심으로)

  • Lee, Saerom;Baek, Hyunmi;Lee, Uijun
    • The Journal of Information Systems
    • /
    • v.29 no.2
    • /
    • pp.149-171
    • /
    • 2020
  • Purpose With the advent of the 4th Industrial Revolution, related technologies such as IoT, big data, and artificial intelligence technologies are developing through not only specific companies but also a number of unspecified developers called open collaboration. For this reason, it is important to understand the nature of the collaboration that leads to successful open collaboration. Design/methodology/approach We focused the relationship between the collaboration characteristics and collaboration performance of developers who participating in open source software development, which is a representative open collaboration. Specifically, we create the country-specific network and draw the individual developers characteristics from the network such as collaboration scope and collaboration intensity. We compare and analyze the characteristics of developers across countries and explore whether there are differences between indicators. We develop a Web crawler for GitHub, a representative OSSD development site, and collected data of developers who located at China, Japan, Korea, the United States, and Canada. Findings China showed the characteristics of cooperation suitable for the form of weak tie theory, and consistent results were not drawn from other countries. This study confirmed the necessity of exploratory research on collaboration characteristics by country considering that there are differences in open collaboration characteristics or software development environments by country.

Web Crawling and PageRank Calculation for Community-Limited Search (커뮤니티 제한 검색을 위한 웹 크롤링 및 PageRank 계산)

  • Kim Gye-Jeong;Kim Min-Soo;Kim Yi-Reun;Whang Kyu-Young
    • Proceedings of the Korean Information Science Society Conference
    • /
    • 2005.07b
    • /
    • pp.1-3
    • /
    • 2005
  • 최근 웹 검색 분야에서는 검색 질을 높이기 위한 기법들이 많이 연구되어 왔으며, 대표적인 연구로는 제한 검색, focused crawling, 웹 클러스터링 등이 있다. 그러나 제한 검색은 검색 범위를 의미적으로 관련된 사이트들로 제한할 수 없으며, focused crawling은 질의 시점에 클러스터링하기 때문에 질의 처리 시간이 오래 걸리고, 웹 클러스터링은 많은 웹 페이지들을 대상으로 클러스터링하기 위한 오버헤드가 크다. 본 논문에서는 검색 범위를 특정 커뮤니티로 제한하여 검색 하는 커뮤니티 제한 검색과 커뮤니티를 구하는 방법으로 cluster crawler를 제안하여 이러한 문제점을 해결한다. 또한, 커뮤니티를 이용하여 PageRank를 2단계로 계산하는 방법을 제안한다. 제안된 방법은 첫 번째 과정에서 커뮤니티 단위로 지역적으로 PageRank를 계산한 후, 두 번째 과정에서 이를 바탕으로 전역적으로 PageRank론 계산한다. 제안된 방법은 Wang에 의해 제안된 방법에 비해 PageRank 근사치의 오차를 $59\%$ 정도로 줄일 수 있다.

  • PDF

The Effects of Social Media on Music-induced Tourism: A Case of Korean Pop Music and Inbound Tourism to Korea

  • Oh, Sehwan;Ahn, JoongHo;Baek, Hyunmi
    • Asia pacific journal of information systems
    • /
    • v.25 no.1
    • /
    • pp.119-141
    • /
    • 2015
  • With the rapid spread of social media, video-sharing social media like YouTube has emerged as a consumption and distribution channel for entertainment goods such as music videos and movie trailers. In tourism research, there has been a lot of research of how the visual media such as movies and soap operas induced tourism. However, no studies have attempted to examine the role of social media as a music consumption channel and its impact on tourism. Expanding a body of media-induced tourism, we analysed the impact of video-sharing social media on music-induced tourism with a case of Korean pop music and inbound tourism to Korea. Developing a Web-crawler, we collected YouTube users' comments data on 166 Korean pop music video clips which were released from 2009 to 2012 with over 1 million view counts. Controlling many of the determinants of tourism and analysing country-by-country impact of YouTube comments with the panel data, we found that engagement of Korean pop music video clips on YouTube is a significant predictor for the flow of inbound tourists to Korea.

Flood monitoring and prediction using online unstructured data (비정형데이터를 활용한 홍수 모니터링 및 예측)

  • Lee, Jeong Ha;Hwang, Seok Hwan
    • Proceedings of the Korea Water Resources Association Conference
    • /
    • 2019.05a
    • /
    • pp.118-118
    • /
    • 2019
  • 현재 홍수예보는 정형데이터인 유량 및 수위 등을 활용하여 이뤄지고 있다. 하지만 실제 사람들이 체감하는 홍수에 대한 위험도는 홍수예보 발령과는 달라 홍수예보가 이뤄지지 않은 지역에서 인명사고가 발생하기도 한다. 이는 수위 측정이 이뤄지지 않는 소규모 하천이나 사람들의 유동성이 큰 도심지역에서 빈번하게 발생한다. 이를 보완하기 위해서는 사람들의 체감 정도 및 인구의 유동성을 고려한 비정형데이터를 활용해야 한다. 특히 소셜 네트워크 서비스(Social Network Commuinty, SNS)를 사용하는 사람들이 많아지면서 기존에 사용되어 왔던 정형데이터 센서 이외의 데이터를 제공한다. 또한 개개인이 작성하는 글은 실시간으로 활용이 가능하여 인구의 유동성 및 시 공간적 데이터를 얻기에 유용하여 활용성이 매우 높은 비정형데이터이다. 따라서 본 연구에서는 SNS 데이터를 추출하고 이를 분석하여 2018년에 발생했던 강우사상과의 패턴을 비교하여 홍수예보에서의 활용성을 분석하였다. 홍수와 관련한 키워드를 중심으로 시 공간적 정보 및 추출이 가능한 웹 크롤러(Web Crawler) 프로그램을 작성하였으며 이를 토대로 데이터를 수집하였다. 수집한 데이터와 실제 홍수사상을 비교 분석을 한 결과 강우량 및 수위와 해당 지역에 대한 데이터의 양이 유사한 패턴을 보인 것으로 확인되었다. 실시간으로 데이터를 수집하고 이를 분석하여 리드타임을 충분히 확보한다면 홍수예측에 활용 가능할 것이라 생각된다. 본 연구는 한국건설기술연구원 19주요-대4-시드사업인 '커뮤니티 빅데이터 패턴 해석을 통한 수난(水難) 발생 및 규모 예측 기술 개발(20190126-001) '로 수행되었습니다.

  • PDF

The Study for Type of Mask Wearing Dataset for Deep learning and Detection Model (딥러닝을 위한 마스크 착용 유형별 데이터셋 구축 및 검출 모델에 관한 연구)

  • Hwang, Ho Seong;Kim, Dong heon;Kim, Ho Chul
    • Journal of Biomedical Engineering Research
    • /
    • v.43 no.3
    • /
    • pp.131-135
    • /
    • 2022
  • Due to COVID-19, Correct method of wearing mask is important to prevent COVID-19 and the other respiratory tract infections. And the deep learning technology in the image processing has been developed. The purpose of this study is to create the type of mask wearing dataset for deep learning models and select the deep learning model to detect the wearing mask correctly. The Image dataset is the 2,296 images acquired using a web crawler. Deep learning classification models provided by tensorflow are used to validate the dataset. And Object detection deep learning model YOLOs are used to select the detection deep learning model to detect the wearing mask correctly. In this process, this paper proposes to validate the type of mask wearing datasets and YOLOv5 is the effective model to detect the type of mask wearing. The experimental results show that reliable dataset is acquired and the YOLOv5 model effectively recognize type of mask wearing.

Effect of Trust in Creators on Class Preference in Knowledge Marketplaces (지식 마켓플레이스에서 크리에이터에 대한 신뢰가 강의 선호도에 미치는 영향)

  • Kang, Young Ju;Kim, Jin Myeong;Lee, Ui Jun;Oh, Se Hwan
    • The Journal of Information Systems
    • /
    • v.31 no.3
    • /
    • pp.19-45
    • /
    • 2022
  • Purpose Since COVID-19, the demand for online class platforms has increased. However, those platforms have not been clearly defined, and related research is also limited. In the context of the knowledge marketplace (KMs), this study examined the effects of class information and trust in creators on class preferences from the perspective of consumption value theory. Design/methodology/approach By establishing a web crawler through Python, this study collected 1,174 class data in Korea's leading knowledge marketplace, Class 101, focusing on diverse class-related information and the number of Instagram followers for individual class creators. Based on class information, this research analyzed the effects of consumers' utilitarian value, social value, and hedonic value on class preference. In addition, this study examined whether consumers' trust in creators moderates the relationship between class information and class preference. Findings According to analysis results, it was found that the higher the consumers' consumption value for each class on KMs, the more positive their preference for the class. Also, it was confirmed that consumers' trust in creators moderates the relationship between class information and class preference.

Analysis of Social Media Utilization based on Big Data-Focusing on the Chinese Government Weibo

  • Li, Xiang;Guo, Xiaoqin;Kim, Soo Kyun;Lee, Hyukku
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.16 no.8
    • /
    • pp.2571-2586
    • /
    • 2022
  • The rapid popularity of government social media has generated huge amounts of text data, and the analysis of these data has gradually become the focus of digital government research. This study uses Python language to analyze the big data of the Chinese provincial government Weibo. First, this study uses a web crawler approach to collect and statistically describe over 360,000 data from 31 provincial government microblogs in China, covering the period from January 2018 to April 2022. Second, a word separation engine is constructed and these text data are analyzed using word cloud word frequencies as well as semantic relationships. Finally, the text data were analyzed for sentiment using natural language processing methods, and the text topics were studied using LDA algorithm. The results of this study show that, first, the number and scale of posts on the Chinese government Weibo have grown rapidly. Second, government Weibo has certain social attributes, and the epidemics, people's livelihood, and services have become the focus of government Weibo. Third, the contents of government Weibo account for more than 30% of negative sentiments. The classified topics show that the epidemics and epidemic prevention and control overshadowed the other topics, which inhibits the diversification of government Weibo.

P-TAF: A Big Data-based Platform for Total Air Traffic Forecast (빅데이터 기반 항공 수요예측 통합 플랫폼 설계 및 실증)

  • Jung, Jooik;Son, Seokhyun;Cha, Hee-June
    • Proceedings of the Korean Society of Computer Information Conference
    • /
    • 2021.01a
    • /
    • pp.281-282
    • /
    • 2021
  • 본 논문에서는 항공 수요예측을 위한 빅데이터 기반 플랫폼의 설계 및 실증 결과를 제시한다. 항공 수요예측 통합 플랫폼은 항공산업 관련 데이터를 Open API, RSS Feed, 웹크롤러(Web Crawler) 등을 이용하여 수집 및 분석하여 자체 개발한 항공 수요예측 알고리즘을 기반으로 결과를 시각화하여 보여주도록 구현되어 있다. 또한, 제안하는 플랫폼의 사용자 인터페이스를 통해 변수 설정을 하여 단위별(Global, National 등), 기간별(단기, 중장기 등), 유형별(여객, 화물 등) 예측 통계 자료를 도출할 수 있다. 플랫폼의 성능 검증을 위해 정형화된 데이터를 비롯하여 소셜네트워크서비스(SNS), 검색엔진 등에서 수집한 비정형 데이터까지 활용하여 특정 키워드의 빈도와 특정 노선에 대한 항공 수요간 상관관계를 분석하였다. 개발한 통합 플랫폼의 지능형 항공 수요예측 알고리즘을 통해 전반적인 공항 운영 및 공항 운영 정책 수립에 기여할 것으로 예상한다.

  • PDF

Building an SNS Crawling System Using Python (Python을 이용한 SNS 크롤링 시스템 구축)

  • Lee, Jong-Hwa
    • Journal of Korea Society of Industrial Information Systems
    • /
    • v.23 no.5
    • /
    • pp.61-76
    • /
    • 2018
  • Everything is coming into the world of network where modern people are living. The Internet of Things that attach sensors to objects allows real-time data transfer to and from the network. Mobile devices, essential for modern humans, play an important role in keeping all traces of everyday life in real time. Through the social network services, information acquisition activities and communication activities are left in a huge network in real time. From the business point of view, customer needs analysis begins with SNS data. In this research, we want to build an automatic collection system of SNS contents of web environment in real time using Python. We want to help customers' needs analysis through the typical data collection system of Instagram, Twitter, and YouTube, which has a large number of users worldwide. It is stored in database through the exploitation process and NLP process by using the virtual web browser in the Python web server environment. According to the results of this study, we want to conduct service through the site, the desired data is automatically collected by the search function and the netizen's response can be confirmed in real time. Through time series data analysis. Also, since the search was performed within 5 seconds of the execution result, the advantage of the proposed algorithm is confirmed.

Developing and Pre-Processing a Dataset using a Rhetorical Relation to Build a Question-Answering System based on an Unsupervised Learning Approach

  • Dutta, Ashit Kumar;Wahab sait, Abdul Rahaman;Keshta, Ismail Mohamed;Elhalles, Abheer
    • International Journal of Computer Science & Network Security
    • /
    • v.21 no.11
    • /
    • pp.199-206
    • /
    • 2021
  • Rhetorical relations between two text fragments are essential information and support natural language processing applications such as Question - Answering (QA) system and automatic text summarization to produce an effective outcome. Question - Answering (QA) system facilitates users to retrieve a meaningful response. There is a demand for rhetorical relation based datasets to develop such a system to interpret and respond to user requests. There are a limited number of datasets for developing an Arabic QA system. Thus, there is a lack of an effective QA system in the Arabic language. Recent research works reveal that unsupervised learning can support the QA system to reply to users queries. In this study, researchers intend to develop a rhetorical relation based dataset for implementing unsupervised learning applications. A web crawler is developed to crawl Arabic content from the web. A discourse-annotated corpus is generated using the rhetorical structural theory. A Naïve Bayes based QA system is developed to evaluate the performance of datasets. The outcome shows that the performance of the QA system is improved with proposed dataset and able to answer user queries with an appropriate response. In addition, the results on fine-grained and coarse-grained relations reveal that the dataset is highly reliable.