• 제목/요약/키워드: data sources

검색결과 4,026건 처리시간 0.029초

빅데이터 분석을 위해 아파치 스파크를 이용한 원시 데이터 소스에서 데이터 추출 (Capturing Data from Untapped Sources using Apache Spark for Big Data Analytics)

  • ;구흥서
    • 전기학회논문지
    • /
    • 제65권7호
    • /
    • pp.1277-1282
    • /
    • 2016
  • The term "Big Data" has been defined to encapsulate a broad spectrum of data sources and data formats. It is often described to be unstructured data due to its properties of variety in data formats. Even though the traditional methods of structuring data in rows and columns have been reinvented into column families, key-value or completely replaced with JSON documents in document-based databases, the fact still remains that data have to be reshaped to conform to certain structure in order to persistently store the data on disc. ETL processes are key in restructuring data. However, ETL processes incur additional processing overhead and also require that data sources are maintained in predefined formats. Consequently, data in certain formats are completely ignored because designing ETL processes to cater for all possible data formats is almost impossible. Potentially, these unconsidered data sources can provide useful insights when incorporated into big data analytics. In this project, using big data solution, Apache Spark, we tapped into other sources of data stored in their raw formats such as various text files, compressed files etc and incorporated the data with persistently stored enterprise data in MongoDB for overall data analytics using MongoDB Aggregation Framework and MapReduce. This significantly differs from the traditional ETL systems in the sense that it is compactible regardless of the data formats at source.

토양.지하수오염원 분류체계 구축방안: 2. 분류체계 구축 및 속성자료 활용방안 (Building a Classification Scheme of Soil and Groundwater Contamination Sources in Korea: 2. Construction of Classification System and Applications of Attribute Data)

  • 안정이;신경희;황상일
    • 한국지하수토양환경학회지:지하수토양환경
    • /
    • 제15권6호
    • /
    • pp.122-127
    • /
    • 2010
  • Constructing the national inventory that can be used as a tool to identify and assess existing or potential contamination is necessary for efficiently managing the soil and groundwater contamination. In order to start this construction, the first step is how we define and classify potential contamination sources of soil and groundwater. After selecting the basic classification model of contamination sources from developed countries, we suggested the classification and list of the potential contamination sources of soil and groundwater which are appropriate for specific conditions of South Korea. In addition, we investigated several databases to confirm the existence of available data sources and then examined established attribute data through chemical accident response information system (CARIS) and water information system (WIS) in National Institute of Environmental Research and mine geographic information system (MGIS) in Mine Reclamation Corporation. All sorts of attribute data in the existing databases can be utilized as significant assessment factors for determining the management priority of potential contamination sources in the future. Therefore, it is required the expanded investigation of additional database sources and the continual modification so that the classification system of potential contamination sources can be improved.

Food behaviors accounting for the recent trends in dietary fatty acid profiles among Korean adults

  • Song, SuJin;Shim, Jae Eun
    • Nutrition Research and Practice
    • /
    • 제16권3호
    • /
    • pp.405-417
    • /
    • 2022
  • BACKGROUND/OBJECTIVES: This study examined the changes in food behaviors of dietary fatty acids over 2007-2018 among Korean adults. SUBJECTS/METHODS: This study used data from the 4th (2007-2009), 5th (2010-2012), 6th (2013-2015), and 7th (2016-2018) Korea National Health and Nutrition Examination Surveys. A total of 46,307 adults aged 19-64 yrs were selected and dietary data were obtained from a single 24-h recall. In the 4th and 7th data, the major food sources for each fatty acid based on the contributing percentage of the food item were compared. The consumption trends in the major food sources were presented as grams per day over 2007-2018 and compared across the survey periods using the multiple regression model. RESULTS: From 2007 to 2018, for total fat, saturated fatty acid (SFA), and monounsaturated fatty acid, the contribution of animal food sources, including beef, chicken, and eggs increased but plant food sources (e.g., tofu, soybean, and plant oil) decreased. As polyunsaturated fatty acid sources, mayonnaise, eggs, and bread showed higher contributions, whereas soybean and tofu showed lower contributions in the 7th data compared to the 4th data. For n-3 fatty acids, the contribution of fish decreased between the 4th and 7th data. Over 12 yrs, the significant increases in the absolute amount of consumption from animal sources were observed. In contrast, decreases in the consumption from plant sources and fish were seen across the survey periods. CONCLUSIONS: In Korean adults, increases in the intake of dietary fatty acids along with changes in the food behaviors during 2007-2018 have evoked great concern for SFA intake, which is a cardiovascular disease risk factor. Healthy food sources of dietary fatty acids should be emphasized in this population.

NORTH ECLIPTIC POLE WIDE SURVEY

  • Lee, Hyung Mok;Kim, Seong Jin;NEP-Wide Team, NEP-Wide Team
    • 천문학논총
    • /
    • 제27권4호
    • /
    • pp.129-134
    • /
    • 2012
  • The North Ecliptic Pole (NEP) Wide survey covered about 5.4 $deg^2$, a nearly circular area centered on the NEP, using nine passbands of InfraRed Camera (IRC). We present the photometric properties of the data sets, and the nature of the sources detected in this field. The number of detected sources varied according to the filter band: with about 109,000 sources in the NIR, about 20,000 sources in the MIR-S, and about 16,000 sources seen in the MIR-L channel. The $5{\sigma}$ detection limits are about 21 mag in the NIR and 19.5 - 18.5 mag in the MIR bands in terms of the AB magnitude. 50% completeness levels are about 19.8 mag at $3{\mu}m$, 18.6 mag at $9{\mu}m$, and 18 mag at $18{\mu}m$ band (in AB magnitude), respectively. In order to validate the detected sources, all of them are confirmed by matching tests with those in other bands. The 'star-like' sources, defined by the high stellarity and magnitude cut from the optical ancillary data, appear statistically to have a high probability of being stars. The nature of the various types of extragalactic sources in this field are discussed using the color-color diagrams of the NIR and MIR bands with the redshift tracks of galaxies providing useful guidelines.

The Distribution of Information Sources within the University Selection Decision-Making Process: A Longitudinal Study

  • LE, Tri D.;NGUYEN, Tan T.;NGUYEN, Phuong N.D.;NGUYEN, Thi Quynh Trang
    • 유통과학연구
    • /
    • 제20권11호
    • /
    • pp.89-98
    • /
    • 2022
  • Purpose: Increasing competition in the higher education sector has prompted universities to enhance their marketing efforts and understand their potential customers. The study aims to explore how information sources are used and changed among prospective Vietnamese students during the decision-making process. Research Design, Data, and Methodology: This study undertakes a longitudinal study involving multiple rounds of data collection to better understand the decision-making process of prospective students. Data was collected from 12th -grade students in Vietnam through two rounds of quantitative surveys with 251 students and one round of qualitative interviews, spanning the duration of their senior year. The three stages of the decision-making process correspond to the three stages of pre-purchase period. Results: Most students decide that attending open days, taking career assessments, and looking up information online are the most important information sources to consider. The WOM sources are more important in the early stages, while university-generated sources and events are important in the later stages. Conclusion: Implications from this study may contribute to the design of more effective marketing communications campaigns as university marketers gain a better understanding of the distribution of information sources utilized for each specific stage of the decision-making process.

호주의 급여의약품 청구데이터의 활용에 대한 고찰: Atorvastatin의 사용량과 청구액 분석 사례를 중심으로 (Application of Australian Pharmaceutical Benefits Scheme data to the drug utilization studies: A case analysis on atorvastatin)

  • 이혜재;유수연
    • 한국임상약학회지
    • /
    • 제30권2호
    • /
    • pp.73-80
    • /
    • 2020
  • Objective: The Australian Pharmaceutical Benefits Scheme (PBS) is a national drug subsidy program. Given the similarity and comprehensiveness of the Australian PBS and the Korean National Health Insurance (NHI) data, these data are increasingly used for pharmacoepidemiological investigations, as well as international comparative studies. This study aims to introduce the various sources of publicly available PBS data and provide a practical guide to researchers conducting drug utilization studies. Methods: We searched literature and websites to detail and compare the collection, structure, components, and characteristics of each PBS data format. We identified different characteristics of the PBS data from the Korean NHI claims data which are mainly owing to their unique co-payment policies and data collection processes. In addition, the utilization and expenditure of atorvastatin, a widely used treatment for hyperlipidemia, were analyzed using two different sources of PBS data and the different results were interpreted. Results: There exist differences in when data were collected or non-subsidized uses of medicine were included among sources of PBS data. Additionally, two countries have different cost sharing methods inmedicine subsidy scheme; co-payment in Australia and co-insurance in Korea. Therefore, it should be noted that prescriptions under co-payment are not included in some data sources in Australia. Conclusion: Despite several analytical challenges, open access and easy data management are the strengths of the PBS data sources. A detailed knowledge of the PBS data can ensure robust methodology and interpretation of pharmacoepidemiological investigations or international comparative studies.

이종의 통계정보를 이용한 품사 부착 기법 (Part-Of-Speech Tagging using multiple sources of statistical data)

  • 조세형
    • 한국지능시스템학회논문지
    • /
    • 제18권4호
    • /
    • pp.501-506
    • /
    • 2008
  • 통계적 방식의 품사부착 문제는 보통 N-그램과 같을 단일 통계정보를 활용하지만 단순한 통계 정보라는 원천적인 한계가 있어 많은 오류가 발생한다. 따라서 다양한 정보를 활용하는 것이 정확도를 높일 가능성이 있다는 데는 이론의 여지가 없다. 그러나 다른 종류의 통계 자료는 배타적 자료가 아닌 한 상충되는 정보를 가질 수밖에 없으므로 이러한 정보들로부터 어떻게 종합적인 결론을 내는가가 문제이다. 본 논문에서는 이러한 상이한 통계정보를 통합하는 방법으로 언어 모델의 구성에서 활용된 바 있는 최대엔트로피 모델의 한국어 품사 부착에의 사용 가능성을 제시한다. 여기서는 이종의 통계정보로서 N그램과, 트리거 쌍을 사용하게 된다. 이러한 트리거 쌍 통계정보를 N그램과 함께 최대엔트로피 모델링을 했을 경우 퍼플렉시티가 어떻게 변화하는지에 대한 실험결과를 관찰하게 될 것이다. 트리거 쌍은 또한 다양하게 문맥사이즈를 변화할 수 있으며, N그램의 확률 모델도 다양하기 때문에 여러 종류의 실험을 통한 많은 향상을 예상 할 수 있다. 본 실험에서는 단일 모델 사용시 94.9 %의 정확도를 가진 3-그램 모델에 트리거 쌍을 최대 엔트로피 방식으로 추가한 견우 95.6% 의 정확도를 보여 0.7% 포인트의 정확도 향상을 기록하였다. 따라서 향후 다양한 정보원을 개발하여 최대엔트로피 방식으로 통합할 경우 지속적인 정확도 향상을 가져 올 수 있는 가능성을 보여준다고 할 수 있다.

Galaxy overdensity around sub-mm sources from SPT-SZ survey

  • KIM, Yeonsik;Shim, Hyujin
    • 천문학회보
    • /
    • 제45권1호
    • /
    • pp.61.2-61.2
    • /
    • 2020
  • We study the overdensity of near-infrared sources around 508 sub-mm sources classified as dusty galaxies in the SPT-SZ survey catalog observed in 95 GHz (3.15 mm), 150 GHz (2 mm) and 250 GHz (1.2 mm) bands. We used the VISTA hemisphere survey data release 6 (VHS DR6) catalog covering the J, H, Ks bands. The mean number of galaxies within a radius of 60 arcsec (corresponding to about 500 kpc at z=2) from 500 randomly selected positions is 14.4, while the galaxy number distribution is approximated as a Gaussian with a standard deviation of 7.9. From the 2500 deg2 of SPT-SZ survey + VHS DR6 data, there were 27 sub-mm sources that have galaxy overdensity higher than 4σ. We present color-magnitude diagram around 27 selected sub-mm sources with enhanced galaxy surface densities, in order to investigate the presence of structure around sub-mm sources.

  • PDF

고정오염원에서의 중금속 배출특성 연구 (Characteristics of Heavy Metal Emissions from Stationary Sources)

  • 박정민;이상보;차준석;권오상;이상학
    • 한국대기환경학회지
    • /
    • 제24권5호
    • /
    • pp.574-583
    • /
    • 2008
  • The results of HAPs emission data using TRI (Toxic Release Inventory), SODAM (Source Data Management system) were investigated and the emissions of 7 heavy metals from their sources and emission processes were also analyzed. Questionnaire for source data analysis as well as the stack sampling were carried out for 17 factories among 6 selected industrial types. The annual amount of emissions was estimated based on the measured concentration and flow rates. All sources were operated with high efficiency control devices and the concentration levels of all heavy metals were shown to be below 0.1 to of regulation standard. The highest emission source of heavy metals was steel manufacturing industry with the annual emission of 342.9 kg/yr and followed by hazardous waste incinerator, paint manufacturing, nonferrous metal manufacturing, rolling & press goods manufacturing and storage battery manufacturing. In the case of Hg, the emissions were quite significant from electric acros of steel manufacturing industry, although the concentration level was below the emission standard, showing the necessity of specific care for its management.

쇼핑정보원 활용에 따른 해외여행자 시장세분화 및 세분시장 특성 연구 (Tourism Market Segmentation Based on Shopping Information Sources)

  • 전양진
    • 한국의상디자인학회지
    • /
    • 제19권2호
    • /
    • pp.117-128
    • /
    • 2017
  • This study confirmed the types of shopping information sources during travel abroad, and developed a profile of tourists in terms of demographics, travel, and shopping behavior. Shopping information sources and characteristics of shopping products were identified first. Thereafter, travelers were segmented by their information-seeking behavior. An online survey method was used to get data from Korean vacationers in their 20s-50s, while factor analysis, cluster analysis, ${\chi}^2$ test and ANOVA were applied to analyze data. The results were as follows. First, the shopping information sources of overseas tourists were composed of four factors including sources from travel agents/media, information from travel books and local sources, and word-of-mouth sources. Also, four factors in product types and four product attributes were identified. Second, tourists were clustered into two groups, active and passive shopping information seekers, based on shopping source behavior. Third, two groups differed in terms of demographics, showing an older age and higher income for active shopping source seekers. Active shopping information users tended to join package trips with family members, and they were more satisfied with the trip. With regard to shopping, active shopping source seekers spent more money for shopping and preferred all kinds of shopping goods with an emphasis on travel shopping. In conclusion, shopping information sources seemed to be a meaningful tool for segmenting tourists. Rich, older, family tourists would be an major target market for local retailers.

  • PDF