• Title/Summary/Keyword: data sources

Search Result 4,026, Processing Time 0.034 seconds

Capturing Data from Untapped Sources using Apache Spark for Big Data Analytics (빅데이터 분석을 위해 아파치 스파크를 이용한 원시 데이터 소스에서 데이터 추출)

  • Nichie, Aaron;Koo, Heung-Seo
    • The Transactions of The Korean Institute of Electrical Engineers
    • /
    • v.65 no.7
    • /
    • pp.1277-1282
    • /
    • 2016
  • The term "Big Data" has been defined to encapsulate a broad spectrum of data sources and data formats. It is often described to be unstructured data due to its properties of variety in data formats. Even though the traditional methods of structuring data in rows and columns have been reinvented into column families, key-value or completely replaced with JSON documents in document-based databases, the fact still remains that data have to be reshaped to conform to certain structure in order to persistently store the data on disc. ETL processes are key in restructuring data. However, ETL processes incur additional processing overhead and also require that data sources are maintained in predefined formats. Consequently, data in certain formats are completely ignored because designing ETL processes to cater for all possible data formats is almost impossible. Potentially, these unconsidered data sources can provide useful insights when incorporated into big data analytics. In this project, using big data solution, Apache Spark, we tapped into other sources of data stored in their raw formats such as various text files, compressed files etc and incorporated the data with persistently stored enterprise data in MongoDB for overall data analytics using MongoDB Aggregation Framework and MapReduce. This significantly differs from the traditional ETL systems in the sense that it is compactible regardless of the data formats at source.

Building a Classification Scheme of Soil and Groundwater Contamination Sources in Korea: 2. Construction of Classification System and Applications of Attribute Data (토양.지하수오염원 분류체계 구축방안: 2. 분류체계 구축 및 속성자료 활용방안)

  • An, Jeong-Yi;Shin, Kyung-Hee;Hwang, Sang-Il
    • Journal of Soil and Groundwater Environment
    • /
    • v.15 no.6
    • /
    • pp.122-127
    • /
    • 2010
  • Constructing the national inventory that can be used as a tool to identify and assess existing or potential contamination is necessary for efficiently managing the soil and groundwater contamination. In order to start this construction, the first step is how we define and classify potential contamination sources of soil and groundwater. After selecting the basic classification model of contamination sources from developed countries, we suggested the classification and list of the potential contamination sources of soil and groundwater which are appropriate for specific conditions of South Korea. In addition, we investigated several databases to confirm the existence of available data sources and then examined established attribute data through chemical accident response information system (CARIS) and water information system (WIS) in National Institute of Environmental Research and mine geographic information system (MGIS) in Mine Reclamation Corporation. All sorts of attribute data in the existing databases can be utilized as significant assessment factors for determining the management priority of potential contamination sources in the future. Therefore, it is required the expanded investigation of additional database sources and the continual modification so that the classification system of potential contamination sources can be improved.

Food behaviors accounting for the recent trends in dietary fatty acid profiles among Korean adults

  • Song, SuJin;Shim, Jae Eun
    • Nutrition Research and Practice
    • /
    • v.16 no.3
    • /
    • pp.405-417
    • /
    • 2022
  • BACKGROUND/OBJECTIVES: This study examined the changes in food behaviors of dietary fatty acids over 2007-2018 among Korean adults. SUBJECTS/METHODS: This study used data from the 4th (2007-2009), 5th (2010-2012), 6th (2013-2015), and 7th (2016-2018) Korea National Health and Nutrition Examination Surveys. A total of 46,307 adults aged 19-64 yrs were selected and dietary data were obtained from a single 24-h recall. In the 4th and 7th data, the major food sources for each fatty acid based on the contributing percentage of the food item were compared. The consumption trends in the major food sources were presented as grams per day over 2007-2018 and compared across the survey periods using the multiple regression model. RESULTS: From 2007 to 2018, for total fat, saturated fatty acid (SFA), and monounsaturated fatty acid, the contribution of animal food sources, including beef, chicken, and eggs increased but plant food sources (e.g., tofu, soybean, and plant oil) decreased. As polyunsaturated fatty acid sources, mayonnaise, eggs, and bread showed higher contributions, whereas soybean and tofu showed lower contributions in the 7th data compared to the 4th data. For n-3 fatty acids, the contribution of fish decreased between the 4th and 7th data. Over 12 yrs, the significant increases in the absolute amount of consumption from animal sources were observed. In contrast, decreases in the consumption from plant sources and fish were seen across the survey periods. CONCLUSIONS: In Korean adults, increases in the intake of dietary fatty acids along with changes in the food behaviors during 2007-2018 have evoked great concern for SFA intake, which is a cardiovascular disease risk factor. Healthy food sources of dietary fatty acids should be emphasized in this population.

NORTH ECLIPTIC POLE WIDE SURVEY

  • Lee, Hyung Mok;Kim, Seong Jin;NEP-Wide Team, NEP-Wide Team
    • Publications of The Korean Astronomical Society
    • /
    • v.27 no.4
    • /
    • pp.129-134
    • /
    • 2012
  • The North Ecliptic Pole (NEP) Wide survey covered about 5.4 $deg^2$, a nearly circular area centered on the NEP, using nine passbands of InfraRed Camera (IRC). We present the photometric properties of the data sets, and the nature of the sources detected in this field. The number of detected sources varied according to the filter band: with about 109,000 sources in the NIR, about 20,000 sources in the MIR-S, and about 16,000 sources seen in the MIR-L channel. The $5{\sigma}$ detection limits are about 21 mag in the NIR and 19.5 - 18.5 mag in the MIR bands in terms of the AB magnitude. 50% completeness levels are about 19.8 mag at $3{\mu}m$, 18.6 mag at $9{\mu}m$, and 18 mag at $18{\mu}m$ band (in AB magnitude), respectively. In order to validate the detected sources, all of them are confirmed by matching tests with those in other bands. The 'star-like' sources, defined by the high stellarity and magnitude cut from the optical ancillary data, appear statistically to have a high probability of being stars. The nature of the various types of extragalactic sources in this field are discussed using the color-color diagrams of the NIR and MIR bands with the redshift tracks of galaxies providing useful guidelines.

The Distribution of Information Sources within the University Selection Decision-Making Process: A Longitudinal Study

  • LE, Tri D.;NGUYEN, Tan T.;NGUYEN, Phuong N.D.;NGUYEN, Thi Quynh Trang
    • Journal of Distribution Science
    • /
    • v.20 no.11
    • /
    • pp.89-98
    • /
    • 2022
  • Purpose: Increasing competition in the higher education sector has prompted universities to enhance their marketing efforts and understand their potential customers. The study aims to explore how information sources are used and changed among prospective Vietnamese students during the decision-making process. Research Design, Data, and Methodology: This study undertakes a longitudinal study involving multiple rounds of data collection to better understand the decision-making process of prospective students. Data was collected from 12th -grade students in Vietnam through two rounds of quantitative surveys with 251 students and one round of qualitative interviews, spanning the duration of their senior year. The three stages of the decision-making process correspond to the three stages of pre-purchase period. Results: Most students decide that attending open days, taking career assessments, and looking up information online are the most important information sources to consider. The WOM sources are more important in the early stages, while university-generated sources and events are important in the later stages. Conclusion: Implications from this study may contribute to the design of more effective marketing communications campaigns as university marketers gain a better understanding of the distribution of information sources utilized for each specific stage of the decision-making process.

Application of Australian Pharmaceutical Benefits Scheme data to the drug utilization studies: A case analysis on atorvastatin (호주의 급여의약품 청구데이터의 활용에 대한 고찰: Atorvastatin의 사용량과 청구액 분석 사례를 중심으로)

  • Lee, Hye-Jae;Yu, Su-Yeon
    • Korean Journal of Clinical Pharmacy
    • /
    • v.30 no.2
    • /
    • pp.73-80
    • /
    • 2020
  • Objective: The Australian Pharmaceutical Benefits Scheme (PBS) is a national drug subsidy program. Given the similarity and comprehensiveness of the Australian PBS and the Korean National Health Insurance (NHI) data, these data are increasingly used for pharmacoepidemiological investigations, as well as international comparative studies. This study aims to introduce the various sources of publicly available PBS data and provide a practical guide to researchers conducting drug utilization studies. Methods: We searched literature and websites to detail and compare the collection, structure, components, and characteristics of each PBS data format. We identified different characteristics of the PBS data from the Korean NHI claims data which are mainly owing to their unique co-payment policies and data collection processes. In addition, the utilization and expenditure of atorvastatin, a widely used treatment for hyperlipidemia, were analyzed using two different sources of PBS data and the different results were interpreted. Results: There exist differences in when data were collected or non-subsidized uses of medicine were included among sources of PBS data. Additionally, two countries have different cost sharing methods inmedicine subsidy scheme; co-payment in Australia and co-insurance in Korea. Therefore, it should be noted that prescriptions under co-payment are not included in some data sources in Australia. Conclusion: Despite several analytical challenges, open access and easy data management are the strengths of the PBS data sources. A detailed knowledge of the PBS data can ensure robust methodology and interpretation of pharmacoepidemiological investigations or international comparative studies.

Part-Of-Speech Tagging using multiple sources of statistical data (이종의 통계정보를 이용한 품사 부착 기법)

  • Cho, Seh-Yeong
    • Journal of the Korean Institute of Intelligent Systems
    • /
    • v.18 no.4
    • /
    • pp.501-506
    • /
    • 2008
  • Statistical POS tagging is prone to error, because of the inherent limitations of statistical data, especially single source of data. Therefore it is widely agreed that the possibility of further enhancement lies in exploiting various knowledge sources. However these data sources are bound to be inconsistent to each other. This paper shows the possibility of using maximum entropy model to Korean language POS tagging. We use as the knowledge sources n-gram data and trigger pair data. We show how perplexity measure varies when two knowledge sources are combined using maximum entropy method. The experiment used a trigram model which produced 94.9% accuracy using Hidden Markov Model, and showed increase to 95.6% when combined with trigger pair data using Maximum Entropy method. This clearly shows possibility of further enhancement when various knowledge sources are developed and combined using ME method.

Galaxy overdensity around sub-mm sources from SPT-SZ survey

  • KIM, Yeonsik;Shim, Hyujin
    • The Bulletin of The Korean Astronomical Society
    • /
    • v.45 no.1
    • /
    • pp.61.2-61.2
    • /
    • 2020
  • We study the overdensity of near-infrared sources around 508 sub-mm sources classified as dusty galaxies in the SPT-SZ survey catalog observed in 95 GHz (3.15 mm), 150 GHz (2 mm) and 250 GHz (1.2 mm) bands. We used the VISTA hemisphere survey data release 6 (VHS DR6) catalog covering the J, H, Ks bands. The mean number of galaxies within a radius of 60 arcsec (corresponding to about 500 kpc at z=2) from 500 randomly selected positions is 14.4, while the galaxy number distribution is approximated as a Gaussian with a standard deviation of 7.9. From the 2500 deg2 of SPT-SZ survey + VHS DR6 data, there were 27 sub-mm sources that have galaxy overdensity higher than 4σ. We present color-magnitude diagram around 27 selected sub-mm sources with enhanced galaxy surface densities, in order to investigate the presence of structure around sub-mm sources.

  • PDF

Characteristics of Heavy Metal Emissions from Stationary Sources (고정오염원에서의 중금속 배출특성 연구)

  • Park, Jung-Min;Lee, Sang-Bo;Cha, Jun-Seok;Kwon, Oh-Sang;Lee, Sang-Hak
    • Journal of Korean Society for Atmospheric Environment
    • /
    • v.24 no.5
    • /
    • pp.574-583
    • /
    • 2008
  • The results of HAPs emission data using TRI (Toxic Release Inventory), SODAM (Source Data Management system) were investigated and the emissions of 7 heavy metals from their sources and emission processes were also analyzed. Questionnaire for source data analysis as well as the stack sampling were carried out for 17 factories among 6 selected industrial types. The annual amount of emissions was estimated based on the measured concentration and flow rates. All sources were operated with high efficiency control devices and the concentration levels of all heavy metals were shown to be below 0.1 to of regulation standard. The highest emission source of heavy metals was steel manufacturing industry with the annual emission of 342.9 kg/yr and followed by hazardous waste incinerator, paint manufacturing, nonferrous metal manufacturing, rolling & press goods manufacturing and storage battery manufacturing. In the case of Hg, the emissions were quite significant from electric acros of steel manufacturing industry, although the concentration level was below the emission standard, showing the necessity of specific care for its management.

Tourism Market Segmentation Based on Shopping Information Sources (쇼핑정보원 활용에 따른 해외여행자 시장세분화 및 세분시장 특성 연구)

  • Jeon, Yangjin
    • Journal of the Korea Fashion and Costume Design Association
    • /
    • v.19 no.2
    • /
    • pp.117-128
    • /
    • 2017
  • This study confirmed the types of shopping information sources during travel abroad, and developed a profile of tourists in terms of demographics, travel, and shopping behavior. Shopping information sources and characteristics of shopping products were identified first. Thereafter, travelers were segmented by their information-seeking behavior. An online survey method was used to get data from Korean vacationers in their 20s-50s, while factor analysis, cluster analysis, ${\chi}^2$ test and ANOVA were applied to analyze data. The results were as follows. First, the shopping information sources of overseas tourists were composed of four factors including sources from travel agents/media, information from travel books and local sources, and word-of-mouth sources. Also, four factors in product types and four product attributes were identified. Second, tourists were clustered into two groups, active and passive shopping information seekers, based on shopping source behavior. Third, two groups differed in terms of demographics, showing an older age and higher income for active shopping source seekers. Active shopping information users tended to join package trips with family members, and they were more satisfied with the trip. With regard to shopping, active shopping source seekers spent more money for shopping and preferred all kinds of shopping goods with an emphasis on travel shopping. In conclusion, shopping information sources seemed to be a meaningful tool for segmenting tourists. Rich, older, family tourists would be an major target market for local retailers.

  • PDF