• Title/Summary/Keyword: Graph Data

Search Result 1,318, Processing Time 0.029 seconds

Performance Optimization Strategies for Fully Utilizing Apache Spark (아파치 스파크 활용 극대화를 위한 성능 최적화 기법)

  • Myung, Rohyoung;Yu, Heonchang;Choi, Sukyong
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.7 no.1
    • /
    • pp.9-18
    • /
    • 2018
  • Enhancing performance of big data analytics in distributed environment has been issued because most of the big data related applications such as machine learning techniques and streaming services generally utilize distributed computing frameworks. Thus, optimizing performance of those applications at Spark has been actively researched. Since optimizing performance of the applications at distributed environment is challenging because it not only needs optimizing the applications themselves but also requires tuning of the distributed system configuration parameters. Although prior researches made a huge effort to improve execution performance, most of them only focused on one of three performance optimization aspect: application design, system tuning, hardware utilization. Thus, they couldn't handle an orchestration of those aspects. In this paper, we deeply analyze and model the application processing procedure of the Spark. Through the analyzed results, we propose performance optimization schemes for each step of the procedure: inner stage and outer stage. We also propose appropriate partitioning mechanism by analyzing relationship between partitioning parallelism and performance of the applications. We applied those three performance optimization schemes to WordCount, Pagerank, and Kmeans which are basic big data analytics and found nearly 50% performance improvement when all of those schemes are applied.

A Case Study on Implementation of the Shipping Market Information Service System (해운시황정보서비스시스템 구현 사례연구)

  • Lee, Seokyong;Jeong, Myounghwan
    • Journal of Korea Port Economic Association
    • /
    • v.29 no.3
    • /
    • pp.73-94
    • /
    • 2013
  • The necessity of shipping market information services has been on the rise which emphasizes the relevance of transaction information and market information to parties both in and outside the shipping industry. However, previous related researches have been restricted to explorations limited by the offerings of existing shipping market information providers. Users today require effective information, an efficient contents management system, interfacing to help the information provider, graphing and spread sheets to facilitate and present the analyzed information through diverse formats, and reliable web and mobile services to provide information effectively with limited human resources. As a first step, service information has to be defined, so that it takes into account user utility, information retrieval and data development. Second, benchmark information and services must be provided from leading shipbrokers and research institutes. Third, a review of the latest technical trends is required to identify the most suitable technologies for servicing shipping market information. Finally, analysis is required on the implementation of a system with selected technologies, as well as the development of channels to post information which have been analyzed by users. Such a process would enable the continual redefinition of the shipping market information users actively need. The application of an X-Internet based WCMS, with a single-window dashboard providing user-customized information, and used to obtain and manage processes, add spread sheets to sustain calculations using the latest information, graph results, and to input additional information following predefined rules. Access to data and use of the system would require agreement that the system will incorporate user data and user-analyzed information into the market report, web portal, and hybrid app to provide current shipping market information appropriately and accurately to service users.

A Study on the Interpretalion of the Synthetic Unit Hydrograph According to the Characteristics of catchment Area and Runoff Routing (유역 특성과 유출추적에 의한 단위도 해석에 관한 고찰)

  • 서승덕
    • Magazine of the Korean Society of Agricultural Engineers
    • /
    • v.8 no.1
    • /
    • pp.1088-1096
    • /
    • 1966
  • The following is a method of synthetic unitgraph derivation based on the routing of a time area diagram through channel storage, studied by Clark-Jonstone and Laurenson. Unithy drograph (or unitgraph) is the hydrograph that would result from unit rainfall\ulcorner excess occuring uniformly with respect to both time and area over a catchment in unit time. By thus standarzing rainfall characteristics and ignoring loss, the unitgraph represents only the effects of catchment characteristics on the time distribution of runoff from a catchment The situation abten arises where it is desirable to derive a unitgraph for the design of dams, large bridge, and flood mitigation works such as levees, floodways and other flood control structures, and are also used in flood forecasting, and the necessary hydrologie records are not available. In such cases, if time and funds permit, it may be desirable to install the necessary raingauges, pruviometers, and stream gaging stations, and collect the necessary data over a period of years. On the otherhand, this procedure may be found either uneconomic or impossible on the grounds of time required, and it then becomes necessary to synthesise a unitgraph from a knowledge of the physical charcteristics of the catchment. In the preparing the approach to the solution of the problem we must select a number of catchment characteristic(shape, stream pattern, surface slope, and stream slope, etc.), a number of parameters that will define the magnitude and shape of the unit graph (e.g. peak discharge, time to peak, and base length, etc.), evaluate the catch-ment characteristics and unitgraph parameters selected, for a number of catchments having adequate rainfall and stream data and obtain Correlations between the two classes of data, and assume the relationships derived in just above question apply to other, ungaged, Catchments in the same region and, knowing the physical characteritics of these catchments, substitute for them in the relation\ulcorner ships to determine the corresponding unitgraph parameters. This method described in this note, based on the routing of a time area diagram through channel storage, appears to provide a logical line of research and they allow a readier correlation of unitgraph parameters with catchment characteristics. The main disadvantage of this method appears to be the error in routing all elements of rainfall excess through the same amount of storage. evertheless, it should be noted that the synthetic unitgraph method is more accurate than the rational method since it takes account of the shape and tophography of the catchment, channel storage, and temporal variation of rainfall excess, all of which are neglected in rational method.

  • PDF

Regional irrigation control modeling and regional climate characteristics Research on the correlation (지역별 관수제어 모델링 및 지역별 기후 특성과의 연관성에 관한 연구)

  • Jeong, Jin-Hyoung;Jo, Jae-Hyun;Kim, Seung-Hun;Choi, Ahnryul;Lee, Sang-Sik
    • The Journal of Korea Institute of Information, Electronics, and Communication Technology
    • /
    • v.14 no.3
    • /
    • pp.184-192
    • /
    • 2021
  • Domestic agriculture is facing real problems, such as a decrease in the population in rural areas, a shortage of labor due to an aging population, and increased risks due to the deepening of climate change. Smart farming technology is being developed to solve these problems. In the development of smart agricultural technology, irrigation control plays an important role in creating an optimal growth environment and is an important issue in terms of environmental protection. This paper is about the study of collecting and analyzing the rhizosphere environmental data of domestic paprika farms for the purpose of improving the quality of crops, reducing production costs, and increasing production. Irrigation control modeling presented in this paper Control modeling is to graphically present changes in a medium weight, feed, and drainage due to regional climatic features. To derive the graph, the parameters were determined through data collection and analysis, and the suggested irrigation control modeling method was applied to the collected rhizosphere environmental data to control irrigation in 6 regions (Gangwon-do, Chungnam, Jeonbuk, Jeonnam, Gyeongbuk, and Gyeongnam). The parameters were obtained and graphs were derived from them. After that, a study was conducted to analyze the derived parameters to verify the validity of the irrigation control modeling method and to correlate them with climatic features (average temperature and precipitation).

Influence of standard Korean and Gyeongsang regional dialect on the pronunciation of English vowels (표준어와 경상 지역 방언의 한국어 모음 발음에 따른 영어 모음 발음의 영향에 대한 연구)

  • Jang, Soo-Yeon
    • Phonetics and Speech Sciences
    • /
    • v.13 no.4
    • /
    • pp.1-7
    • /
    • 2021
  • This study aims to enhance English pronunciation education for Korean students by examining the impact of standard Korean and Gyeongsang regional dialect on the articulation of English vowels. Data were obtained through the Korean-Spoken English Corpus (K-SEC). Seven Korean words and ten English mono-syllabic words were uttered by adult, male speakers of standard Korean and Gyeongsang regional dialect, in particular, speakers with little to no experience living abroad were selected. Formant frequencies of the recorded corpus data were measured using spectrograms, provided by the speech analysis program, Praat. The recorded data were analyzed using the articulatory graph for formants. The results show that in comparison with speakers using standard Korean, those using the Gyeongsang regional dialect articulated both Korean and English vowels in the back. Moreover, the contrast between standard Korean and Gyeongsang regional dialect in the pronunciation of Korean vowels (/으/, /어/) affected how the corresponding English vowels (/ə/, /ʊ/) were articulated. Regardless of the use of regional dialect, a general feature of vowel pronunciation among Korean people is that they show more narrow articulatory movements, compared with that of native English speakers. Korean people generally experience difficulties with discriminating tense and lax vowels, whereas native English speakers have clear distinctions in vowel articulation.

Fabrication of Three-Dimensional Scanning System for Inspection of Mineshaft Using Multichannel Lidar (다중채널 Lidar를 이용한 수직갱도 조사용 3차원 형상화 장비 구현)

  • Soolo, Kim;Jong-Sung, Choi;Ho-Goon, Yoon;Sang-Wook, Kim
    • Tunnel and Underground Space
    • /
    • v.32 no.6
    • /
    • pp.451-463
    • /
    • 2022
  • Whenever a mineshaft accidentally collapses, speedy risk assessment is both required and crucial. But onsite safety diagnosis by humans is reportedly difficult considering the additional risk of collapse of the unstable mineshaft. Generally, drones equipped with high-speed lidar sensors can be used for such inspection. However, the drone technology is restrictively applicable at very shallow depth, failing in mineshafts with depths of hundreds of meters because of the limit of wireless communication and turbulence inside the mineshaft. In previous study, a three-dimensional (3D) scanning system with a single channel lidar was fabricated and operated using towed cable in a mineshaft to a depth of 200 m. The rotation and pendulum movement errors of the measuring unit were compensated for by applying the data of inertial measuring unit and comparing the similarity between the scan data of the adjacent depths (Kim et al., 2020). However, the errors grew with scan depth. In this paper, a multi-channel lidar sensor to obtain a continuous cross-sectional image of the mineshaft from a winch system pulled from bottom upward. In this new approach, within overlapped region viewed by the multi-channel lidar, rotation error was compensated for by comparing the similarity between the scan data at the same depth. The fabricated system was applied to scan 0-165 m depth of the mineshaft with 180 m depth. The reconstructed image was depicted in a 3D graph for interpretation.

Stochastic Self-similarity Analysis and Visualization of Earthquakes on the Korean Peninsula (한반도에서 발생한 지진의 통계적 자기 유사성 분석 및 시각화)

  • JaeMin Hwang;Jiyoung Lim;Hae-Duck J. Jeong
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.12 no.11
    • /
    • pp.493-504
    • /
    • 2023
  • The Republic of Korea is located far from the boundary of the earthquake plate, and the intra-plate earthquake occurring in these areas is generally small in size and less frequent than the interplate earthquake. Nevertheless, as a result of investigating and analyzing earthquakes that occurred on the Korean Peninsula between the past two years and 1904 and earthquakes that occurred after observing recent earthquakes on the Korean Peninsula, it was found that of a magnitude of 9. In this paper, the Korean Peninsula Historical Earthquake Record (2 years to 1904) published by the National Meteorological Research Institute is used to analyze the relationship between earthquakes on the Korean Peninsula and statistical self-similarity. In addition, the problem solved through this paper was the first to investigate the relationship between earthquake data occurring on the Korean Peninsula and statistical self-similarity. As a result of measuring the degree of self-similarity of earthquakes on the Korean Peninsula using three quantitative estimation methods, the self-similarity parameter H value (0.5 < H < 1) was found to be above 0.8 on average, indicating a high degree of self-similarity. And through graph visualization, it can be easily figured out in which region earthquakes occur most often, and it is expected that it can be used in the development of a prediction system that can predict damage in the event of an earthquake in the future and minimize damage to property and people, as well as in earthquake data analysis and modeling research. Based on the findings of this study, the self-similar process is expected to help understand the patterns and statistical characteristics of seismic activities, group and classify similar seismic events, and be used for prediction of seismic activities, seismic risk assessments, and seismic engineering.

A Study on the Capacity Review of One-lane Hi-pass Lanes on Highways : Focusing on Using Bootstrapping Techniques (고속도로 단차로 하이패스차로 용량 검토에 관한 연구 : 부트스트랩 기법 활용 중심으로)

  • Bosung Kim;Donghee Han
    • The Journal of The Korea Institute of Intelligent Transport Systems
    • /
    • v.23 no.3
    • /
    • pp.1-16
    • /
    • 2024
  • In the present highway design guidelines suggest that the capacity of one-lane hi-pass lanes is 2,000 veh/h for mainline toll plaza and 1,700 veh/h for interchange toll plaza. However, in a study conducted in early 2010, capacity of the mainline toll plaza was presented with 1,476 veh/h/ln to 1,665 veh/h/ln, and capacity of the interchange toll plaza was presented as 1,443 veh/h/ln. Accordingly, this study examined the feasibility of the capacity of the currently proposed highway one-lane hi-pass lane. Based on the 2021 individual vehicle passing data collected from the one-lane hi-pass gantry, the speed-traffic volume relationship graph and headway were used to calculate and compare capacity. In addition, the bootstrapping technique was introduced to utilize the headway and new processing methods for collected data were reviewed. As a result of the analysis, the one-lane hi-pass capacity could be estimated at 1,700 veh/h/ln for the interchange toll plaza, and at least 1,700 veh/h/ln for the mainline toll plaza. In addition, by using the bootstrap technique when using headway data, it was possible to present an estimated capacity similar to the observed capacity.

Comparison of Deep Learning Frameworks: About Theano, Tensorflow, and Cognitive Toolkit (딥러닝 프레임워크의 비교: 티아노, 텐서플로, CNTK를 중심으로)

  • Chung, Yeojin;Ahn, SungMahn;Yang, Jiheon;Lee, Jaejoon
    • Journal of Intelligence and Information Systems
    • /
    • v.23 no.2
    • /
    • pp.1-17
    • /
    • 2017
  • The deep learning framework is software designed to help develop deep learning models. Some of its important functions include "automatic differentiation" and "utilization of GPU". The list of popular deep learning framework includes Caffe (BVLC) and Theano (University of Montreal). And recently, Microsoft's deep learning framework, Microsoft Cognitive Toolkit, was released as open-source license, following Google's Tensorflow a year earlier. The early deep learning frameworks have been developed mainly for research at universities. Beginning with the inception of Tensorflow, however, it seems that companies such as Microsoft and Facebook have started to join the competition of framework development. Given the trend, Google and other companies are expected to continue investing in the deep learning framework to bring forward the initiative in the artificial intelligence business. From this point of view, we think it is a good time to compare some of deep learning frameworks. So we compare three deep learning frameworks which can be used as a Python library. Those are Google's Tensorflow, Microsoft's CNTK, and Theano which is sort of a predecessor of the preceding two. The most common and important function of deep learning frameworks is the ability to perform automatic differentiation. Basically all the mathematical expressions of deep learning models can be represented as computational graphs, which consist of nodes and edges. Partial derivatives on each edge of a computational graph can then be obtained. With the partial derivatives, we can let software compute differentiation of any node with respect to any variable by utilizing chain rule of Calculus. First of all, the convenience of coding is in the order of CNTK, Tensorflow, and Theano. The criterion is simply based on the lengths of the codes and the learning curve and the ease of coding are not the main concern. According to the criteria, Theano was the most difficult to implement with, and CNTK and Tensorflow were somewhat easier. With Tensorflow, we need to define weight variables and biases explicitly. The reason that CNTK and Tensorflow are easier to implement with is that those frameworks provide us with more abstraction than Theano. We, however, need to mention that low-level coding is not always bad. It gives us flexibility of coding. With the low-level coding such as in Theano, we can implement and test any new deep learning models or any new search methods that we can think of. The assessment of the execution speed of each framework is that there is not meaningful difference. According to the experiment, execution speeds of Theano and Tensorflow are very similar, although the experiment was limited to a CNN model. In the case of CNTK, the experimental environment was not maintained as the same. The code written in CNTK has to be run in PC environment without GPU where codes execute as much as 50 times slower than with GPU. But we concluded that the difference of execution speed was within the range of variation caused by the different hardware setup. In this study, we compared three types of deep learning framework: Theano, Tensorflow, and CNTK. According to Wikipedia, there are 12 available deep learning frameworks. And 15 different attributes differentiate each framework. Some of the important attributes would include interface language (Python, C ++, Java, etc.) and the availability of libraries on various deep learning models such as CNN, RNN, DBN, and etc. And if a user implements a large scale deep learning model, it will also be important to support multiple GPU or multiple servers. Also, if you are learning the deep learning model, it would also be important if there are enough examples and references.

Marketing Standardization and Firm Performance in International E.Commerce (국제전자상무중적영소표준화화공사표현(国际电子商务中的营销标准化和公司表现))

  • Fritz, Wolfgang;Dees, Heiko
    • Journal of Global Scholars of Marketing Science
    • /
    • v.19 no.3
    • /
    • pp.37-48
    • /
    • 2009
  • The standardization of marketing has been one of the most focused research topics in international marketing. The term "global marketing" was often used to mean an internationally standardized marketing strategy based on similarities between foreign markets. Marketing standardization was discussed only within the context of traditional physical marketplaces. Since then, the digital "marketspace" of the Internet had emerged in the 90's, and it became one of the most important drivers of the globalization process opening new opportunities for the standardization of global marketing activities. On the other hand, the opinion that a greater adoption of the Internet by customers may lead to a higher degree of customization and differentiation of products rather than standardization is also quite popular. Considering this disagreement, it is notable that comprehensive studies which focus upon the marketing standardization especially in the context of global e-commerce are missing to a high degree. On this background, the two basic research questions being addressed in this study are: (1) To what extent do companies standardize their marketing in international e-commerce? (2) Is there an impact of marketing standardization on the performance (or success) of these companies? Following research hypotheses were generated based upon literature review: H 1: Internationally engaged e-commerce firms show a growing readiness for marketing standardization. H 2: Marketing standardization exerts positive effects on the success of companies in international e-commerce. H 3: In international e-commerce, marketing mix standardization exerts a stronger positive effect on the economic as well as the non-economic success of companies than marketing process standardization. H 4: The higher the non-economic success in international e-commerce firms, the higher the economic success. The data for this research were obtained from a questionnaire survey conducted from February to April 2005. The international e-commerce companies of various industries in Germany and all subsidiaries or headquarters of foreign e-commerce companies based in Germany were included in the survey. 118 out of 801 companies responded to the questionnaire. For structural equation modelling (SEM), the Partial-Least. Squares (PLS) approach in the version PLS-Graph 3.0 was applied (Chin 1998a; 2001). All of four research hypotheses were supported by result of data analysis. The results show that companies engaged in international e-commerce standardize in particular brand name, web page design, product positioning, and the product program to a high degree. The companies intend to intensify their efforts for marketing mix standardization in the future. In addition they want to standardize their marketing processes also to a higher degree, especially within the range of information systems, corporate language and online marketing control procedures. In this study, marketing standardization exerts a positive overall impact on company performance in international e-commerce. Standardization of marketing mix exerts a stronger positive impact on the non-economic success than standardization of marketing processes, which in turn contributes slightly stronger to the economic success. Furthermore, our findings give clear support to the assumption that the non-economic success is highly relevant to the economic success of the firm in international e-commerce. The empirical findings indicate that marketing standardization is relevant to the companies' success in international e-commerce. But marketing mix and marketing process standardization contribute to the firms' economic and non-economic success in different ways. The findings indicate that companies do standardize numerous elements of their marketing mix on the Internet. This practice is in part contrary to the popular concept of a "differentiated standardization" which argues that some elements of the marketing mix should be adapted locally and others should be standardized internationally. Furthermore, the findings suggest that the overall standardization of marketing -rather than the standardization of one particular marketing mix element - is what brings about a positive overall impact on success.

  • PDF