• Title/Summary/Keyword: Distributed Data Analysis

Search Result 2,350, Processing Time 0.029 seconds

Development of Distributed Rainfall-Runoff Modelling System Integrated with GIS (지리정보시스템과 통합된 분포형 강우-유출 모의 시스템 개발)

  • Choi, Yun-Seok;Kim, Kyung-Tak;Shim, Myung-Pil
    • Journal of the Korean Association of Geographic Information Studies
    • /
    • v.12 no.3
    • /
    • pp.76-87
    • /
    • 2009
  • Most distributed models have been developed for data interchange between model for hydrological analysis and GIS(Geographic Information System). And some interface systems between them have been developed to operate the model conveniently. This study is about developing integrated system between model and GIS not coupled system based on file interchange or interface system. In this study, HyGIS-GRM which is integrated system between GRM(Grid based Rainfall-runoff Model) which is physically based distributed rainfall-runoff model and HyGIS(Hydro Geographic Information System) have been developed. HyGIS-GRM can carry out all the processes from preparing input data to appling them to model in the same system, and this operation environment can improve the efficiency of running the model and analyzing modeling results. HyGIS-GRM can provide objective modeling environment through establishing the process of integrated operation of GIS and distributed model, and we can obtain fundamental technologies for developing integrated system between GIS and water resources model.

  • PDF

A Study on the Improving Performance of Massively Small File Using the Reuse JVM in MapReduce (MapReduce에서 Reuse JVM을 이용한 대규모 스몰파일 처리성능 향상 방법에 관한 연구)

  • Choi, Chul Woong;Kim, Jeong In;Kim, Pan Koo
    • Journal of Korea Multimedia Society
    • /
    • v.18 no.9
    • /
    • pp.1098-1104
    • /
    • 2015
  • With the widespread use of smartphones and IoT (Internet of Things), data are being generated on a large scale, and there is increased for the analysis of such data. Hence, distributed processing systems have gained much attention. Hadoop, which is a distributed processing system, saves the metadata of stored files in name nodes; in this case, the main problems are as follows: the memory becomes insufficient; load occurs because of massive small files; scheduling and file processing time increases because of the increased number of small files. In this paper, we propose a solution to address the increase in processing time because of massive small files, and thus improve the processing performance, using the Reuse JVM method provided by Hadoop. Through environment setting, the Reuse JVM method modifies the JVM produced conventionally for every task, so that multiple tasks are reused sequentially in one JVM. As a final outcome, the Reuse JVM method showed the best processing performance when used together with CombineFileInputFormat.

Development of a distributed hydrological model considering hydrological change

  • Kim, Deasik;An, Hyunuk;Jang, Minwon;Kim, Seongjoon
    • Korean Journal of Agricultural Science
    • /
    • v.45 no.3
    • /
    • pp.521-532
    • /
    • 2018
  • In recent decades, the dry stream phenomena of small and medium sized rivers have been attracting much attention as an important social problem. To prevent dry stream phenomena, it is necessary to build an infrastructure that manages rivers. To accurately determine the progress of dry stream phenomena, it is necessary to continuously measure the discharge and other hydrological factors for small and medium sized rivers. However, until now, the flow data for small and medium rivers in Korea has been insufficient. To overcome the lack of supporting data for supporting rational decision-making in policy and project implementation, a short- and long-term hydrological model was developed that takes into consideration hydrological changes such as the increase of the impervious area due to urban development and groundwater pumping, the construction of a large-scale sewage treatment plant, the maintenance of stream-oriented rivers, etc. In the developed model, the distributed grid is represented by three layers: Surface flow, interflow, and groundwater flow. The surface flow and intermediate flow flowed along the flow direction, and the groundwater flow was calculated by a two-dimensional groundwater analysis model such that the outflow occurred in all directions without a specific flow direction. The effects of land use and cover on evapotranspiration and infiltration and the effects of multiple landscapes can be simulated in the developed model.

Design and Realization of Distributed Real-time Message Management Scheme for Naval Combat System Development Tool (함정 전투 시스템 개발 툴을 위한 분산 실시간 메시지 관리 기법 설계 및 구현)

  • Im, Jin Yong;Kim, Dong Seong;Song, Kyung Sub;Choi, Yoon Suk
    • Journal of Institute of Control, Robotics and Systems
    • /
    • v.22 no.7
    • /
    • pp.570-577
    • /
    • 2016
  • This paper proposes the design of a novel distributed message management scheme using a message-oriented management and analysis tool (MOMAT) for naval combat system (NCS) middle-ware. If a message is not guaranteed real-time of the NCS with each node, it causes the loss of data and decreases the reliability of systems. To solve these problems, improved message management schemes are proposed. Message management schemes are considering a real-time user management scheme and a real-time traffic management scheme. The proposed schemes are simulated with a developed simulation tool, data publisher, and subscriber connected through nodes in middle-ware. The simulation results show improved results in terms of message round-trip time (RTT), End-to-End delay, and throughput.

Minimum Bandwidth Regenerating Codes Based on Cyclic VFR Codes

  • Wang, Jing;Wang, Shuxia;Wang, Tiantian;Zhang, Xuefei
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.13 no.7
    • /
    • pp.3583-3598
    • /
    • 2019
  • In order to improve the reliability and repair efficiency of distributed storage systems, minimum bandwidth regenerating (MBR) codes based on cyclic variable fractional repetition (VFR) codes are constructed in this thesis, which can repair failed nodes accurately. Specifically, in order to consider the imbalance of data accessed by the users, cyclic VFR codes are constructed according to that data with different heat degrees are copied in different repetition degrees. Moreover, we divide the storage nodes into groups, and construct MBR codes based on cyclic VFR codes to improve the file download speed. Performance analysis and simulation results show that, the repair locality of a single node failure is always 2 when MBR codes based on cyclic VFR codes are adopted in distributed storage systems, which is obviously superior to the traditional MBR codes. Compared with RS codes and simple regenerating codes, the proposed MBR codes based on cyclic VFR codes have lower repair locality, repair complexity and bandwidth overhead, as well as higher repair efficiency. Moreover, relative to FR codes, the MBR codes based on cyclic VFR codes can be applicable to more storage systems.

Rebuilding Operational Risk Management Capabilities: Lessons Learned from COVID-19

  • JADWANI, Barkha;PARKHI, Shilpa;KARANDE, Kiran;BARGE, Prashant;BHIMAVARAPU, Venkata Mrudula;RASTOGI, Shailesh
    • The Journal of Asian Finance, Economics and Business
    • /
    • v.9 no.9
    • /
    • pp.249-261
    • /
    • 2022
  • Globally, COVID-19 has significantly impacted many different organizations and people. From the banks' perspective, this pandemic has affected banks' corporate and retail customers. Also, banks had to adjust to distributed workforce model. This paper analyses the lessons learned from the COVID-19 pandemic, which can be effectively used to rebuild banks' Operational Risk Management capabilities. The present study used the survey research methodology, which includes structured questionnaires completed by senior banking professionals to analyze the learnings from COVID-19 and understand the distributed workforce model and remote working effectiveness. Findings: The Pandemic accelerated the pace of digital transformation. The lockdown imposed due to the pandemic led to employees working remotely, which has been effective because of enhanced digital capabilities. However, enhanced monitoring is required to prevent data-related issues, and action needs to be taken to address challenges faced in having a remote distributed workforce model, like negative impact on on-the-job learning, data-related risks, and employee wellbeing. COVID-19 is an unprecedented event that could not have been predicted in any scenario analysis. This crisis has highlighted various systemic drawbacks that need to be addressed. Banks can apply the lesson learned from this Pandemic to become more robust in the future.

STRIDE-based threat modeling and DREAD evaluation for the distributed control system in the oil refinery

  • Kyoung Ho Kim;Kyounggon Kim;Huy Kang Kim
    • ETRI Journal
    • /
    • v.44 no.6
    • /
    • pp.991-1003
    • /
    • 2022
  • Industrial control systems (ICSs) used to be operated in closed networks, that is, separated physically from the Internet and corporate networks, and independent protocols were used for each manufacturer. Thus, their operation was relatively safe from cyberattacks. However, with advances in recent technologies, such as big data and internet of things, companies have been trying to use data generated from the ICS environment to improve production yield and minimize process downtime. Thus, ICSs are being connected to the internet or corporate networks. These changes have increased the frequency of attacks on ICSs. Despite this increased cybersecurity risk, research on ICS security remains insufficient. In this paper, we analyze threats in detail using STRIDE threat analysis modeling and DREAD evaluation for distributed control systems, a type of ICSs, based on our work experience as cybersecurity specialists at a refinery. Furthermore, we verify the validity of threats identified using STRIDE through case studies of major ICS cybersecurity incidents: Stuxnet, BlackEnergy 3, and Triton. Finally, we present countermeasures and strategies to improve risk assessment of identified threats.

Parameter and Modeling Uncertainty Analysis of Semi-Distributed Hydrological Model using Markov-Chain Monte Carlo Technique (Markov-Chain Monte Carlo 기법을 이용한 준 분포형 수문모형의 매개변수 및 모형 불확실성 분석)

  • Choi, Jeonghyeon;Jang, Suhyung;Kim, Sangdan
    • Journal of Korean Society on Water Environment
    • /
    • v.36 no.5
    • /
    • pp.373-384
    • /
    • 2020
  • Hydrological models are based on a combination of parameters that describe the hydrological characteristics and processes within a watershed. For this reason, the model performance and accuracy are highly dependent on the parameters. However, model uncertainties caused by parameters with stochastic characteristics need to be considered. As a follow-up to the study conducted by Choi et al (2020), who developed a relatively simple semi-distributed hydrological model, we propose a tool to estimate the posterior distribution of model parameters using the Metropolis-Hastings algorithm, a type of Markov-Chain Monte Carlo technique, and analyze the uncertainty of model parameters and simulated stream flow. In addition, the uncertainty caused by the parameters of each version is investigated using the lumped and semi-distributed versions of the applied model to the Hapcheon Dam watershed. The results suggest that the uncertainty of the semi-distributed model parameters was relatively higher than that of the lumped model parameters because the spatial variability of input data such as geomorphological and hydrometeorological parameters was inherent to the posterior distribution of the semi-distributed model parameters. Meanwhile, no significant difference existed between the two models in terms of uncertainty of the simulation outputs. The statistical goodness of fit of the simulated stream flows against the observed stream flows showed satisfactory reliability in both the semi-distributed and the lumped models, but the seasonality of the stream flow was reproduced relatively better by the distributed model.

In situ side-aspect target strength of Japanese anchovy (Engraulis japonicus) in northwestern Pacific Ocean (북서 태평양 멸치(Japanese anchovy)에 대한 측면 음향 반사강도 특성)

  • Lee, Hyung-Been;Kang, Don-Hyug
    • Journal of the Korean Society of Fisheries and Ocean Technology
    • /
    • v.46 no.3
    • /
    • pp.248-256
    • /
    • 2010
  • Acoustic side-aspect target strength (TS) of living Japanese anchovy (Engraulis japonicus) was measured at 120kHz during in situ experiments. The data were collected by lowering and horizontally projecting the splitbeam transducer into the anchovy school. For analysis and interpretation of the side-aspect TS data, acoustic theoretical model, based on the fish morphology, and dorsal-aspect TS data were used. Total length of the anchovy ranged from 6.6 to 12.8cm (mean length 9.3cm). The side-aspect TS distributed between -40 and -55dB, has an obvious length dependency. The mean side-aspect TS of the anchovy was -47.8dB, and the TS was about 2dB higher than mean TS generated from dorsal-aspect measurements. With reference to maximum TS, the results of the side-aspect TS were distributed within the range of the theoretical and dorsal-aspect TS. Apparently these tendency indicates that side-aspect TS measured from the study is useful data. These in situ measurements of side-aspect TS can be applied to improve acoustic detection and estimates of the anchovy, and is necessary to measure with a various frequency and length for making enhance data.

A Distributed Real-time 3D Pose Estimation Framework based on Asynchronous Multiviews

  • Taemin, Hwang;Jieun, Kim;Minjoon, Kim
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.17 no.2
    • /
    • pp.559-575
    • /
    • 2023
  • 3D human pose estimation is widely applied in various fields, including action recognition, sports analysis, and human-computer interaction. 3D human pose estimation has achieved significant progress with the introduction of convolutional neural network (CNN). Recently, several researches have proposed the use of multiview approaches to avoid occlusions in single-view approaches. However, as the number of cameras increases, a 3D pose estimation system relying on a CNN may lack in computational resources. In addition, when a single host system uses multiple cameras, the data transition speed becomes inadequate owing to bandwidth limitations. To address this problem, we propose a distributed real-time 3D pose estimation framework based on asynchronous multiple cameras. The proposed framework comprises a central server and multiple edge devices. Each multiple-edge device estimates a 2D human pose from its view and sendsit to the central server. Subsequently, the central server synchronizes the received 2D human pose data based on the timestamps. Finally, the central server reconstructs a 3D human pose using geometrical triangulation. We demonstrate that the proposed framework increases the percentage of detected joints and successfully estimates 3D human poses in real-time.