• Title/Summary/Keyword: Data Scalability Problem

Search Result 116, Processing Time 0.022 seconds

An Application of MapReduce Technique over Peer-to-Peer Network (P2P 네트워크상에서 MapReduce 기법 활용)

  • Ren, Jian-Ji;Lee, Jae-Kee
    • Journal of KIISE:Computing Practices and Letters
    • /
    • v.15 no.8
    • /
    • pp.586-590
    • /
    • 2009
  • The objective of this paper describes the design of MapReduce over Peer-to-Peer network for dynamic environments applications. MapReduce is a software framework used for Cloud Computing which processing large data sets in a highly-parallel way. Based on the Peer-to-Peer network character which node failures will happen anytime, we focus on using a DHT routing protocol which named Pastry to handle the problem of node failures. Our results are very promising and indicate that the framework could have a wide application in P2P network systems while maintaining good computational efficiency and scalability. We believe that, P2P networks and parallel computing emerge as very hot research and development topics in industry and academia for many years to come.

NoSQL-based Sensor Web System for Fine Particles Analysis Services (미세먼지 분석 서비스를 위한 NoSQL 기반 센서 웹 시스템)

  • Kim, Jeong-Joon;Kwak, Kwang-Jin;Park, Jeong-Min
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.19 no.2
    • /
    • pp.119-125
    • /
    • 2019
  • Recently, it has become a social problem due to fine particles. There are more people wearing masks, weather alerts and disaster notices. Research and policy are actively underway. Meteorologically, the biggest damage caused by fine particles is the inversion layer phenomenon. In this study, we designed a system to warn fine Particles by analyzing inversion layer and wind direction. This weather information system proposes a system that can efficiently perform scalability and parallel processing by using OGC sensor web enablement system and NoSQL storage for sensor control and data exchange.

PESA: Prioritized experience replay for parallel hybrid evolutionary and swarm algorithms - Application to nuclear fuel

  • Radaideh, Majdi I.;Shirvan, Koroush
    • Nuclear Engineering and Technology
    • /
    • v.54 no.10
    • /
    • pp.3864-3877
    • /
    • 2022
  • We propose a new approach called PESA (Prioritized replay Evolutionary and Swarm Algorithms) combining prioritized replay of reinforcement learning with hybrid evolutionary algorithms. PESA hybridizes different evolutionary and swarm algorithms such as particle swarm optimization, evolution strategies, simulated annealing, and differential evolution, with a modular approach to account for other algorithms. PESA hybridizes three algorithms by storing their solutions in a shared replay memory, then applying prioritized replay to redistribute data between the integral algorithms in frequent form based on their fitness and priority values, which significantly enhances sample diversity and algorithm exploration. Additionally, greedy replay is used implicitly to improve PESA exploitation close to the end of evolution. PESA features in balancing exploration and exploitation during search and the parallel computing result in an agnostic excellent performance over a wide range of experiments and problems presented in this work. PESA also shows very good scalability with number of processors in solving an expensive problem of optimizing nuclear fuel in nuclear power plants. PESA's competitive performance and modularity over all experiments allow it to join the family of evolutionary algorithms as a new hybrid algorithm; unleashing the power of parallel computing for expensive optimization.

SDN-COR: An Efficient Network Coding Opportunistic Routing Method for Software-Defined Wireless Sensor Networks

  • Yifan Hu;Xiqiang Hou;Fuqiang Wang
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.18 no.7
    • /
    • pp.1795-1816
    • /
    • 2024
  • A Software-Defined Wireless Sensor Networks (SDWSNs) architecture is firstly proposed to address the issues of inflexible architecture strategies and low scalability of traditional WSNs in this article. The SDWSNs architecture involves the design of a software-defined sensor network model and a customized controller architecture, along with an analysis of the functionalities of each management module within the controller architecture. Secondly, to tackle limited energy problem of sensor nodes, a network coding opportunistic routing method (SDN-COR) is presented based on SDWSNs. This method incorporates considerations of coding opportunities, vertical distance, and remaining energy of nodes to design a metric for encoding opportunistic routing. By combining opportunistic forwarding mechanisms, candidate forwarding sets are selected and sorted based on priority to prioritize data transmission by higher-priority nodes. Simulation results indicate that, comparing with conventional methods, this approach achieves reduction in energy consumption by an average of 21.5%, improves network throughput by 24%, and extends network lifetime by 20%.

Design and Implementation of on XML Data Encryption System considering Validation (유효성을 고려한 XML 데이타 암호화 시스템의 설계 및 구현)

  • 남궁영환;박대하;허승호;백두권
    • Journal of KIISE:Databases
    • /
    • v.29 no.6
    • /
    • pp.417-428
    • /
    • 2002
  • XML(extensible Markup Language) is effective to information retrieval and sharing but has defects related to the data security. And, as a solution of this problem, the current XML security researches such as XML digital signature, XML data encryption, and XML access control exclude the validation property of XML document. The validation of XML should be considered for the secure information sharing in the XML-based environment. In this paper, we design and implement the system to support both security and validation to XML document. Our system performs data encryption and maintenance of valid status of XML document by referencing new XML schema namespace. In addition, it also provides the XML schema security function through the XML schema digital signature. During generating XML schema digital signature, DOMHash method which has the advantage of the faster speed than canonical XML method is applied to XML schema. In conclusion, our system shows the improved functions in flexibility, scalability, and reliability compared with the existing XML security researches.

A Distributed Trust Model Based on Reputation Management of Peers for P2P VoD Services

  • Huang, Guimin;Hu, Min;Zhou, Ya;Liu, Pingshan;Zhang, Yanchun
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.6 no.9
    • /
    • pp.2285-2301
    • /
    • 2012
  • Peer-to-Peer (P2P) networks are becoming more and more popular in video content delivery services, such as Video on Demand (VoD). Scalability feature of P2P allows a higher number of simultaneous users at a given server load and bandwidth to use stream service. However, the quality of service (QoS) in these networks is difficult to be guaranteed because of the free-riding problem that nodes download the recourses while never uploading recourses, which degrades the performance of P2P VoD networks. In this paper, a distributed trust model is designed to reduce node's free-riding phenomenon in P2P VoD networks. In this model, the P2P network is abstracted to be a super node hierarchical structure to monitor the reputation of nodes. In order to calculate the reputation of nodes, the Hidden Markov Model (HMM) is introduced in this paper. Besides, a distinction algorithm is proposed to distinguish the free-riders and malicious nodes. The free-riders are the nodes which have a low frequency to free-ride. And the malicious nodes have a high frequency to free-ride. The distinction algorithm takes different measures to response to the request of these two kinds of free-riders. The simulation results demonstrate that this proposed trust model can improve QoS effectively in P2P VoD networks.

Publish/Subscrib Service based Selective Sensor Data Monitoring System using Mesh Network (메쉬 네트워크에서 가입/게시(subscribe/publish) 서비스기반 선택적 센서정보 모니터링 시스템)

  • Kim, Yong-Hyuck;Kim, Young-Han
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.35 no.3B
    • /
    • pp.493-502
    • /
    • 2010
  • In this paper we propose a scalable sensor network system that makes mesh network among the sink nodes to solve the scalability problem of existing sensor network which is caused by multi-hop networking between the sensor nodes. In the proposed system, the sink nodes have the wireless networking ability to communicate with another sink nodes in mesh fashion, and with the monitoring nodes which is located in the local area or internet area. Especially, the system includes L4(Application Layer) routing mechanism that provides subscribe/publish service to serve selective transmission of sensor data to the specific monitoring nodes. The collected sensor data is transmitted to the monitoring nodes when the sensor data is matched with the monitoring node's interesting value.

Design of A new Algorithm by Using Standard Deviation Techniques in Multi Edge Computing with IoT Application

  • HASNAIN A. ALMASHHADANI;XIAOHENG DENG;OSAMAH R. AL-HWAIDI;SARMAD T. ABDUL-SAMAD;MOHAMMED M. IBRAHM;SUHAIB N. ABDUL LATIF
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.17 no.4
    • /
    • pp.1147-1161
    • /
    • 2023
  • The Internet of Things (IoT) requires a new processing model that will allow scalability in cloud computing while reducing time delay caused by data transmission within a network. Such a model can be achieved by using resources that are closer to the user, i.e., by relying on edge computing (EC). The amount of IoT data also grows with an increase in the number of IoT devices. However, building such a flexible model within a heterogeneous environment is difficult in terms of resources. Moreover, the increasing demand for IoT services necessitates shortening time delay and response time by achieving effective load balancing. IoT devices are expected to generate huge amounts of data within a short amount of time. They will be dynamically deployed, and IoT services will be provided to EC devices or cloud servers to minimize resource costs while meeting the latency and quality of service (QoS) constraints of IoT applications when IoT devices are at the endpoint. EC is an emerging solution to the data processing problem in IoT. In this study, we improve the load balancing process and distribute resources fairly to tasks, which, in turn, will improve QoS in cloud and reduce processing time, and consequently, response time.

A New Conference Information Data Model in SIP based Distributed Conference Architecture (SIP 기반 분산형 컨퍼런스 구조에서의 새로운 컨퍼런스 정보 데이터 모델)

  • Jang, Choon-Seo;Lee, Ky-Soo
    • Journal of the Korea Society of Computer and Information
    • /
    • v.14 no.5
    • /
    • pp.85-91
    • /
    • 2009
  • The centralized conference architecture has a restriction in scalability due to the performance reduction as the number of conference participants increases. To solve this problem several distributed conference architectures have been studied recently. In these architectures new conference servers are added dynamically to the conference environment. In this paper, We have proposed a new conference information data model which can be used in these distributed conference architectures. In our newly proposed conference information data model. several components has been added for exchanging conference information between primary conference server and multiple secondary conference servers. We also proposed a procedure of conference information exchange between these conference servers. And the management of conference informations and SIP(Session Initiation Protocol) notifications to the total conference participants can be processed distributedly with these conference servers, therefore the load to the primary conference server can be decreased by using this method. The performance of our proposed model has been evaluated by experiments.

Thompson sampling for multi-armed bandits in big data environments (빅데이터 환경에서 다중 슬롯머신 문제에 대한 톰슨 샘플링 방법)

  • Min Kyong Kim;Beom Seuk Hwang
    • The Korean Journal of Applied Statistics
    • /
    • v.37 no.5
    • /
    • pp.663-673
    • /
    • 2024
  • The multi-armed bandits (MAB) problem, involves selecting actions to maximize rewards within dynamic environments. This study explores the application of Thompson sampling, a robust MAB algorithm, within the context of big data analytics and statistical learning theory. By leveraging large-scale banner click data from recommendation systems, we evaluate Thompson sampling's performance across various simulated scenarios, employing advanced approximation techniques. Our findings demonstrate that Thompson sampling, particularly with Langevin Monte Carlo approximation, maintains robust performance and scalability in big data environments. This underscores its practical significance and adaptability, aligning with contemporary challenges in statistical learning.