• Title/Summary/Keyword: Large-memory data processing

Search Result 192, Processing Time 0.024 seconds

Performance Optimization of Numerical Ocean Modeling on Cloud Systems (클라우드 시스템에서 해양수치모델 성능 최적화)

  • JUNG, KWANGWOOG;CHO, YANG-KI;TAK, YONG-JIN
    • The Sea:JOURNAL OF THE KOREAN SOCIETY OF OCEANOGRAPHY
    • /
    • v.27 no.3
    • /
    • pp.127-143
    • /
    • 2022
  • Recently, many attempts to run numerical ocean models in cloud computing environments have been tried actively. A cloud computing environment can be an effective means to implement numerical ocean models requiring a large-scale resource or quickly preparing modeling environment for global or large-scale grids. Many commercial and private cloud computing systems provide technologies such as virtualization, high-performance CPUs and instances, ether-net based high-performance-networking, and remote direct memory access for High Performance Computing (HPC). These new features facilitate ocean modeling experimentation on commercial cloud computing systems. Many scientists and engineers expect cloud computing to become mainstream in the near future. Analysis of the performance and features of commercial cloud services for numerical modeling is essential in order to select appropriate systems as this can help to minimize execution time and the amount of resources utilized. The effect of cache memory is large in the processing structure of the ocean numerical model, which processes input/output of data in a multidimensional array structure, and the speed of the network is important due to the communication characteristics through which a large amount of data moves. In this study, the performance of the Regional Ocean Modeling System (ROMS), the High Performance Linpack (HPL) benchmarking software package, and STREAM, the memory benchmark were evaluated and compared on commercial cloud systems to provide information for the transition of other ocean models into cloud computing. Through analysis of actual performance data and configuration settings obtained from virtualization-based commercial clouds, we evaluated the efficiency of the computer resources for the various model grid sizes in the virtualization-based cloud systems. We found that cache hierarchy and capacity are crucial in the performance of ROMS using huge memory. The memory latency time is also important in the performance. Increasing the number of cores to reduce the running time for numerical modeling is more effective with large grid sizes than with small grid sizes. Our analysis results will be helpful as a reference for constructing the best computing system in the cloud to minimize time and cost for numerical ocean modeling.

Parallel k-Modes Algorithm for Spark Framework (스파크 프레임워크를 위한 병렬적 k-Modes 알고리즘)

  • Chung, Jaehwa
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.6 no.10
    • /
    • pp.487-492
    • /
    • 2017
  • Clustering is a technique which is used to measure similarities between data in big data analysis and data mining field. Among various clustering methods, k-Modes algorithm is representatively used for categorical data. To increase the performance of iterative-centric tasks such as k-Modes, a distributed and concurrent framework Spark has been received great attention recently because it overcomes the limitation of Hadoop. Spark provides an environment that can process large amount of data in main memory using the concept of abstract objects called RDD. Spark provides Mllib, a dedicated library for machine learning, but Mllib only includes k-means that can process only continuous data, so there is a limitation that categorical data processing is impossible. In this paper, we design RDD for k-Modes algorithm for categorical data clustering in spark environment and implement an algorithm that can operate effectively. Experiments show that the proposed algorithm increases linearly in the spark environment.

A Study on Light-weight Algorithm of Large scale BIM data for Visualization on Web based GIS Platform (웹기반 GIS 플랫폼 상 가시화 처리를 위한 대용량 BIM 데이터의 경량화 알고리즘 제시)

  • Kim, Ji Eun;Hong, Chang Hee
    • Spatial Information Research
    • /
    • v.23 no.1
    • /
    • pp.41-48
    • /
    • 2015
  • BIM Technology contains data from the life cycle of facility through 3D modeling. For these, one building products the huge file because of massive data. One of them is IFC which is the standard format, and there are issues that large scale data processing based on geometry and property information of object. It increases the rendering speed and constitutes the graphic card, so large scale data is inefficient for screen visualization to user. The light weighting of large scale BIM data has to solve for process and quality of program essentially. This paper has been searched and confirmed about light weight techniques from domestic and abroad researches. To control and visualize the large scale BIM data effectively, we proposed and verified the technique which is able to optimize the BIM character. For operating the large scale data of facility on web based GIS platform, the quality of screen switch from user phase and the effective memory operation were secured.

Network Anomaly Traffic Detection Using WGAN-CNN-BiLSTM in Big Data Cloud-Edge Collaborative Computing Environment

  • Yue Wang
    • Journal of Information Processing Systems
    • /
    • v.20 no.3
    • /
    • pp.375-390
    • /
    • 2024
  • Edge computing architecture has effectively alleviated the computing pressure on cloud platforms, reduced network bandwidth consumption, and improved the quality of service for user experience; however, it has also introduced new security issues. Existing anomaly detection methods in big data scenarios with cloud-edge computing collaboration face several challenges, such as sample imbalance, difficulty in dealing with complex network traffic attacks, and difficulty in effectively training large-scale data or overly complex deep-learning network models. A lightweight deep-learning model was proposed to address these challenges. First, normalization on the user side was used to preprocess the traffic data. On the edge side, a trained Wasserstein generative adversarial network (WGAN) was used to supplement the data samples, which effectively alleviates the imbalance issue of a few types of samples while occupying a small amount of edge-computing resources. Finally, a trained lightweight deep learning network model is deployed on the edge side, and the preprocessed and expanded local data are used to fine-tune the trained model. This ensures that the data of each edge node are more consistent with the local characteristics, effectively improving the system's detection ability. In the designed lightweight deep learning network model, two sets of convolutional pooling layers of convolutional neural networks (CNN) were used to extract spatial features. The bidirectional long short-term memory network (BiLSTM) was used to collect time sequence features, and the weight of traffic features was adjusted through the attention mechanism, improving the model's ability to identify abnormal traffic features. The proposed model was experimentally demonstrated using the NSL-KDD, UNSW-NB15, and CIC-ISD2018 datasets. The accuracies of the proposed model on the three datasets were as high as 0.974, 0.925, and 0.953, respectively, showing superior accuracy to other comparative models. The proposed lightweight deep learning network model has good application prospects for anomaly traffic detection in cloud-edge collaborative computing architectures.

Non-Causal Filter의 PC-NC에의 응용

  • 장현상;최종률
    • Proceedings of the Korean Society of Precision Engineering Conference
    • /
    • 1995.10a
    • /
    • pp.1039-1042
    • /
    • 1995
  • In real time application such as motion control, it is hard to find the application of non-causal filtering due to its need for future position data, even though it shows wide usage in off-line digital signal processing. Recently, some of motion control areas such as learning and repetitive control use non-causal filtering technique in their application. these kinds of zero-lag non-causal filter application are very usful not only to reduce the machine vibration, but also to increase control accuracy with comparatively less work. In this paper, genuine method to implement zero-lag non-causal filter in a CNC is introduced. Also the variation of this implementation for the learning operation is suggested to give the NC better control performance for a specific job. By adopting the new NC architecture call Soft-NC, all these implementions are made possible here, and especially large memory requirement which hinders their usage for many years is no longer barrier in their real world application.

  • PDF

Phase-Locked Three-Dimensional Structures in the Cylinder Wake Observed from Cinematic PIV Data (Cinematic PIV에 의한 실린더 후류의 위상평균된 3차원 구조)

  • Sung, Jae-Yong;Park, Kang-Kuk;Yoo, Jung-Yul
    • Proceedings of the KSME Conference
    • /
    • 2000.04b
    • /
    • pp.661-666
    • /
    • 2000
  • Near-wake flow field of a circular cylinder is studied by means of a cinematic PIV system with high sampling rate and large internal memory block. Experiments are conducted in a closed-cycle water tunnel system and a cross-correlation algorithm in conjunction with FFT (Fast Fourier Transform) analysis and an offset correlation technique is used for vector processing. With the help of very high sampling frequency compared to the shedding frequency, it is possible to obtain phase-averaged information of the three-dimensional wake, even though the shedding is not forced but natural. Phase-locked vortical structures observed simultaneously from the spanwise and cross-stream planes are displayed in the wake-transition regime where fine-scale secondary vortices have a spanwise wavelength or around one diameter. Spatial relations and temporal evolutions of the primary Karman vortex and the secondary vortex are also discussed schematically.

  • PDF

ASIC design of high speed CAM for connectionless server of ATM network (ATM망의 비연결형 서버를 위한 고속 CAM ASIC 설계)

  • 백덕수;김형균;이완범
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.22 no.7
    • /
    • pp.1403-1410
    • /
    • 1997
  • Because streaming mode connection server suitable to wide area ATM networks performs transmission, reception and lookup with time restriction for the transmission time of a cell, it has demerits of large cell loss incase that burst traffic occurs. Therefore, in this paper to decrease cell loss we propose a high speed CAM (Content Addressable Memory) which is capable of processing data of streaming mode connections server at a high speed. the proposed CAM is applied to forwarding table VPC map which performs function to output connection numbers about input VPI(Virtual Path Identifier)/VCI(Virtual Channel Identifier). The designed high speed CAM consist of DBL(Dual Bit Line) CAM structure performed independently write operation and match operation and two-port SRAM structure. Also, its simulation verification and full-custom layout is performed by Hspice and Composs tools in 0.8 .$\mu$m design rule.

  • PDF

The improved contrast control method for LCD monitor

  • Kwon, Byong-Heon;Park, Myung-Ryul;Youn, Jin-Suk
    • Proceedings of the IEEK Conference
    • /
    • 2002.07c
    • /
    • pp.1811-1814
    • /
    • 2002
  • In this paper, we propose an improved contrast control method f9r image improvement of multi-gray scale image. The proposed contrast control method can improve contrast of image by changing gradient of weight as the type of input image. In addition, the proposed method does not require field and frame memory for large amount of computed data. And, the proposed method can be easily applied to the FPD fur real-time processing because of its less hardware complexity than that of the conventional methods. Also it can flexibly control the contrast of input gray level by varying the weight values that control the contrast range. The operation and performance of the proposed controller have been verified using computer simulation.

  • PDF

RDNN: Rumor Detection Neural Network for Veracity Analysis in Social Media Text

  • SuthanthiraDevi, P;Karthika, S
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.16 no.12
    • /
    • pp.3868-3888
    • /
    • 2022
  • A widely used social networking service like Twitter has the ability to disseminate information to large groups of people even during a pandemic. At the same time, it is a convenient medium to share irrelevant and unverified information online and poses a potential threat to society. In this research, conventional machine learning algorithms are analyzed to classify the data as either non-rumor data or rumor data. Machine learning techniques have limited tuning capability and make decisions based on their learning. To tackle this problem the authors propose a deep learning-based Rumor Detection Neural Network model to predict the rumor tweet in real-world events. This model comprises three layers, AttCNN layer is used to extract local and position invariant features from the data, AttBi-LSTM layer to extract important semantic or contextual information and HPOOL to combine the down sampling patches of the input feature maps from the average and maximum pooling layers. A dataset from Kaggle and ground dataset #gaja are used to train the proposed Rumor Detection Neural Network to determine the veracity of the rumor. The experimental results of the RDNN Classifier demonstrate an accuracy of 93.24% and 95.41% in identifying rumor tweets in real-time events.

WAP Protocol Adaptation Requirements for Internet Service of Portable Handset in the Wireless Environment (무선 휴대 단말기 환경에서의 인터넷 서비스를 위한 WAP 프로토콜 수용방안)

  • 권영미;조웅기
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 1999.11a
    • /
    • pp.363-367
    • /
    • 1999
  • As the subscription rate to both Internet services and wireless communications services increases rapidly, traffic in the future wireless communications will be dominated by Internet applications. Current research for wireless technology is focused to the multimedia data transmission mainly for the personal computer or notebook whose display which has large display and abundant memory. But in the near future, Internet access request from the handphone or PDA device which is already spreaded in the world will be large. Special handset terminals which has small display monitor, low capacity memory and poor processing power has to be managed differently from torrent wireless communications protocols. So, WAP Forum is organized with wireless handset manufacturers in the world and the standardization process for wireless application is being actively developed. In this paper, basic model and architecture of WAP is introduced and adaptation requirements for change from HTML documents to WML formats are proposed. Also, compression ratio gained in the transform from the existing web documents to WML is shown.

  • PDF