• Title/Summary/Keyword: huge data

Search Result 1,411, Processing Time 0.028 seconds

Discretization of Continuous Attributes based on Rough Set Theory and SOM (러브집합이론과 SOM을 이용한 연속형 속성의 이산화)

  • Seo Wan-Seok;Kim Jae-Yearn
    • Journal of Korean Society of Industrial and Systems Engineering
    • /
    • v.28 no.1
    • /
    • pp.1-7
    • /
    • 2005
  • Data mining is widely used for turning huge amounts of data into useful information and knowledge in the information industry in recent years. When analyzing data set with continuous values in order to gain knowledge utilizing data mining, we often undergo a process called discretization, which divides the attribute's value into intervals. Such intervals from new values for the attribute allow to reduce the size of the data set. In addition, discretization based on rough set theory has the advantage of being easily applied. In this paper, we suggest a discretization algorithm based on Rough Set theory and SOM(Self-Organizing Map) as a means of extracting valuable information from large data set, which can be employed even in the case where there lacks of professional knowledge for the field.

Scalable Big Data Pipeline for Video Stream Analytics Over Commodity Hardware

  • Ayub, Umer;Ahsan, Syed M.;Qureshi, Shavez M.
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.16 no.4
    • /
    • pp.1146-1165
    • /
    • 2022
  • A huge amount of data in the form of videos and images is being produced owning to advancements in sensor technology. Use of low performance commodity hardware coupled with resource heavy image processing and analyzing approaches to infer and extract actionable insights from this data poses a bottleneck for timely decision making. Current approach of GPU assisted and cloud-based architecture video analysis techniques give significant performance gain, but its usage is constrained by financial considerations and extremely complex architecture level details. In this paper we propose a data pipeline system that uses open-source tools such as Apache Spark, Kafka and OpenCV running over commodity hardware for video stream processing and image processing in a distributed environment. Experimental results show that our proposed approach eliminates the need of GPU based hardware and cloud computing infrastructure to achieve efficient video steam processing for face detection with increased throughput, scalability and better performance.

A Design and Development of Big Data Indexing and Search System using Lucene (루씬을 이용한 빅데이터 인덱싱 및 검색시스템의 설계 및 구현)

  • Kim, DongMin;Choi, JinWoo;Woo, ChongWoo
    • Journal of Internet Computing and Services
    • /
    • v.15 no.6
    • /
    • pp.107-115
    • /
    • 2014
  • Recently, increased use of the internet resulted in generation of large and diverse types of data due to increased use of social media, expansion of a convergence of among industries, use of the various smart device. We are facing difficulties to manage and analyze the data using previous data processing techniques since the volume of the data is huge, form of the data varies and evolves rapidly. In other words, we need to study a new approach to solve such problems. Many approaches are being studied on this issue, and we are describing an effective design and development to build indexing engine of big data platform. Our goal is to build a system that could effectively manage for huge data set which exceeds previous data processing range, and that could reduce data analysis time. We used large SNMP log data for an experiment, and tried to reduce data analysis time through the fast indexing and searching approach. Also, we expect our approach could help analyzing the user data through visualization of the analyzed data expression.

Effect and Development Strategies of a Village Development Project Using It's Traditional Specific Items in Hwaseong City (화성시 농촌전통테마마을 운영성과와 발전 방안)

  • Suh, Gyu-Sun
    • Journal of Agricultural Extension & Community Development
    • /
    • v.13 no.1
    • /
    • pp.49-67
    • /
    • 2006
  • The purpose of this study was to suggest development strategies of a village of Hwaseong-si where several programs using it's traditional items have been operated since 2003 according to the policy of Rural Traditional Thema Village Development implemented by Rural Development Administration(RDA). The village is located in Yodang-ri, Yanggam-myun, hwaseong-si in Gyounggi province. The village is called as 'Eunheng Namu Maeul' which means 'ginkgo tree village' since the tree is almost 350 years old and beautifully huge. Including this big tree there are much more traditional items such as organic dairy farming, hand-made cheese, legends and traditional plays. Using this items and government subsidies, the village has managed various tour programs and other income increasing projects. This study analyzed the strengths, weaknesses, opportunities and threats of the current situation of the village with the related materials and data to find out development strategies for the village-based programs and projects. This study recommended the followings as a major result of this study. The huge ginkgo tree at the village could be a better traditional attractive item when paths and wood of ginkgo trees will be built up especially utilizing the original huge one around the village. Like this, the item of hand made cheese could be a much more valuable traditional item when there will be an advanced facility for the people's working together. The social actives of the village have been weakened because of few young dwellers living there, therefore there needs a special subsidizing project for the village to hire a young manager having some social skills and knowledges. The situation being urbanized in front of the village needs precisely checking and implementing the Hwaseong-si's urbanization policy so that the urbanization could be harmonized with the maintenance and development of the traditional items of the village.

  • PDF

Development Trend of Optical Data Storage Media and Design and Fabrication of High Density optical Disk Substrate (광 정보 저장 미디어의 개발 동향 및 광 디스크 기판의 초정밀 설계 및 성형)

  • Kim, Dong-Mook;Kang, Shin-Ill;Rhim, Yoon-Chul
    • Journal of the Korean Society for Precision Engineering
    • /
    • v.18 no.4
    • /
    • pp.46-54
    • /
    • 2001
  • Technology of data storage device has developed noticeably as demands and needs of new media increase, Huge data can be conveniently handled using removable type optical disk. In the present paper, the trend and current issue of development for optical disk media are introduced. Standardization of next generation optical disk media, technology of recording and reading, and applications of magneto-optical devices are also discussed. Finally, a methodology of process optimization for design and fabrication of high density optical disk substrate is proposed.

  • PDF

A Novel Simulation Method of PV Generation System using Field Data (실제 데이터를 이용한 태양광 발전시스템의 시뮬레이션)

  • Park, Min-Won;Kim, Bong-Tae;Yu, In-Keun
    • Proceedings of the KIEE Conference
    • /
    • 2000.11a
    • /
    • pp.52-54
    • /
    • 2000
  • In PV power generation system study, huge system apparatuses are needed in order to verify the effect of system efficiency and stability considering the size of solar panels, the sort of converter types, and the load conditions and so on. And also, under the same weather and load conditions it is impossible to compare a certain MPPT control scheme to others. In this paper, in order to obtain effective solutions for the above mentioned topics, the solar cell array is simulated with it's VI characteristic equations, and the real field data of weather conditions is interfaced to EMTDC using Fortran program interface method. Consequently the simulation of PV power generation system using field data is realized in this paper, and acceptable results, which show close match between the real data of PV panel and the simulated data, were obtained.

  • PDF

Improvement of trajectory tracking control performance by using ILC

  • Le, Dang-Khanh;Nam, Taek-Kun
    • Journal of Advanced Marine Engineering and Technology
    • /
    • v.38 no.10
    • /
    • pp.1281-1286
    • /
    • 2014
  • This paper presents an iterative learning control (ILC) approach for tracking problems with specified data points that are desired points at certain time instants. To design ILC systems for such problems, unlike traditional ILC approaches, an algorithm which updates not only the control signal but also the reference trajectory at each trial will be developed. The relationship between the reference trajectory and ILC control in tracking problems where there are specified data points through which the system should pass is investigated as the rate of convergence. In traditional ILC, the desired data is stored in a tracking profile file. Due to the huge size of the data file containing the target points, it is important to reduce the computational cost. Finally, simulation results of the presented technique are mentioned and compared to other related works to confirm the effectiveness of proposed scheme.

DEVELOPMENT OF ROI PROCESSING SYSTEM USING QUICK LOOK IMAGE

  • Ahn, Sang-Il;Kim, Tae-Hoon;Kim, Tae-Young;Koo, In-Hoi
    • Proceedings of the KSRS Conference
    • /
    • 2007.10a
    • /
    • pp.526-529
    • /
    • 2007
  • Due to its inherent feature of high-resolution satellite, there is strong need in some specific area to minimize the processing time required to get a standard image on hand from downlink signal acquisition. However, in general image processing system, it takes considerable time to get image data up to certain level from raw data acquisition because the huge amount of data is dealt sequentially as input data. This paper introduces the high-speed image processing system which generates the image data only for the area selected by user. To achieve the high speed performance, this system includes Quick Look Image display function with sampling, ROI selection function, Image Line Index function, and Distributed processing function. The developed RPS was applied to KOMPSAT-2 320Mbps downlink channel and its effectiveness was successfully demonstrated. This feature to provide the image product very quickly is expected to promote the application of high resolution satellite image.

  • PDF

Equipment Management Information System Using Wireless Application Protocol (Wireless Application Protocol을 이용한 기자재 관리 정보시스템)

  • 임영문;최영두;김홍기
    • Journal of the Korea Safety Management & Science
    • /
    • v.2 no.3
    • /
    • pp.129-140
    • /
    • 2000
  • Nowadays the role of information systems is getting more and more increased according to the development of information technology. In order to manage complex, various and huge data, it is vital to construct efficient information system. For this effective information system, data have properly to be stored, encoded and represented when needed. This paper presents equipment management information system using wireless application protocol. This system enables us to have remote control of data searching and data management. Also, through the technique of data mining, database resulted from this system can be utilized into expectation and analysis about life-cycle, characteristic, and failure time of equipment, pattern recognition of users, and state of movement, etc.

  • PDF

Financial Performance Evaluation using Self-Organizing Maps: The Case of Korean Listed Companies (자기조직화 지도를 이용한 한국 기업의 재무성과 평가)

  • 민재형;이영찬
    • Journal of the Korean Operations Research and Management Science Society
    • /
    • v.26 no.3
    • /
    • pp.1-20
    • /
    • 2001
  • The amount of financial information in sophisticated large data bases is huge and makes interfirm performance comparisons very difficult or at least very time consuming. The purpose of this paper is to investigate whether neural networks in the form of self-organizing maps (SOM) can be successfully employed to manage the complexity for competitive financial benchmarking. SOM is known to be very effective to visualize results by projecting multi-dimensional financial data into two-dimensional output space. Using the SOM, we overcome the problems of finding an appropriate underlying distribution and the functional form of data when structuring and analyzing a large data base, and show an efficient procedure of competitive financial benchmarking through clustering firms on two-dimensional visual space according to their respective financial competitiveness. For the empirical purpose, we analyze the data base of annual reports of 100 Korean listed companies over the years 1998, 1999, and 2000.

  • PDF