• Title/Summary/Keyword: aggregate data

Search Result 672, Processing Time 0.026 seconds

Efficient Processing of Multiple Group-by Queries in MapReduce for Big Data Analysis (맵리듀스에서 빅데이터 분석을 위한 다중 Group-by 질의의 효율적인 처리 기법)

  • Park, Eunju;Park, Sojeong;Oh, Sohyun;Choi, Hyejin;Lee, Ki Yong;Shim, Junho
    • KIISE Transactions on Computing Practices
    • /
    • v.21 no.5
    • /
    • pp.387-392
    • /
    • 2015
  • MapReduce is a framework used to process large data sets in parallel on a large cluster. A group-by query is a query that partitions the input data into groups based on the values of the specified attributes, and then evaluates the value of the specified aggregate function for each group. In this paper, we propose an efficient method for processing multiple group-by queries using MapReduce. Instead of computing each group-by query independently, the proposed method computes multiple group-by queries in stages with one or more MapReduce jobs in order to reduce the total execution cost. We compared the performance of this method with the performance of a less sophisticated method that computes each group-by query independently. This comparison showed that the proposed method offers better performance in terms of execution time.

Design of an Inference Control Process in OLAP Data Cubes (OLAP 데이터 큐브에서의 추론통제 프로세스 설계)

  • Lee, Duck-Sung;Choi, In-Soo
    • Journal of the Korea Society of Computer and Information
    • /
    • v.14 no.5
    • /
    • pp.183-193
    • /
    • 2009
  • Both On-Line Analytical Processing (OLAF) data cubes and Statistical Databases (SDBs) deal with multidimensional data sets. and both are concerned with statistical summarizations over the dimensions of the data sets. However, there is a distinction between the two that can be made. While SDBs are usually derived from other base data, OLAF data cubes often represent directly the base data. In other word, the base data of SDBs are the macro-data, whereas the core cubiod data in OLAF data cubes are the micro-data. The base table in OLAF is used to populate the data cube with values of the measure attribute, and each record in the base tables is used to populate a cell of the core cuboid. The fact that OLAF data cubes mostly represent the micro-data may make some records be absent in the base table. Some cells of the core cuboid remain empty, if corresponding records are absent in the base table. Wang and others proposed a method for securing OLAF data cubes against privacy breaches. They assert that the proposed method does not depend on specific types of aggregation functions. In this paper, however, it is found that their assertion on aggregate functions is wrong whenever any cell of the core cuboid remains empty. The objective of this study is to design an inference control process in OLAF data cubes which rectifying Wang's error.

Energy-Efficient Data Aggregation and Dissemination based on Events in Wireless Sensor Networks (무선 센서 네트워크에서 이벤트 기반의 에너지 효율적 데이터 취합 및 전송)

  • Nam, Choon-Sung;Jang, Kyung-Soo;Shin, Dong-Ryeol
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.11 no.1
    • /
    • pp.35-40
    • /
    • 2011
  • In this paper, we compare and analyze data aggregation methods based on event area in wireless sensor networks. Data aggregation methods consist of two methods: the direct transmission method and the aggregation node method. The direct aggregation method has some problems that are data redundancy and increasing network traffic as all nodes transmit own data to neighbor nodes regardless of same data. On the other hand the aggregation node method which aggregate neighbor's data can prevent the data redundancy and reduce the data. This method is based on location of nodes. This means that the aggregation node can be selected the nearest node from a sink or the centered node of event area. So, we describe the benefits of data aggregation methods that make up for the weak points of direct data dissemination of sensor nodes. We measure energy consumption of the existing ways on data aggregation selection by increasing event area. To achieve this, we calculated the distance between an event node and the aggregation node and the distance between the aggregation node and a sink node. And we defined the equations for distance. Using these equations with energy model for sensor networks, we could find the energy consumption of each method.

Storm-Based Dynamic Tag Cloud for Real-Time SNS Data (실시간 SNS 데이터를 위한 Storm 기반 동적 태그 클라우드)

  • Son, Siwoon;Kim, Dasol;Lee, Sujeong;Gil, Myeong-Seon;Moon, Yang-Sae
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.6 no.6
    • /
    • pp.309-314
    • /
    • 2017
  • In general, there are many difficulties in collecting, storing, and analyzing SNS (social network service) data, since those data have big data characteristics, which occurs very fast with the mixture form of structured and unstructured data. In this paper, we propose a new data visualization framework that works on Apache Storm, and it can be useful for real-time and dynamic analysis of SNS data. Apache Storm is a representative big data software platform that processes and analyzes real-time streaming data in the distributed environment. Using Storm, in this paper we collect and aggregate the real-time Twitter data and dynamically visualize the aggregated results through the tag cloud. In addition to Storm-based collection and aggregation functionalities, we also design and implement a Web interface that a user gives his/her interesting keywords and confirms the visualization result of tag cloud related to the given keywords. We finally empirically show that this study makes users be able to intuitively figure out the change of the interested subject on SNS data and the visualized results be applied to many other services such as thematic trend analysis, product recommendation, and customer needs identification.

A Study on Data Quality Evaluation of Administrative Information Dataset (행정정보데이터세트의 데이터 품질평가 연구)

  • Song, Chiho;Yim, Jinhee
    • The Korean Journal of Archival Studies
    • /
    • no.71
    • /
    • pp.237-272
    • /
    • 2022
  • In 2019, the pilot project to establish a record management system for administrative information datasets started in earnest under the leadership of the National Archives. Based on the results of the three-year project by 2021, the improved administrative information dataset management plan will be reflected in public records-related laws and guidelines. Through this, the administrative information dataset becomes the target of full-scale public record management. Although public records have been converted to electronic documents and even the datasets of administrative information systems have been included in full-scale public records management, research on the quality requirements of data itself as raw data constituting records is still lacking. If data quality is not guaranteed, all four properties of records will be threatened in the dataset, which is a structure of data and an aggregate of records. Moreover, if the reliability of the quality of the data of the administrative information system built by reflecting the various needs of the working departments of the institution without considering the standards of the standard records management system is insufficient, the reliability of the public records itself can not be secured. This study is based on the administrative information dataset management plan presented in the "Administrative Information Dataset Recorded Information Service and Utilization Model Study" conducted by the National Archives of Korea in 2021. A study was conducted. By referring to various data, especially public data-related policies and guides, which are being promoted across the government, we would like to derive quality evaluation requirements in terms of records management and present specific indicators. Through this, it is expected that it will be helpful for record management of administrative information dataset which will be in full swing in the future.

Experimental Studies on the Properties of Epoxy Resin Mortars (에폭시 수지 모르터의 특성에 관한 실험적 연구)

  • 연규석;강신업
    • Magazine of the Korean Society of Agricultural Engineers
    • /
    • v.26 no.1
    • /
    • pp.52-72
    • /
    • 1984
  • This study was performed to obtain the basic data which can be applied to the use of epoxy resin mortars. The data was based on the properties of epoxy resin mortars depending upon various mixing ratios to compare those of cement mortar. The resin which was used at this experiment was Epi-Bis type epoxy resin which is extensively being used as concrete structures. In the case of epoxy resin mortar, mixing ratios of resin to fine aggregate were 1: 2, 1: 4, 1: 6, 1: 8, 1:10, 1 :12 and 1:14, but the ratio of cement to fine aggregate in cement mortar was 1 : 2.5. The results obtained are summarized as follows; 1.When the mixing ratio was 1: 6, the highest density was 2.01 g/cm$^3$, being lower than 2.13 g/cm$^3$ of that of cement mortar. 2.According to the water absorption and water permeability test, the watertightness was shown very high at the mixing ratios of 1: 2, 1: 4 and 1: 6. But then the mixing ratio was less than 1 : 6, the watertightness considerably decreased. By this result, it was regarded that optimum mixing ratio of epoxy resin mortar for watertight structures should be richer mixing ratio than 1: 6. 3.The hardening shrinkage was large as the mixing ratio became leaner, but the values were remarkably small as compared with cement mortar. And the influence of dryness and moisture was exerted little at richer mixing ratio than 1: 6, but its effect was obvious at the lean mixing ratio, 1: 8, 1:10,1:12 and 1:14. It was confirmed that the optimum mixing ratio for concrete structures which would be influenced by the repeated dryness and moisture should be rich mixing ratio higher than 1: 6. 4.The compressive, bending and splitting tensile strenghs were observed very high, even the value at the mixing ratio of 1:14 was higher than that of cement mortar. It showed that epoxy resin mortar especially was to have high strength in bending and splitting tensile strength. Also, the initial strength within 24 hours gave rise to high value. Thus it was clear that epoxy resin was rapid hardening material. The multiple regression equations of strength were computed depending on a function of mixing ratios and curing times. 5.The elastic moduli derived from the compressive stress-strain curve were slightly smaller than the value of cement mortar, and the toughness of epoxy resin mortar was larger than that of cement mortar. 6.The impact resistance was strong compared with cement mortar at all mixing ratios. Especially, bending impact strength by the square pillar specimens was higher than the impact resistance of flat specimens or cylinderic specimens. 7.The Brinell hardness was relatively larger than that of cement mortar, but it gradually decreased with the decline of mixing ratio, and Brinell hardness at mixing ratio of 1 :14 was much the same as cement mortar. 8.The abrasion rate of epoxy resin mortar at all mixing ratio, when Losangeles abation testing machine revolved 500 times, was very low. Even mixing ratio of 1 :14 was no more than 31.41%, which was less than critical abrasion rate 40% of coarse aggregate for cement concrete. Consequently, the abrasion rate of epoxy resin mortar was superior to cement mortar, and the relation between abrasion rate and Brinell hardness was highly significant as exponential curve. 9.The highest bond strength of epoxy resin mortar was 12.9 kg/cm$^2$ at the mixing ratio of 1:2. The failure of bonded flat steel specimens occurred on the part of epoxy resin mortar at the mixing ratio of 1: 2 and 1: 4, and that of bonded cement concrete specimens was fond on the part of combained concrete at the mixing ratio of 1 : 2 ,1: 4 and 1: 6. It was confirmed that the optimum mixing ratio for bonding of steel plate, and of cement concrete should be rich mixing ratio above 1 : 4 and 1 : 6 respectively. 10.The variations of color tone by heating began to take place at about 60˚C, and the ultimate change occurred at 120˚C. The compressive, bending and splitting tensile strengths increased with rising temperature up to 80˚ C, but these rapidly decreased when temperature was above 800 C. Accordingly, it was evident that the resistance temperature of epoxy resin mortar was about 80˚C which was generally considered lower than that of the other concrete materials. But it is likely that there is no problem in epoxy resin mortar when used for unnecessary materials of high temperature resistance. The multiple regression equations of strength were computed depending on a function of mixing ratios and heating temperatures. 11.The susceptibility to chemical attack of cement mortar was easily affected by inorganic and organic acid. and that of epoxy resin mortar with mixing ratio of 1: 4 was of great resistance. On the other hand, when mixing ratio was lower than 1 : 8 epoxy resin mortar had very poor resistance, especially being poor resistant to organicacid. Therefore, for the structures requiring chemical resistance optimum mixing of epoxy resin mortar should be rich mixing ratio higher than 1: 4.

  • PDF

Power and Offset Allocation for Spatial-Multiplexing MIMO System with Rate Adaptation for Optical Wireless Channels (다중 입출력 무선 광채널에서의 공간 다중화 기법의 적응적 전송을 위한 광출력과 오프셋 할당 기법)

  • Park, Ki-Hong;Ko, Young-Chai
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.36 no.1A
    • /
    • pp.8-18
    • /
    • 2011
  • Visible light communication (VLC) using optical sources which can be simultaneously utilized for illumination and communication is currently an attractive option for wireless personal area network. Improving the data rate in optical wireless communication system is challenging due to the limited bandwidth of the optical sources. In this paper, we design the singular value decomposition (SVD)-based multiplexing multi-input multi-output (MIMO) system to support two data streams in optical wireless channels. In order to improve the spectral efficiency, the rate adaptation using multi-level pulse amplitude modulation (PAM) is applied according to the channel condition and we propose the method to allocate the optical power, the offset and the size of modulation scheme theoretically under the constraints of the nonnegativity of the modulated signals, the aggregate optical power and the bit error rate (BER) requirement. The simulation results show that the proposed allocation method gives the better performance than the method to allocate the optical power equally for each data stream.

An Energy Aware Network Construction and Routing Method for Wireless Sensor Network (무선센서네트워크를 위한 에너지 인지형 네트워크 구성 및 라우팅 기법)

  • Hosen, A.S.M. Sanwar;Lee, Hyeak-Ro;Cho, Gi-Hawn
    • Journal of the Institute of Electronics and Information Engineers
    • /
    • v.49 no.9
    • /
    • pp.225-234
    • /
    • 2012
  • In Wireless Sensor Networks (WSNs) where deployed sensors are not stationary, the most important demand of is to design a cost effective and reliable network. This paper proposes an energy aware network construction and routing scheme, which is based on the hierarchical approach to distribute the task in some sensors in order to prolong the network lifetime. It aims to make even the energy consumption on constitute nodes. With the node hierarchy, the sink initiates the construction by electing gateway nodes in the network and the elected gateway nodes participate to form logical clusters by electing a cluster head in each cluster. Then, the cluster heads aggregate data from the sensing sensors and transmit the data to the sink through the gateway. Our simulation result illustrates that the proposed scheme provides a basement to reduce the source of energy dissipation in network construction, and as well as in data routing.

Effects of Exchange Rate, GDP, ODI on Export to the East Asia: Application the Panel FMOLS Approach (환율, GDP, 해외직접투자가 한국의 대동아시아 수출에 미치는 영향: 패널 FMOLS기법의 적용)

  • Kim, Chang-Beom
    • International Commerce and Information Review
    • /
    • v.14 no.3
    • /
    • pp.307-322
    • /
    • 2012
  • The purpose of this paper is to examine determinants of export to the East Asia region, using panel unit root, panel cointegration framework, panel VECM (vector error correction model), panel FMOLS (fully modified OLS). Different panel unit root tests confirm that the data series are integrated processes with unit roots. When applying cointegration tests to long-run effect for aggregate panel data, a primary concern is to construct the estimators in a way that does not constrain the transitional dynamics to be similar among different countries of the panel. The regression equations are estimated by various panel cointegration estimators. The panel data causality results reveal that exchange rates has unidirectional effects on export and GDP, and there exists bidirectional causality between export and GDP. Also, the results from the panel FMOLS tests overwhelmingly reject the null hypothesis of zero coefficient. The panel cointegrating vectors show that the export has positive relationship with the GDP and ODI (overseas direct investment).

  • PDF

Tracking Moving Objects Using Signature-based Data Aggregation in Sensor Network (센서네트워크에서 시그니처 기반 데이터 집계를 이용한 이동객체 트래킹 기법)

  • Kim, Yong-Ki;Kim, Young-Jin;Yoon, Min;Chang, Jae-Woo
    • Journal of Korea Spatial Information System Society
    • /
    • v.11 no.2
    • /
    • pp.99-110
    • /
    • 2009
  • Currently, there are many applications being developed based on sensor network technology. A tracking method for moving objects in sensor network is one of the main issue of this field. There is a little research on this issue, but most of the existing work has two problems. The first problem is a communication overhead for visiting sensor nodes many times to track a moving object. The second problem is an disability for dealing with many moving objects at a time. To resolve the problems, we, in this paper, propose a signature-based tracking method using efficient data aggregation for moving objects, called SigMO-TRK. For this, we first design a local routing hierarchy tree to aggregate moving objects' trajectories efficiently by using a space filtering technique. Secondly, we do the tracking of all trajectories of moving objects by using signature in a efficient way, our approach generates signatures to method. In addition, by extending the SigMO-TRK, we can retrieve the similar trajectories of moving objects for given a query. Finally, by using the TOSSIM simulator, we show that our signature-based tracking method outperforms the existing tracking method in terms of energy efficiency.

  • PDF