• Title/Summary/Keyword: SPARK

Search Result 1,526, Processing Time 0.043 seconds

A performance comparison for Apache Spark platform on environment of limited memory (제한된 메모리 환경에서의 아파치 스파크 성능 비교)

  • Song, Jun-Seok;Kim, Sang-Young;Lee, Jung-June;Youn, Hee-Yong
    • Proceedings of the Korean Society of Computer Information Conference
    • /
    • 2016.01a
    • /
    • pp.67-68
    • /
    • 2016
  • 최근 빅 데이터를 이용한 시스템들이 여러 분야에서 활발히 이용되기 시작하면서 대표적인 빅데이터 저장 및 처리 플랫폼인 하둡(Hadoop)의 기술적 단점을 보완할 수 있는 다양한 분산 시스템 플랫폼이 등장하고 있다. 그 중 아파치 스파크(Apache Spark)는 하둡 플랫폼의 속도저하 단점을 보완하기 위해 인 메모리 처리를 지원하여 대용량 데이터를 효율적으로 처리하는 오픈 소스 분산 데이터 처리 플랫폼이다. 하지만, 아파치 스파크의 작업은 메모리에 의존적이므로 제한된 메모리 환경에서 전체 작업 성능은 급격히 낮아진다. 본 논문에서는 메모리 용량에 따른 아파치 스파크 성능 비교를 통해 아파치 스파크 동작을 위해 필요한 적정 메모리 용량을 확인한다.

  • PDF

Preparation of $TiB_2$ Dispersed Cu Alloy by Spark Plasma Sintering

  • Kim, Kyong-Ju;Lee, Gil-Geun;Park, Ik-Min
    • Proceedings of the Korean Powder Metallurgy Institute Conference
    • /
    • 2006.09a
    • /
    • pp.523-524
    • /
    • 2006
  • The $TiB_2$ dispersion strengthened copper alloy was attracted as thermal and electrical functional material for the high mechanical strength, high thermal stability and good conductivity of $TiB_2$. In the present study, the focus is on the synthesis of $TiB_2$ dispersed copper alloy by spark plasma sintering process using copper oxide and titanium diboride as raw materials. The mechanical, thermal and electrical properties of sintered bodies were discussed with the sintering parameters, and developed microstructure and phase of sintered bodies.

  • PDF

Fabrication of Deep-Sub-Millimeter-Thick Compacts Using Spark Plasma Sintering

  • Ohashi, T.;Tanaka, T.;Oshiro, K.;Fujimori, H.;Kurisu, H.;Matsuura, M.;Yamamoto, S.
    • Proceedings of the Korean Powder Metallurgy Institute Conference
    • /
    • 2006.09a
    • /
    • pp.270-271
    • /
    • 2006
  • Nd-Fe-B type powder was sintered using spark plasma sintering method. Fabricated compact sintered at the temperature of $700\;^{\circ}C$, is found to be a composite magnet with Nd-Fe-Co-B and ${\alpha}-Fe$. The compact sintered at $700\;^{\circ}C$ shows slightly low coercivity and large remanent magnetization comparing to the compact sintered at $600^{\circ}C$ due to the formation of ${\alpha}-Fe$ phase, resulting in the large maximum energy product. Maximum energy product tends to decrease with decreasing thickness of sintered compacts below 0.5 mm in thickness.

  • PDF

EXPERIMENTAL INVESTIGATION ON THE EFFECT OF MAGNETIC FLUX TO REDUCE EMISSIONS AND IMPROVE COMBUSTION PERFORMANCE IN A TWO-STROKE, CATALYTIC-COATED, SPARK-IGNITION ENGINE

  • Govindasamy, P.;Dhandapani, S.
    • International Journal of Automotive Technology
    • /
    • v.8 no.5
    • /
    • pp.533-542
    • /
    • 2007
  • The two stroke spark ignition engine is the greatest contributor of the total vehicular pollution in a country like India. It is therefore an item that requires great attention in order to reduce fuel consumption and its concomitant pollution. The use of strong magnetic charge in the fuel line gives a complete and clean burn so that power is increased while operating expenses are reduced. The magnetic flux on the fuel line dramatically reduces harmful exhaust emissions while increasing mileage, thereby saving money and improving engine performance. It increases combustion efficiency and provides higher-octane performance. The experimental results show that the magnetic flux on fuel reduces the carbon monoxide emission up to 13% in a base engine, 23% in a copper-coated engine and 29% in a zirconia-coated engine.

A Damped Sinusoidal Electromagnetic Pulse Generator using a Charged Line (충전선로를 이용한 Damped Sinusoidal 전자기펄스 발생장치)

  • Ryu, Ji-Heon
    • Journal of the Korea Institute of Military Science and Technology
    • /
    • v.9 no.2 s.25
    • /
    • pp.136-142
    • /
    • 2006
  • A damped sinusoidal electromagnetic pulse generator was designed, fabricated and tested. The pulse generator consisted of an oscillator(a spark gap switch and an initially charged low impedance line) and a high impedance antenna. This generator was capable of producing damped sinusoidal pulses at closure of the spark gap switch. A Marx generator was employed to supply the Pulse generator with high voltage pulses. While the pulse generator was provided with the high voltage pulses of 200kV from the Marx generator, its output power was maximized by controlling the pressure of the gas contained in the spark gap switch. The output power of the damped sinusoidal electromagnetic pulse oscillator was 1.3GW and the amplitude of electric field radiated from the pulse generator was 4kV/m at the range of 25m.

k-NN Join Based on LSH in Big Data Environment

  • Ji, Jiaqi;Chung, Yeongjee
    • Journal of information and communication convergence engineering
    • /
    • v.16 no.2
    • /
    • pp.99-105
    • /
    • 2018
  • k-Nearest neighbor join (k-NN Join) is a computationally intensive algorithm that is designed to find k-nearest neighbors from a dataset S for every object in another dataset R. Most related studies on k-NN Join are based on single-computer operations. As the data dimensions and data volume increase, running the k-NN Join algorithm on a single computer cannot generate results quickly. To solve this scalability problem, we introduce the locality-sensitive hashing (LSH) k-NN Join algorithm implemented in Spark, an approach for high-dimensional big data. LSH is used to map similar data onto the same bucket, which can reduce the data search scope. In order to achieve parallel implementation of the algorithm on multiple computers, the Spark framework is used to accelerate the computation of distances between objects in a cluster. Results show that our proposed approach is fast and accurate for high-dimensional and big data.

A Study on Protection Device for Electrical Fire Prevention on Joint/Contact Badness Faults (접속·접촉불량 사고에 의한 전기화재 방지용 보호장치에 관한 연구)

  • Choi, Jung-Kyu;Kwak, Dong-Kurl;Lee, Seung-Chul;Lee, Bong-Seob;Park, Young-Jic
    • Proceedings of the KIPE Conference
    • /
    • 2014.07a
    • /
    • pp.5-6
    • /
    • 2014
  • According to 2013 fire statistical yearbook in the National Emergency Management Agency, the main cause of an electrical fire are classified to short circuit fault, overload fault, electric leakage, partial disconnection, and joint/contact badness. The occurrence factor of fire is electric arc or spark accompanied with electrical faults. Residual Current Protective Devices(RCDs) of high sensitivity type used at single phase (220V) cut off earth leakage and overload but the RCDs can not cut off electric arc or spark to be a main cause of electrical fire. To be improved on such problem, this thesis is proposed to a auxiliary control device for RCD cut off electric arc or spark.

  • PDF

A Development of Personalized Recommendation System using Spark GraphX (Spark GraphX를 활용한 개인 추천 시스템 개발)

  • Kim, Sungsook;Park, Kiejin;Lu, Sun
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2018.05a
    • /
    • pp.41-43
    • /
    • 2018
  • 소설 데이터는 인터넷 상의 수 많은 개인과 개인의 상호 작용에 의하여 연결되어 있으며, 이러한 데이터를 분석하여, 분석 대상에 내재하고 있는 구조와 특성을 파악하는 일은 중요하다. 특히, 개인 추천을 위해서는 개별 데이터들의 관계 그래프를 활용하여 빠르고 정확하게 추천 값을 도출하는 것이 효율적이다. 하지만, 기존 추천 기법으로는 신규 사용자와 아이템이 끊임없이 등장하는 상황을 즉각적으로 반영하기가 어렵고, 또한 많은 결측값을 포함하는 sparse 한 데이터일 경우에는 추천 시스템의 연산 공간과 시간에 많은 제약이 있다. 이에 본 논문에서는 Spark GraphX 를 활용한 개인 추천 시스템을 설계 및 개발하였으며, 이를 통하여 사용자와 아이템간에 내재하는 복합 요인이 반영된 그래프 기반 추천을 실행하여, 개인 추천 결과의 우수성을 확인하였다.

High Rate Denial-of-Service Attack Detection System for Cloud Environment Using Flume and Spark

  • Gutierrez, Janitza Punto;Lee, Kilhung
    • Journal of Information Processing Systems
    • /
    • v.17 no.4
    • /
    • pp.675-689
    • /
    • 2021
  • Nowadays, cloud computing is being adopted for more organizations. However, since cloud computing has a virtualized, volatile, scalable and multi-tenancy distributed nature, it is challenging task to perform attack detection in the cloud following conventional processes. This work proposes a solution which aims to collect web server logs by using Flume and filter them through Spark Streaming in order to only consider suspicious data or data related to denial-of-service attacks and reduce the data that will be stored in Hadoop Distributed File System for posterior analysis with the frequent pattern (FP)-Growth algorithm. With the proposed system, we can address some of the difficulties in security for cloud environment, facilitating the data collection, reducing detection time and consequently enabling an almost real-time attack detection.

Comparison of Lean Combustion Performance in a Spark-Ignition Engine Fueled with Natural Gas and Hydrogen (스파크점화 엔진에서 천연가스와 수소의 희박연소 성능 비교)

  • Park, Hyunwook;Lee, Junsun;Oh, Seungmook;Kim, Changup;Lee, Yonggyu;Kang, Kernyong
    • Journal of ILASS-Korea
    • /
    • v.26 no.4
    • /
    • pp.204-211
    • /
    • 2021
  • Lean combustion performance of natural gas and hydrogen was compared in a spark-ignition engine. The lean combustion engine operation with natural gas was limited due to combustion instability at an excess air ratio (EAR) above 1.8. The total hydrocarbon (THC) emissions increased significantly with increasing EAR. The nitrogen oxides (NOX) emissions were also high due to the limitation of increasing EAR. The lean combustion engine operation with hydrogen showed superior combustion stability as well as low THC and NOX emissions, even at high EARs. However, boosting technology was required to reach the high EARs.