• Title/Summary/Keyword: Big data Processing

Search Result 1,063, Processing Time 0.029 seconds

Visualization Algorithm for Similarity Connection based on Data Transmutability (데이터 변형성 기반 유사성 연결을 위한 시각화 알고리즘)

  • Kim, Boon-Hee
    • The Journal of the Korea institute of electronic communication sciences
    • /
    • v.9 no.11
    • /
    • pp.1249-1254
    • /
    • 2014
  • Big data based on numerous data made by the people are used in order to obtain useful information. We can obtain more useful information if it can apply machine learning techniques added deformation of human memory on the characteristics of the computer program. And big data is predicted by using these conclusions. Humans are used to remember similar data as an original data, so big data processing technology should reflect these human characteristics. In this study, this algorithm to provide the selectivity of information is proposed. This algorithm is the technology to reflect the above factors. This algorithm is selected the data with high selectivity to determine similar data based on the deformation characteristics of the data.

Multi-Attribute based on Data Management Scheme in Big Data Environment (빅 데이터 환경에서 다중 속성 기반의 데이터 관리 기법)

  • Jeong, Yoon-Su;Kim, Yong-Tae;Park, Gil-Cheol
    • Journal of Digital Convergence
    • /
    • v.13 no.1
    • /
    • pp.263-268
    • /
    • 2015
  • Put your information in the object-based sensors and mobile networks has been developed that correlate with ubiquitous information technology as the development of IT technology. However, a security solution is to have the data stored in the server, what minimal conditions. In this paper, we propose a data management method is applied to a hash chain of the properties of the multiple techniques to the data used by the big user and the data services to ensure safe handling large amounts of data being provided in the big data services. Improves the safety of the data tied to the hash chain for the classification to classify the attributes of the data attribute information according to the type of data used for the big data services, functions and characteristics of the proposed method. Also, the distributed processing of big data by utilizing the access control information of the hash chain to connect the data attribute information to a geographically dispersed data easily accessible techniques are proposed.

A Novel Perceptual Hashing for Color Images Using a Full Quaternion Representation

  • Xing, Xiaomei;Zhu, Yuesheng;Mo, Zhiwei;Sun, Ziqiang;Liu, Zhen
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.9 no.12
    • /
    • pp.5058-5072
    • /
    • 2015
  • Quaternions have been commonly employed in color image processing, but when the existing pure quaternion representation for color images is used in perceptual hashing, it would degrade the robustness performance since it is sensitive to image manipulations. To improve the robustness in color image perceptual hashing, in this paper a full quaternion representation for color images is proposed by introducing the local image luminance variances. Based on this new representation, a novel Full Quaternion Discrete Cosine Transform (FQDCT)-based hashing is proposed, in which the Quaternion Discrete Cosine Transform (QDCT) is applied to the pseudo-randomly selected regions of the novel full quaternion image to construct two feature matrices. A new hash value in binary is generated from these two matrices. Our experimental results have validated the robustness improvement brought by the proposed full quaternion representation and demonstrated that better performance can be achieved in the proposed FQDCT-based hashing than that in other notable quaternion-based hashing schemes in terms of robustness and discriminability.

A Study on Structural Holes of Privacy Protection for Life Logging Service as analyzing/processing of Big-Data (빅데이터 분석/처리에 따른 생활밀착형 서비스의 프라이버시 보호 측면에서의 구조혈 연구)

  • Kang, Jang-Mook;Song, You-Jin
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.14 no.1
    • /
    • pp.189-193
    • /
    • 2014
  • SNS (Social Network Service) has evolved to life-friendly service with the combination of local services. Unlike exsiting mobile services, life-friendly service is expected to be personalized with gathering of local information, location information and social network service information. In the process of gathering various kinds of information, Big-data technology and Cloud technology is needed. The effective algorithem has researched for this already, however the privacy protection model hasn't researched enough in life-friendly service or big-data using circumstance. In this paper, the privacy issue is dealt with in terms of 'Structure hole', and the privacy issue comes from big-data technology of life-friendly service.

Cost-Effective MapReduce Processing in the Cloud (클라우드 환경에서의 비용 효율적인 맵리듀스 처리)

  • Ryu, Wooseok
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2018.10a
    • /
    • pp.114-115
    • /
    • 2018
  • This paper studies a mechanism for cost-effective analysis of big data in the cloud environment. Recently, as a storage of electronic medical records can be managed outside the hospital, there is a growing demand for cloud-based big data analysis in small-and-medium hospitals. This paper firstly analyze the Amazon Elastic MapReduce which is a popular cloud framework for big data analysis, and proposes a cost model for analyzing big data using Amazon EMR with less cost. Using the proposed model, the user can construct a cost-effective computing cluster, which maximize the effectiveness of the analysis per operational cost.

  • PDF

Transformer-based Language Recognition Technique for Big Data (빅데이터를 위한 트랜스포머 기반의 언어 인식 기법)

  • Hwang, Chi-Gon;Yoon, Chang-Pyo;Lee, Soo-Wook
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2022.10a
    • /
    • pp.267-268
    • /
    • 2022
  • Recently, big data analysis can use various techniques according to the development of machine learning. Big data collected in reality lacks an automated refining technique for the same or similar terms based on semantic analysis of the relationship between words. Big data is usually in the form of sentences, and morphological analysis or understanding of the sentences is required. Accordingly, NLP, a technique for analyzing natural language, can understand the relationship of words and sentences. In this paper, we study the advantages and disadvantages of Transformers and Reformers, which are techniques that complement the disadvantages of RNN, which is a time series approach to big data.

  • PDF

A Study on Security Improvement in Hadoop Distributed File System Based on Kerberos (Kerberos 기반 하둡 분산 파일 시스템의 안전성 향상방안)

  • Park, So Hyeon;Jeong, Ik Rae
    • Journal of the Korea Institute of Information Security & Cryptology
    • /
    • v.23 no.5
    • /
    • pp.803-813
    • /
    • 2013
  • As the developments of smart devices and social network services, the amount of data has been exploding. The world is facing Big data era. For these reasons, the Big data processing technology which is a new technology that can handle such data has attracted much attention. One of the most representative technologies is Hadoop. Hadoop Distributed File System(HDFS) designed to run on commercial Linux server is an open source framework and can store many terabytes of data. The initial version of Hadoop did not consider security because it only focused on efficient Big data processing. As the number of users rapidly increases, a lot of sensitive data including personal information were stored on HDFS. So Hadoop announced a new version that introduces Kerberos and token system in 2009. However, this system is vulnerable to the replay attack, impersonation attack and other attacks. In this paper, we analyze these vulnerabilities of HDFS security and propose a new protocol which complements these vulnerabilities and maintains the performance of Hadoop.

Extracting User-Specific Advertising Keywords Based on Textual Data Mining from KakaoTalk (카카오톡에서의 텍스트 데이터 마이닝 기반의 사용자별 적합 광고 키워드 도출 )

  • Yerim Jeon;Dayeong So;Jimin Lee;Eunjin (Jinny) Jo;Jihoon Moon
    • Annual Conference of KIPS
    • /
    • 2023.05a
    • /
    • pp.368-369
    • /
    • 2023
  • 대화 데이터 기반 광고 추천은 광고 마케팅에서 고객 맞춤형 광고 제공, 마케팅 효과 극대화 등을 위한 중요한 기술로 주목받고 있다. 본 논문에서는 모바일 인스턴스 메신저인 카카오톡 대화창에서 발생한 텍스트 데이터를 기반으로 대화 내용을 분석하여 대화 주제별 적절한 광고 키워드를 제안한다. 이를 위해 주제별 대화 내용을 미용, 식음료, 상거래로 세분하고 KoNLPy 의 Okt 를 이용하여 텍스트 전처리를 수행하고 키워드별로 빈도수를 뽑아 워드 클라우드를 제시한다. 또한, 잠재 디리클레 할당(Latent Dirichlet Allocation, LDA)을 기반으로 대화 주제를 세분화한 뒤 라벨링을 통해 주제별 대화 키워드를 분석한다. 실험 결과, 대화 주제를 온라인 쇼핑, 헤어, 뷰티 관리, 음식으로 나눌 수 있었으며, 토픽별 상위 키워드를 Word2Vec 을 통해 특정 단어와 유사한 키워드를 도출하여 적절한 광고 키워드를 제시할 수 있었다.

Understanding the Food Hygiene of Cruise through the Big Data Analytics using the Web Crawling and Text Mining

  • Shuting, Tao;Kang, Byongnam;Kim, Hak-Seon
    • Culinary science and hospitality research
    • /
    • v.24 no.2
    • /
    • pp.34-43
    • /
    • 2018
  • The objective of this study was to acquire a general and text-based awareness and recognition of cruise food hygiene through big data analytics. For the purpose, this study collected data with conducting the keyword "food hygiene, cruise" on the web pages and news on Google, during October 1st, 2015 to October 1st, 2017 (two years). The data collection was processed by SCTM which is a data collecting and processing program and eventually, 899 kb, approximately 20,000 words were collected. For the data analysis, UCINET 6.0 packaged with visualization tool-Netdraw was utilized. As a result of the data analysis, the words such as jobs, news, showed the high frequency while the results of centrality (Freeman's degree centrality and Eigenvector centrality) and proximity indicated the distinct rank with the frequency. Meanwhile, as for the result of CONCOR analysis, 4 segmentations were created as "food hygiene group", "person group", "location related group" and "brand group". The diagnosis of this study for the food hygiene in cruise industry through big data is expected to provide instrumental implications both for academia research and empirical application.

A study on development method for practical use of Big Data related to recommendation to financial item (금융 상품 추천에 관련된 빅 데이터 활용을 위한 개발 방법)

  • Kim, Seok-Soo
    • Journal of the Korea Society of Computer and Information
    • /
    • v.19 no.8
    • /
    • pp.73-81
    • /
    • 2014
  • This study proposed development method for practical use techniques compromise data storage layer, data processing layer, data analysis layer, visualization layer. Data of storage, process, analysis of each phase can see visualization. After data process through Hadoop, the result visualize from Mahout. According to this course, we can capture several features of customer, we can choose recommendation of financial item on time. This study introduce background and problem of big data and discuss development method and case study that how to create big data has new business opportunity through financial item recommendation case.