• The Journal of Korean Institute of Next Generation Computing
• /
• v.13 no.2
• /
• pp.18-25
• /
• 2017

In computer science, an encoding is a standardization of converting information to one format for audio, video or text. Therefore, the encoding information of the data should be known to open and read it and there are algorithms detecting encoder of the data. However, some informations of data could be disappeared by packet loss when transmitted on network, especially, if the data is snatched by packet sniffing or eavesdropping from wireless communications. In this paper, we improve the performance of encoding detection algorithm of 'uchardet' program for multi-byte encoded data with errors based on bit-shift algorithm. To simulate the performance, we generated Korean and Japanese text data with errors that is removed some random bits at random positions. Then the detection algorithm are tested using the data and 'uchardet-bitshift' showed better performance than 'uchardet'. When Korean texts are used, 'uchardet' could detect perfectly with ≤0.005% errors but it showed 0% detection rate with ≥1% errors while 'uchardet-bitshift' detected perfectly with ≤0.05% errors and it showed correct detection cases with ≥1% errors. Japanese texts with errors tend to report falsely as Chinese encoding because Japanese texts include lots of Chinese characters. As a results, we improved encoding detection algorithms by applying bit shift operation.