• Journal of the Korea Institute of Information Security & Cryptology
• /
• v.33 no.6
• /
• pp.1043-1053
• /
• 2023

Malware analysts work diligently to analyze and counteract malware, while developers persistently devise evasion tactics, notably through packing and obfuscation techniques. Although previous works have proposed general unpacking approaches, they inadequately address techniques like OEP obfuscation and API obfuscation employed by modern packers, leading to occasional failures during the unpacking process. This paper examines the OEP and API obfuscation techniques utilized by various packers and introduces a system designed to automatically de-obfuscate them. The system analyzes the memory of packed programs, detects trampoline codes, and identifies obfuscated information, for program reconstruction. Experimental results demonstrate the effectiveness of our system in de-obfuscating programs that have undergone OEP and API obfuscation techniques.

• KIPS Transactions on Computer and Communication Systems
• /
• v.7 no.6
• /
• pp.155-164
• /
• 2018

Many malicious programs have been compressed or encrypted using various commercial packers to prevent reverse engineering, So malicious code analysts must decompress or decrypt them first. The OEP (Original Entry Point) is the address of the first instruction executed after returning the encrypted or compressed executable file back to the original binary state. Several unpackers, including PinDemonium, execute the packed file and keep tracks of the addresses until the OEP appears and find the OEP among the addresses. However, instead of finding exact one OEP, unpackers provide a relatively large set of OEP candidates and sometimes OEP is missing among candidates. In other words, existing unpackers have difficulty in finding the correct OEP. We have developed new tool which provides fewer OEP candidate sets by adding two methods based on the property of the OEP. In this paper, we propose two methods to provide fewer OEP candidate sets by using the property that the function call sequence and parameters are same between packed program and original program. First way is based on a function call. Programs written in the C/C++ language are compiled to translate languages into binary code. Compiler-specific system functions are added to the compiled program. After examining these functions, we have added a method that we suggest to PinDemonium to detect the unpacking work by matching the patterns of system functions that are called in packed programs and unpacked programs. Second way is based on parameters. The parameters include not only the user-entered inputs, but also the system inputs. We have added a method that we suggest to PinDemonium to find the OEP using the system parameters of a particular function in stack memory. OEP detection experiments were performed on sample programs packed by 16 commercial packers. We can reduce the OEP candidate by more than 40% on average compared to PinDemonium except 2 commercial packers which are can not be executed due to the anti-debugging technique.