Automatic Generation of MAEC and STIX Standards for Android Malware Threat Intelligence

Abstract

Due to the increasing number of malicious software (also known as malware), methods for sharing threat information are being studied by various organizations. The Malware Attribute Enumeration and Characterization (MAEC) format of malware is created by analysts, converted to Structured Threat Information Expression (STIX), and distributed by using Trusted Automated eXchange of Indicator Information (TAXII) protocol. Currently, when sharing malware analysis results, analysts have to manually input them into MAEC. Not many analysis results are shared publicly. In this paper, we propose an automated MAEC conversion technique for sharing analysis results of malicious Android applications. Upon continuous research and study of various static and dynamic analysis techniques of Android Applications, we developed a conversion tool by classifying parts that can be converted automatically through MAEC standard analysis, and parts that can be entered manually by analysts. Also using MAEC-to-STIX conversion, we have discovered that the MAEC file can be converted into STIX. Although other researches have been conducted on automatic conversion techniques of MAEC, they were limited to Windows and Linux only. In further verification of the conversion rate, we confirmed that analysts could improve the efficiency of analysis and establish a faster sharing system to cope with various Android malware using our proposed technique.

1. Introduction

Recently, cyber threat has become one of the most serious threats to the economy and national security. It threatens the security of social infrastructures which are directly connected with the lives of people. To cope with such the situation, the United States has made efforts to share cyber threat information between public and private institutions by establishing standards for cyber threat information sharing and enacting related laws based on DHS [1]. We believe that cyber threat information sharing would upgrade the cyber countermeasure and analysis of malware, and hacking information could be utilized as big data for defending intelligent attacks. By sharing such cyber threat information, existing attacks can be mitigated more effectively due to security firms and other organizations that can identify attack trends as well as the evolving new techniques of these attacks so that countermeasures can be built appropriately. We believe that cyber threat information sharing can lead to the following effects. First, effective defense against existing attacks. Second, quick detection of attack trends and the emergence of new techniques. Third, the capability to swiftly build a proper countermeasure security system from information gathered in advance.

MITRE, a nonprofit research and development center under DHS, announced MAEC in 2010 and CybOX standards in 2012 for sharing malware analysis. In 2010, MITRE started development to share automate cyber threat information and announced TAXII and STIX in 2013[2]. As stated in the introduction MAEC is used by malware analysts and reverse engineers to share information about malware. STIX uses the MAEC schema to describe attack techniques related to malware in detail. In 2018, MITRE released MAEC 5.0, which provides features for malware sharing system in Android and iOS to cope with malware in mobile environments[3]. However, in the case of MAEC, analysts have to write their data based on the analyzed results. Therefore, research is underway on technologies that can automatically identify and input components[4][5]. Although MAEC 5.0 was released recently, additional research and development are required on automatic conversion techniques through the analysis of Android malware. In this paper, for the establishment of an efficient and accurate malware sharing system for Android, we first proposed and implement a technique of creating MAEC through APK file analysis and automate conversion into STIX. Our proposed contributions are as follows:

▪ Analyze the malware analysis method defined in MAEC 5.0 and performed the task of matching with the analyzed result through our developed analysis system.

▪ Develop technology to automatically convert Android malware analysis results to the MAEC 5.0 standard.

▪ Calculate the rate of automation to see how well it performs.

▪ Attach the implementation results of the technology to provide Android malware sharing by changing to STIX based on MAEC 5.0.

In Section 2, we explain the cyber threat information sharing system such as MAEC and STIX. We describe Android malware definition, dynamic analysis technique in Section 3. In Section 4, we introduce the techniques for the automatic conversion of static and dynamic analysis results to MAEC. In Section 5, we present the experimental results and conclude our study in Section 6.

2. Background

In this section, STIX and MAEC, the standards for sharing cyber threat information, are discussed in detail. NIST issued the "Guide to Cyber Threat Information Sharing" document in October 2016[6]. This document describes the current problems, benefits of sharing cyber threat information with organizations, and summarizes the features that should be provided for sharing. It is limited at the time, due to the lack of sharing information. Threats were not described in detail and were developed in various forms (xls, PDF, RSS, xml, e-mail, etc.) In order to solve it, there was another proposal of MAEC and STIX versions in 2017, which are currently being used.

2.1 MAEC

MAEC is a language for expressing structured information on malware based on attributes such as malware behavior, artifacts, and attack patterns[7]. By eliminating ambiguity and inaccuracies that may exist in the description of malware and reducing reliance on signatures, it is possible to improve communication between person-to-person, person-to-tool, and tool-to-tool. In addition, it is possible to develop a faster countermeasure method by using a function within a case of malware[8][9]. It also enables association, integration, and automation, which can reduce the possibility of duplication of researchers' analysis of malware. In June 2014, MAEC 4.1 was introduced, and MAEC 5.0 was released in October 2017. In the previous MAEC 4 version, MAEC Bundle, Package, and Container are classified according to usage. Each model has its own XML schema. MAEC 5.0 represents a significant refactoring and simplification of previous MAEC data models. Broadly, the MAEC 5.0 development goals were to simplify and refactor any components that were overly complicated or nested, to deprecate any unused components, and to align with the design principles of STIX 2[10].

2.2 STIX

STIX (Structured Threat Information eXpression) is a structured language for describing cyber threat information. Users can share, store, and analyze cyber threat information in a consistent way using STIX [11] The leading language for threat information within the United States is STIX. It is a standard language being developed in collaboration with any and all interested parties for the specification, capture, characterization, and communication of standardized cyber threat information. It does so in a structured fashion to support more effective cyber threat management processes and the application of automation. STIX provides a common mechanism for addressing structured cyber threat information across and among this full range of use cases improving consistency, efficiency, interoperability, and overall situational awareness. It consists of eight components. Users can express threat information using each component depending on different factors. The implementation has been succinctly changed from existing XML to JSON[12].

3. Android Malware Analysis

In this section, we introduce methods to acquire the necessary content for automatic conversion to MAEC through Android malware analysis. Using existing Android malware and dynamic analysis tools, it is possible to check whether the output is available for automatic conversion.

3.1 Android malware static analysis

Different static analysis tools can be used for analyzing Android applications. Apktool is a reverse engineering tool that can decompile APK files into smali code[13]. It is also possible to decode an APK, modify the extracted resource, rebuild it, and repackage it into a new APK. The result of decompiling an APK file with apktool contains an AndroidManifest.xml, images, XML files for UI layout, values, and a source file decoded in the form of smali code. Parsed AndroidManifest.xml files can be used for MAEC conversion. Certificates, signatures, etc. can also be retrieved in the Static features in MAEC 5.0. Dex2jar is a reverse engineering tool that can convert classes.dex files extracted from APK file into JAR files. Using this tool, Java code can be obtained using JAR extracted from classes.dex files[14]. Java decompiling tools like jdx and jd-gui can be used for checking the Java source code from the extracted JAR file. The extracted Java code appears similar to the original, so source level analysis is possible. JEB is a commercial tool, but it has the advantage of inserting APK files or DEX files as input data and seeing results of dex2jar, Java code, and apktool as a result of smali code[15]. It also provides a cross-reference function to track the call relationship of a function or variable, allowing faster analysis. If it can be restored in the original source form, it can be applied to MAEC by confirming the contents of the API call.

3.2 Android malware dynamic analysis

3.2.1 Existing dynamic analysis method

The main executable code can sometimes be hidden from decompiling an APK file using static analysis tools due to various solutions to protect the app. For example, the actual loading function is hidden using Java's reflection API, or the executable code encryption technique causes the main executable code part to dynamically decompress (or decrypt) into memory during runtime. In this case, it is limited when running static analysis alone, therefore the app needs to be analyzed dynamically for further investigation. In the case of Android malware dynamic analysis, there are various methods such as analyzing logs by connecting ADB to a rooted phone and using an emulator[16]. One can also take advantage of the dynamic analysis tools available from various companies.

3.2.2 Our dynamic analysis method

In this study, we used our own dynamic analysis platform and constructed it to monitor malware, from sensitive APIs to system calls in SDK to NDK. Extract the events received from the main activities, services, and broadcast receivers from the app's AndroidManifest.xml file. It creates a process that records the analysis log in real-time before running the app. At this time, record the analysis log using the modified logcat so that the log cannot be deleted. Then run the main activity extracted from the app. When the main activity is executed, the process of the app is created. If the main activity of the app does not exist, the app process is created when the service is first executed. After executing the main activity, all services included in the app are executed to improve code coverage, and all intent events received from the broadcast receiver are executed. After that, the UI event is randomly generated with the Monkey tool to test the app.

When the dynamic analysis process is completed, the recorded dynamic analysis log is collected, the Android is stopped, and the analysis is terminated. A total of 153 behaviors are defined in the MAEC specification. There are a total of 131 behaviors related to mobile malware and can be declared using Android dynamic analysis data such as capture camera input, capture GPS data, encrypt data, encrypt files, etc. These traces persist after system reboot and steal SMS database is included. In addition, the JSON file classified as malware action is declared for actions such as connect to IP address, call library function, create files, and create threads. In this case, further analysis of system calls should be conducted. Section 4 shows an example of identifying these behaviors.

3.3 Verification by many AV vendors using VirusTotal

VirusTotal is a website that provides free file checking [21] by using up to 70 different virus scanning software products. The files to be scanned can be uploaded via a website or e-mail; up to 128MB Analysis results can be retrieved by public or private APIs by importing HTTP Post using the issued token. Scanned results of each vendor display detected (true or false), version, and update. Based on this information, it is possible to output the result of MAEC 5.0 x-maec-avclass.json file. The date of scan, the date of submission, the result of detection, the name of the malware classified from the vendor, and AV tool name, and engine version etc. of each vendor can be retrieved [23].

4. Definition for automatic generation of Android malware

This section identifies requirements to automatically convert Android malware to MAEC. In the case of MAEC, it is divided into 20 files. First, some automatic conversion files, the files that need to be created, and the explanation of automatic conversion function for each part. Among the automatically convertible files, the parts that need to be created in detail are specified separately. Table 1. below shows the files that can be converted automatically in MAEC 5.0 and the files that require to be created.

Table 1. Automatically generated file classification

There are cases where it is possible to automatically convert each of these JSON files, and some areas may not be automatically converted even in a JSON file. For example, in the case of the comment property in analysis-metadata.json, this is the area that the analyst should add to the comment. These parts cannot be converted automatically and are the parts that the analyst should enter later. The following is an explanation of the contents of automatically converting each JSON file from the analysis result.

In the case of Analysis-metadata.json, it shows the analyzed version, start time, operation time, etc. This information is obtained by storing the information of the definite and dynamic analyzer that the initial analyst constructed in the database, separating the fixed value and the fluctuation value parsing possibility through the query.

In the case of Collection.json, the grouping is performed based on the static and dynamic analysis results by displaying the association between the objects defined in MAEC. For example, if a process that compresses a file and sends a socket is regarded as a data leak, these operations are grouped and defined as a rule in advance, and the relationship is displayed when such an action is sequentially logged.

For the Api-call.json file, we use the static and dynamic analysis to analyze the system calls that are called from the Java API and the source code that the Android app calls. We then record the function names, parameters, and return values. In particular, when the API is called with a return value, we can create value through the dynamic analysis hooking log for the returned value.

In the case of Behavior-json, we can define and use behaviors that malware performs based on the Android API and system call. The following Table 2. shows examples analyzed based on action and API call for behavior definition.

Table 2. Example of Action based classification

For the Binary-obfuscation.json file, we use the open-source project APKiD to identify the obfuscation and packer tools used by malware (e.g. DexGuard) and record obfuscation techniques applied to malware and techniques used in obfuscation tools (e.g. code encryption). However, since obfuscation and packer techniques were developed recently, additional obfuscation and packer detection techniques should be applied.

In the case of the Capability.json file, it is possible to output the system call confirmation that detects the environment using gdb, confirmation of the rebooting process, etc. as logs related to environment detection, system access, reboot, etc.

For the Malware-action.json file, we use dynamic analysis to define the malware activity that the malware instance performs based on the Android Java API and system calls actually made by the application. An instance of a malware refers to a Java API or system call that is called for the execution of an API call type and an action and specifies an operation for accessing a directory and a file. Examples of malware-actions are shown in the Table 3. below. In the case of the SDK, it refers to the behavior occurring in the Java domain. In the case of the NDK, it refers to an action using the SO file.

Table 3. Example of API call-based classification

In addition, existing research on dynamic analysis defines various behaviors for Android analysis results. According to TRAPDROID, they defined the behavior characteristics for system call and Binder transaction in 7 categories including Network, File system, Process, and Injection[18]. According to Y Zhang et al. they have proposed a system called VetDroid to identify all calls to the Android API and define behavior by checking how resources are used[19]. However, VetDroid has been monitoring through rooting and it has been difficult to cope with recent intelligent malware. According to O Somarriba et al. defines a behavior detection rule to distinguish malware family using SMS[20]. Also, it shows how each API can act on API usage. For example, the use of an API such as getApplicationInfo() was associated with Search Service behavior, and in the case of getIMEI, Utils behavior. According to Fan Yuhui et al., they classify the classifiable behaviors based on the permissions used in static analysis and dynamic analysis[21]. In static analysis, 17 behaviors were classified and in dynamic analysis, 5 behavioral groups were classified. These results are difficult to classify and judge the malware as a whole, although it is difficult to express them in the MAEC because they use their own analyzers to define and express the behavior for the analyzable parts. Therefore, in this paper, we declared all 131 dynamic analysis activities and defined the related action. Table 4. below is a comparing the number of behaviors defined in various studies and the number of dynamic analysis behaviors classified in this study.

Table 4. Various Malware behavior classification in previous work

In the dynamic-feature.json file, depending on the type of STIX network traffic, the malware performs the configuration for the actual operation and network traffic in the instance. In the case of a network, a tool such as TCPdump is used to monitor the actual network traffic and to check the specific ports.

The Malware-family.json file, the name.json file, and the x-maec-avclass.json file are based on the results obtained using the VirusTotal API. The malware family obtained from VirusTotal is fully identified, and the malware-family.json file can be created through most family classifications. In the case of x-maec-avclass, the VirusTotal result is shown as described in Section 3.

For the Malware-instance.json file, we use static and dynamic analysis to refer to the dynamic features, static functionality, and metadata of the malware instance. We then, record the analytical environment in which the family and malware run.

In the case of the Package.json file, it converts the malware instance and family into the standard output format. It then expresses the reference values for malware-family.json, name.json, and the like from the past. Also, the version, architecture, etc. of the analyzing system are applied as fixed values.

In the Static-feature.json file, it records the string contained in the obfuscation tool, packer tool, and header information used, extracts the certificate in the APK file from the static analysis stage, and parses the information. In addition, extraction results for other files such as image files included in the APK file would be created. The list of files extracted in the APK file static analysis step is automatically inputted into the database, which can be called through the query.

In the case of Process-tree-node.json, the dynamic analysis should be performed to obtain information such as parent_action_ref by checking the generation relationship between the parent process and the child process through gdb. To achieve this, we output the log using a command like pod of program_name for process monitoring and fetch it later.

In the case of Signature-metadata.json, it is not applicable in this study, but it can be applied automatically to various AV vendors. Signature information of various malware and, reference material for malware maker is required. It can be applied to AV vendors having a large amount of data by mapping. However, in this study, it is out of scope to collect a database of AV vendors.

In the case of a relationship.json that cannot be analyzed automatically, it requires analysis of using the correlation between one application and another at the same time. For example, when a vaccine app is executed before the execution of a banking application, which should extract information about your app. In cases like this, automation is difficult because the analysis information of other apps is required and the reference model to be compared has to be mapped.

Filed-data.json lists information about where malware started when it was first discovered, and so on. The malware starting point cannot be included in the static or dynamic analysis and should be based on the information held by the analyst.

In the case of malware-development-environment, it is difficult to identify information about the tools used, which should be written in the environment that the malware was produced. Of course, it is possible to check information about obfuscation and packing tools, but it is not possible to accurately grasp the tools (e.g. android studio, eclips, etc.) in which actual source code is developed.

Finally, relationship.json is the part that includes the relationship with other malicious apps. It reflects the results of other malicious apps and the analyst's opinion. After the analyst verifies the characteristics and analysis results of the app through its family name where an app contains the same source code, it is difficult to automatically, statically and dynamically analyze one app to express its relationship with other malicious apps. Therefore, this part should be accompanied by an additional process for mapping based on various databases in AV Vendors.

5. Evaluation

This section shows a number of features that can be automatically changed when automatic conversion to MAEC occurs. We calculated the automatic conversion rate through our Android malware and dynamic analysis system. MAEC.py created a rule-based on static analysis and dynamic analysis results. It performs the role of automatically converting input values based on the result of static analysis. It includes basic contents based on Package name, time, analysis information, Permission, etc. taken from static analysis. API calls from dynamic analysis, and actions defined based on system call information. As described above, the rule was created based on call information in accordance with the MAEC standard and was automatically converted according to the rule matching result

Table 5. MAEC 5.0 Format automatic generation rate

5.1 Behavior automatic generation rate

It is not possible to automate 17 behaviors that are not performed in mobiles such as test-for-firewall, persist-after-os-changes, and persist-after-hardware-changes. As our platform targets application testing, we do not automate the content that requires external connection actions, such as prevent-concurrent-execution.

Table 6. Behavior automatic generation rate

5.2 Action automatic generation rate

It is impossible to create and automate 68 items that are not made in mobile, such as Add-windows-hook, create-window, and enumerate-windows. It is difficult to automate the actions for content that has not been defined in the past in rules such as create-critical-section, create-semaphore, etc. In the case of the action part, it represents a lot of the internal operation and the network behavior of the system, so tracing of the system call and analysis of the network packet should be performed simultaneously.

Table 7. Action automatic generation rate​​​​​​​

5.3 Capability name automatic generation rate

It is difficult to confirm persistence through the analysis of malicious apps and dynamic analysis or confirm the propagation and persistence through device infection due to the characteristics of the analysis tool. Also, the following three areas such as persistence, infection-propagation, and availability-violation are impossible to perform.

Table 8. Capability name automatic generation rate​​​​​​​

5.4 Entity Associations automatic generation rate

File system entity, network entity, and process entity may be automatically inputted into Collection.json files. 100% auto-input is possible.

Table 9. Entity Associations automatic generation rate​​​​​​​

5.5 Common Attribute automatic generation rate

Capabilities.json and behavior.json are attributes that can be expressed mostly but are not linked with Common Vulnerabilities and Exposures (CVE) or Open Source Vulnerability Database (OSVDB). Also, in this analysis, it is impossible to express the timing of malicious action triggers because the logic bomb does respond properly.

Table 10. Common Attribute automatic generation rate​​​​​​​

5.6 Compare with existing method

To demonstrate the feasibility of the automation process, we measured the automation rate and time. Existing MAEC 5.0 does not provide the automatic conversion. However, for comparison purposes, we have identified components that do not change, such as analyst name and analytic OS. We determined that this was an automation area in the existing MAEC 5.0. The results of the automation rate are as follows.

Table 11. Automatic generation rate​​​​​​​

And we measured time. Conventional MAEC 5.0 and STIX creations cannot be compared for a time as the analysts enter them directly. Therefore, we measured the time varying according to the amount of log extracted from the analyzed APK. Logs extracted from static analysis results include information such as Permission, Activity, and Service. Dynamic analysis results include information obtained through hooking. We have measured the time taken from input to finally storing it in the database. The result is shown in the figure below. Based on this result, it can be seen that even if a lot of logs occur, it can be automatically converted in a short time compared to the time input by the analyst.

Table 12. Common Attribute automatic generation rate​​​​​​​

6. Implementation Result

The flow for the automatic conversion of malware for Android is shown in Fig. 1. below. When APK is received and dynamic analysis is performed and the result of VirusTotal is received, each received JSON file is subjected to primary transformation in MAEC form. Thereafter, the MAEC file is automatically transformed into STIX and applied to the secondary transformation.

Fig. 1. Android malware to MAEC automatic generation flow​​​​​​​

The result of this transformation is shown in the figure below. Fig. 2. shows the MAEC type, version, etc. when it is converted to MAEC. Also, the OS version and architecture that performed the operation will be described as well as the dynamic feature. The reference value of the behavior is automatically generated, and the value of the reference value is additionally provided as shown in Fig. 3. below. The behavior is mapped to the action and all of these are included in the report.

Fig. 2. MAEC transformation result

Fig. 3. Explain for behavior​​​​​​​

This is converted to the MAEC standard and finally converted to the STIX format. When converting to STIX format, the result parsed from MAEC is expressed as STIX. We found a MAEC-to-STIX tool, which automatically converts MAEC to STIX, but it has not yet been finalized for MAEC 5.0. For MAEC-to-STIX, it provides APIs and scripts for wrapping MAEC packages in STIX and extracting STIX indicators from dynamic analysis data captured at MAEC[17]. The following figure is a screen converted to STIX. Unlike MAEC, the initial type is changed, but the id value is the same. Generated time, malware label, etc., and show the contents expressed in MAEC as being useful for sharing.

7. Conclusion

Recently, various versions of Android malware has been increasing in number. Various types of malware are introduced by different countries. malware increasingly becoming intelligent enough to prove difficult to analyze. In order to respond and prepare for these increasing malware, countries such as the US as well as nations in Europe have established a cyber threat sharing system. The DHS created a standard to express cyber threat information and it can be used in a wide range to include international standards because they use automatic sharing and automatic search queries. In addition, various global corporations, US government agencies, and the European Union(EU) are currently promoting cyber threat information sharing systems using STIX. MAEC has been offered as an analytical standard for Android and iOS since 2017, as malware has become more common in mobile environments. In the past, mobile environments had to be linked to PC environments for malware analysis and sharing, which made. It was difficult to automate and organize the malware due to different parts. However, recently these problems have been solved, and the malware analysis result can be sufficiently provided. In summary, we first performed the automatic conversion of Android malware analysis to MAEC format. Since MAEC is not fully automated, we automated many parts using static analysis, dynamic analysis, and VirusTotal results analysis as well as working with analysts' opinions and existing analytical results. One can then establish an efficient and fast sharing system. If the analysis result is properly parsed into MAEC, and then implemented using open source tools such as MAEC-to-STIX provided by MITRE, it is possible to express it in an accurate sharing system. However, intelligent applications with technologies that interfere with static analysis such as obfuscation, packing, etc., along with interference with dynamic analysis such as routing detection, emulator detection, etc., have emerged, and, as result, further research should be conducted.