A Systematic Study on Static Control Flow Obfuscation Techniques in Java

A Systematic Study on Static Control Flow Obfuscation Techniques in Java

Renuka Kumar Amrita Center for Cybersecurity Systems & Networks, Amritapuri Campus,
Amrita Vishwa Vidyapeetham
   Anjana Mariam Kurian Amrita Center for Cybersecurity Systems & Networks, Amritapuri Campus,
Amrita Vishwa Vidyapeetham
Abstract

Control flow obfuscation (CFO) alters the control flow path of a program without altering its semantics. Existing literature has proposed several techniques; however, a quick survey reveals a lack of clarity in the types of techniques proposed, and how many are unique. What is also unclear is whether there is a disparity in the theory and practice of CFO. In this paper, we systematically study CFO techniques proposed for Java programs, both from papers and commercially available tools. We evaluate 13 obfuscators using a dataset of 16 programs with varying software characteristics, and different obfuscator parameters. Each program is carefully reverse engineered to study the effect of obfuscation. Our study reveals that there are 36 unique techniques proposed in the literature and 7 from tools. Three of the most popular commercial obfuscators implement only 13 of the 36 techniques in literature. Thus there appears to be a gap between the theory and practice of CFO. We propose a novel classification of the obfuscation techniques based on the underlying component of a program that is transformed. We identify the techniques that are potent against reverse engineering attacks, both from the perspective of a human analyst and an automated program decompiler. Our analysis reveals that majority of the tools do not implement these techniques, thus defeating the protection obfuscation offers. We furnish examples of select techniques and discuss our findings. To the best of our knowledge, we are the first to assemble such a research. This study will be useful to software designers to decide upon the best techniques to use based upon their needs, for researchers to understand the state-of-the-art and for commercial obfuscator developers to develop new techniques.

Keywords: Code Obfuscation, Control Flow Obfuscation, Java, Opaque Predicate

1 Introduction

Obfuscation is a program transformation technique that renders a program unreadable to a human while preserving its semantics. A software developer uses static code obfuscation techniques to secure code and data from reverse engineering attacks. There are three types of obfuscation- lexical, data and control flow obfuscation. Lexical obfuscation replaces class, field and method identifiers with new random identifiers or words from a dictionary. Data obfuscation modifies data structures used by a program. Control flow obfuscation (CFO) alters the control flow path of a program. Malware authors use obfuscation to impede security analysts from reverse engineering their code or to evade automated program analyzers.

In this paper, we focus on control flow obfuscation techniques. While existing literature proposes several techniques, the differences between them is unclear. What is also unknown is if there is a disparity in the theory and practice of CFO. This knowledge can aid software developers to decide on the best techniques to use to secure their code. Commercial obfuscator developers and researchers can use this knowledge to advance the state-of-the-art and steer research in this area.

There are a few existing surveys on control flow obfuscation [26],[19],[27], [24],[11] [16]. Colberg et al. propose groundbreaking work on the taxonomy of obfuscating transformations. They classify 15 control flow transformations into four types - Opaque Predicate, Computation Transformation, Aggregation Transformation and Ordering Transformations. However, their classification is based on theoretical foundations alone. Schrittwieser et al. [16] classifies obfuscation techniques into three categories- data obfuscation, static code rewriting and dynamic code rewriting. They survey literature to analyze the strength of obfuscation against common adverserial goals.

Balakrishnan et al. [24] presents a literature survey of all code obfuscation techniques used to thwart static analysis and program disassembly. However, all the CFO techniques they propose is also discussed by Colberg et al. [26]. You et al. [11] discusses a survey of obfuscation techniques in Windows malware binaries. Majumdar et al. [27] survey the strengths and weaknesses of two CFO techniques - control flow flattening and opaque predicates. Xu et al. [48] compares and contrasts code-oriented and model-oriented obfuscation techniques to conclude that there are no secure and usable obfuscation techniques. Lim [46] evaluates various code obfuscation techniques and measures performance of each. Low et al. [19] discusses ways to create resilient and stealthy opaque predicates.

In this paper, we systematize knowledge of control flow obfuscation on Java programs by studying the techniques proposed in the literature and those implemented by obfuscation tools. We assemble a list of known techniques into a single study and classify it into 5 categories based on the component (or level) of a program on which the transformation is applied. Though we use the taxonomy proposed by Colberg et al. [26] as the basis of our research, we deviate substantially in our classification. We evaluate 13 obfuscation tools and determine that only 6 are viable for analysis. The remaining obfuscators were either unavailable for download, end-of-lived or had unresolvable program errors that made it unusable. Obfuscators used for Android programs can also be used for Java applications. However, certain Android obfuscators will not work on Java bytecode. Hence we evaluate tools that can obfuscate Java programs only.

In order to evaluate obfuscators, we choose a sample set of 16 programs with varying control flow properties. Each obfuscator is repeatedly applied to the 16 sample programs by varying their configuration parameters. The bytecode of the obfuscated program is then manually analyzed and compared with that of the original program to tabulate our findings. Wherever possible, we confirm our results with any documentation provided by the tool. We apply no lexical or data obfuscation and restrict our analysis to CFO only.

From our study, we identified a total of 43 unique techniques, 36 from literature and 7 from tools. Of the 36 techniques from literature, 12 are not implemented in any of the tools. Three of the most popular commercial obfuscators implement only 13 of the 36 techniques from the literature. Thus there appears to be a gap between the theory and practice of CFO. We assess the potency of each technique from the perspective of a human analyst and a program decompiler. We determine that techniques that are applied on a basic block are the most potent. However, majority of the techniques do not implement them, thus defeating the reliability of obfuscation. We identify the most popular techniques in tools and identify those that are incorrectly cited as control flow obfuscating. In the end, we discuss our findings from this study.

To summarize, we make the following contributions:

  • We survey literature for CFO techniques and identify 36 unique techniques from literature. We furnish examples of select techniques and identify those that are incorrectly cited as control flow obfuscating.

  • We evaluate 13 obfuscators and identify the 6 viable tools that can control flow obfuscate Java programs.

  • We evaluate each tool on a set of 16 programs with varying software characteristics by repeated application of each obfuscator with varying configuration parameters. We identify 7 novel techniques. Three of the most popular commercial obfuscators implement only 13 of the 36 techniques. Thus there appears to be a gap between the theory and practice of CFO. We also identify the most commonly applied obfuscation techniques in tools.

  • We propose a classification of the 43 techniques based on the component of a program that is transformed. We discuss the potency of the techniques, and identify those that are resilient to decompilers. We determine that majority of the obfuscators do not implement these techniques, thus defeating its purpose.

2 Running Example

Listing 1 is an iterative implementation of a binary search program in Java. The program will be used as the running example for the remainder of this paper. The example has all the components present in a typical program such as if-else condition, loop etc. The function binarySearch takes two parameters as input - an array and the element to be searched. It returns the index at which the element was found and otherwise.

3 Techniques in Literature

In this section we discuss techniques found in the literature grouped by the obfuscating paradigm used. Wherever possible, we provide examples of each technique using the running example. The code snippet on the left shows the original code. Obfuscated code is on the right. Examples for some techniques have been omitted when it is similar to already stated techniques or when there are other literature that cites them.

1class FindElement{
2   static int binarySearch(int arr[], int x){
3       int l = 0, r = arr.length - 1;
4       while (l <= r){
5           int m = l + (r-l)/2;
6           if (arr[m] == x)  // Check if x is present at mid
7               return m+1;
8           if (arr[m] < x) // If x greater, ignore left half
9               l = m + 1;
10           else // If x is smaller, ignore right half
11               r = m - 1;
12       }
13       return -1;
14   }
15}
16class BinarySearch extends FindElement{
17    public void printResult(int result,boolean isPresent) {
18       if(isPresent==false)
19           System.out.println(”Element not present”);
20       else
21           System.out.println(”Element was found at index  + result);
22   }
23   public static void main(String args[]){
24       FindElement ob1=new FindElement();
25       BinarySearch ob2=new BinarySearch();
26       boolean isPresent=false;
27       int arr[] = new int[args.length];
28       if (args == null) return;
29       for (int i = 0; i < args.length; i++)
30         arr[i] = Integer.parseInt(args[i]);
31
32       //print the array
33       for (int i =0; i < arr.length; i++)
34           System.out.println(arr[i]);
35       int index = ob1.binarySearch(arr, 10);
36       if(index!=-1){
37           isPresent=true;
38           ob2.printResult(index,isPresent);
39       }
40       else
41           ob2.printResult(index,isPresent);
42   }
43}
Listing 1: Running Example

3.1 Using Opaque Predicates

An opaque predicate is a predicate whose value is known apriori to the obfuscator. There are several ways in which an opaque predicate can be used to obfuscate a program [31], [1], [2], [3], [4], [10], [12], [13], [16], [17], [21], [26], [27], [28], [29], [30], [34], [35], [36], [37], [41], [42], [43], [44], [46]. Below we discuss the techniques that use opaque predicates. Existing literature also proposes algebraic techniques to create resilient opaque predicates; however, we do not study resilience of predicates in this work.Â

  1. Extending conditionals. An opaque predicate is inserted into a conditional expression of either an if-else statement or a loop. Here inserting the opaque predicate does not alter the evaluation of the expression [25], [26].

  2. Adding Redundant Operands. Opaque variables or expressions are added as operands to arithmetic expressions [25], [26].

  3. Dead Code Insertion. An opaque predicate that guards a set of statements is inserted into the program in such a way that the code block guarded by the predicate is never executed (dead code) [6], [10], [11], [15], [16], [20], [26], [40], [41], [44].

The example below shows all three techniques discussed above. In the obfuscated program, the original if statement is modified by adding an opaquely false predicate denoted by the variable bool. The arithmetic expression of the return statement is changed to also use a redundant operand called redop. The obfuscated code also contains an additional if statement guarded by the predicate bool whose consequent never gets executed.

if (arr[m] == x)
    return m+1;
boolean bool;
int redop = 1;
if (arr[m] == x && !bool)
    if(bool){
        /*deadcode*/
        return 0;
    }
    return (m+1) * redop;
}

3.2 Irrelevant Code Insertion

In this technique, the obfuscator inserts a set of irrelevant statements into the program [16]. Unlike Dead Code Insertion, the program executes the inserted statements. However, they may be irrelevant to it, or not present in the original program. The string variable str is the irrelevant code in the example.

if (isPresent == false)
   System.out.println(”Element
                 not present”);
if (isPresent == false) {
   String str=”Hello World!”;
   System.out.println(”Element
                 not present”);
}

3.3 Ordering obfuscation.

Ordering obfuscation changes the order of evaluation of expressions, loops, statements or methods while preserving dependencies [10], [16], [18], [25], [26], [46].

  1. Reordering Expressions. This technique changes the order of evaluation of sub-expressions without changing its valuation. The example below shows how to obfuscate a conditional of the if statement.

    if (arr[m] < x)
        l = m + 1;
    if (x >= arr[m])
        l = m + 1;
  2. Reordering Loop. The example shows how to change the evaluation of a loop index.

    for(int i=0;i<arr.length;i++)
        System.out.println(arr[i]);
    for(int i=arr.length-1;i>=0;i–)
        System.out.println(arr[i]);
  3. Reordering Statements. In this technique, an obfuscator reorders statements within a basic block to preserve dependencies [40], [44].

  4. Reordering Code Blocks. This technique reorders basic blocks while preserving dependencies.

  5. Reordering Methods. This techniques changes the order of its subroutines [11], [40]. This technique can generate n! different variants of a program, where n is the number of subroutines. This approach moves the structural position of the method in a file.

3.4 Instruction Substitution.

In this technique, the obfuscator replaces a sequence of instructions with an alternate sequence of semantically equivalent instructions [6], [11], [20], [40]. For instance, the goto at line 47 is replaced with iload_2 and ifeq instructions.

47 : goto 107
50 : iload_1
107: return
66  : iload_2
67  : ifeq 129
70  : iload_1
129 : return

A specific implementation of this technique has been mentioned in [50] called the Replacing Goto obfuscation. In this technique goto instructions are replaced with conditional branch instructions.

3.5 Control Flow Flattening

This technique works by first flattening the control flow graph of a program such that all the basic blocks have the same successor and predecessor. A dispatcher variable is then introduced to guide the control flow of the program [1], [3], [4], [6], [7], [12], [13], [14], [21], [22], [25], [34], [35], [37], [38], [39], [47]. In the example, the original code snippet has two basic blocks. The control flow flattened program encapsulates both the basic blocks into a switch block with each basic block forming a case statement. The dispatcher variable here is var. When var = 2, control flow enters the second case statement that contains the first basic block.

int arr[] = {2, 3, 4, 10, 40};
for (int i =0; i < arr.length; i++)
    System.out.println(arr[i]);
int var=2;
while(var!=0){
 switch(var){
  case 1:
     for(int i =0;i<arr.length;i++)
       System.out.println(arr[i]);
     var=0;
     break;
  case 2:
     int arr[] = {2, 3, 4, 10, 40};
     var=1;
     break;
  default:break;
  }
}

3.6 Method Transformation

This technique obscures method invocations. There are four types of transformation:

  1. Inline Methods. The technique replaces a method invocation with the body of the method [31], [5], [10], [15], [18], [24], [26], [44], [46]. For instance, the body of the function printResult replaces its invocation (line 17) in the running example.

  2. Outline Methods. This technique is the inverse of inlining; it replaces a sequence of statements with a function call [31], [5], [10], [24], [26], [44], [46].

  3. Clone Methods. This obfuscation modifies a method’s call site by invoking clones of the same method [31], [16], [24], [26]. The call to binarySearch is obfuscated by inserting clones of the same function.

    int result = binarySearch(arr, 10);
    int choice=1;
    int result;
    if(choice==1)
      result = binarySearch1(arr, 10);
    else
      result = binarySearch2(arr, 10);
  4. Interleave Methods. The technique merges two separate methods into a single method [31], [26].

3.7 Replacing if(non) null instructions with try-catch blocks

The instructions ifnull and ifnonnull are used for obfuscation. The ifnull instruction checks if the topmost element on the stack is null, whereas the ifnonnull instruction checks if the top most instruction is non-null. These instructions then jump to a location as specified by a label. These instructions are replaced by try-catch constructs to alter the control flow. This may be done either by inserting a statement that will cause an exception or by explicitly throwing an exception [32]. Below is an example of the former.

if (args == null) return;
for (int i = 0; i < args.length; i++)
  arr[i] = Integer.parseInt(args[i]);
try {
  for (int i = 0; i < args.length; i++)
    arr[i] = Integer.parseInt(args[i]);
} catch (NullPointerException ne) {
  return;
}

3.8 Loop Transformations

These transformations manipulate loops without changing their behaviour [16], [26]. It is of four types.

  1. Loop Fission. The technique splits a loop into multiple loops [21]. For instance, the for loop in the original program is split into two by causing the loops to iterate only half the array.

    for(int i=1;i<arr.length;i++)
        System.out.println(arr[i]);
    for(int i=1;i< arr.length/2;i++)
        System.out.println(arr[i]);
    for(int i=arr.length/2;
        i< arr.length;i++)
        System.out.println(arr[i]);
  2. Loop Blocking. This is a well-known optimization technique used by compilers for caching. Here, a loop is partitioned into small blocks. For instance, the loop in the original program is blocked by a block of size 2.

    for(int i=0;i< arr.length;i++)
        System.out.println(arr[i]);
    for (int j=0;j<arr.length;j+=2)
     for(int i=j; j< min(
        arr.length, j+2); i++)
         System.out.println(arr[i]);
  3. Loop Unrolling. Also known as loop unwinding, this is yet another compiler optimization that trades space for execution speed[5], [10], [46]. It reduces the number of iterations of a loop by increasing the size of the program. For instance, in the obfuscated program, the print statement is unrolled.

    for (int i =0; i < arr.length; i++)
       System.out.println(arr[i]);
    for (int i=0;i<arr.length; i=i+3) {
        System.out.print(arr[i]);
        System.out.println(arr[i+1]);
        System.out.println(arr[i+2]);
    }
  4. Replace with Equivalent Codes. This obfuscation inserts, modifies or removes statements, identifiers, and literals [33]. This is also called as Code Clone III. In the example below, the elemet arr[i] is assigned to another variable num.

    for (int i =0; i < arr.length; i++) {
       System.out.println(arr[i]);
    }
    for (int i =0; i < arr.length; i++){
       int num = arr[i];
       System.out.println(num);
    }
  5. Code Clone-Type IV. Here code snippets are modified to produce syntactically different, yet semantically identical variants [33]. For instance, an obfuscator replaces a for loop with a while loop.

  6. Inserting Dummy Loop. In this technique empty loops are inserted into the code. An alternative implementation is combining basic blocks into a loop.

  7. Intersecting Loops.This approach inserts intersecting loops into the control flow and guarded by an opaque predicate to skip the loop [50]. The opaque predicate causes the the inserted code to act like dead code.

3.9 Basic Block Fission

This technique splits the chosen code blocks into finer pieces and inserts opaque predicates and goto instructions in them [50]. The use of goto may cause the decompiler to fail in this case.

3.10 Remove Library and Program Idioms

The technique uses obfuscated version of library functions and removes/replaces programming idioms from the original program [16], [25], [26]. For example, System.out.println library call is replaced with a semantically equivalent implementation that use BufferedWriter.

System.out.println(”Element not
present”);
BufferedWriter bw = new
BufferedWriter(new
OutputStreamWriter(System.out));
bw.write(”Element not present”);

3.11 Convert a Reducible to Non-Reducible Flow-graph

In this technique, instructions that have no direct correspondence with the source language is used to obfuscate a program. This makes the transformation language-breaking or non-reducible [26]. A commonly used approach is to use the goto statement to express arbitrary control flow. This makes it a non-reducible flow-graph as the Java language itself can only express structured control flow (reducible flow-graph). This can easily be done by inserting a bogus jump to the middle of the body of a while loop using an opaque predicate that will never get executed. When the loop has multiple headers or entry points, it no longer becomes structured and hence will become non-reducible.

3.12 Table Interpretation

This technique is also called as Virtualization [8], [9], [23], [42], [26]. A program’s bytecode is converted into a custom instruction set that is executed by a VM interpreter included within the obfuscated application. This transformation induces additional runtime over head and hence is reserved only for extremely sensitive portions of an application.

3.13 Parallelizing the Code

In this technique, multiple processes are created to execute code blocks that have no data dependencies. This may be done either by creating dummy processes and running them in parallel with the instructions of the program or by splitting a sequential section of the application into multiple sections that can be executed in parallel. Code block that have data dependencies can be parallelized using synchronizing functions [15], [16], [25], [26].

Obfuscator Software Version Availability Input Evaluated
Zelix Klassmaster 6.1 Paid version Bytecode Y
Allatori 6.4 Trial version Bytecode Y
DashO 8.3.0 Trial version Bytecode Y
JBCO 2.5.0 Not-Available Bytecode N
JMOT 3.0 Open-source Bytecode Y
Sandmark 2.1 Open-source Bytecode Y
Jfuscator Unspecified Trial version Byte/Sourcecode Y
JOAD 2.1 Open-source Bytecode N
JShield Unspecified Not available Source code N
Arxan Unspecified Paid version Bytecode N
Cloakware Unspecified Not available Sourcecode N
Smokescreen 3.11 Not available Bytecode N
Codeshield 2.0 Not available Bytecode N
Table 1: List of Control Flow Obfuscators in the Study

4 Tools Evaluation

We identify 13 tools as listed in table 1 that support control flow obfuscation. We restrict our evaluation to control flow obfuscators and exclude lexical and data obfuscators such as Proguard. Of the 13 tools Only 6 tools are viable - DashO, Allatori, Zelix Klassmaster, Sandmark, JMOT, and JFuscator. JBCO is the only tool which we could not evaluate, yet is included since they furnish a detailed documentation of their techniques. For those that were not evaluated, we indicate why. In case of paid proprietary tools, we use their trial versions for evaluation, except for Zelix Klassmaster, for which we acquired a licensed version.

In order to conduct the study, we use a set of 16 synthetic programs with and without control flow constructs. We intended the programs to contain a heterogeneous mix of properties. The programs have one or more of the following characteristics: 1) non-nested if statements 2) non-nested if-else statements 3) one/many for or while loops 4) switch-case statements 5) nested conditional statements 6) nested try-catch blocks 7) use of complex data structures such as hashmaps 8) variable number of methods 9) accepts user inputs. In a case where the conditionals are nested, we introduce a nesting level of up to 3 levels.

The source/bytecode of each of the 16 programs are repeatedly supplied as input to each control flow obfuscator for different obfuscator parameters. We manually manually examine the bytecode output to identify the differences between the original and the transformed programs using the javap command. This is because, the decompiled output of a program may vary depending on the decompiler used. Certain obfuscations also produce programs that cannot be decompiled. Lastly, while some of the techniques do not appear to alter the control flow of the program from the perspective of source code, it is still a control flow obfuscation if it alters the control flow of the bytecode. We use JD-GUI decompiler wherever possible to verify our findings.

Table 2 details the list of obfuscation techniques supported by each tool.

4.1 Zelix Klassmaster

Zelix supports 3 levels of control flow obfuscation- light, normal and aggressive, and data obfuscation. Zelix uses only techniques proposed by the literature. They using static boolean variables as opaque predicates and insert emtpty if statements guarded by opaquely false predicates. Zelix also reorders the constant pool index, which is a structural obfuscation technique.

4.2 Allatori

This obfuscator is available as a plugin for Eclipse. It supports incremental obfuscation of code to allow for consistency with the previous obfuscated versions of a program.

Obfuscator Obfuscation Technique Zelix Irrelevant Code Insertion Extending Conditionals Opaque Branch Insertion Opaque Predicate Insertion Dead Code Insertion Instruction Substitution Convert Reducible to Irreducible Flowgraph Replacing if(non) Null Instructions with TCB Reordering Statements Allatori Method Reordering Instruction Substitution Reordering Statements Replacing if(non) Null Instructions with TCB Code Clone Type IV Using Opaque Predicate Dead Code Insertion Branch Inversion DashO Opaque Branch Insertion Dead Code Insertion Reordering Statements Goto Instruction Augmentation Insert Dummy Loop Replacing if(non) Null Instructions with TCB JMOT Data Sanitization JFuscator Irrelevant Code Insertion Instruction Substitution Obfuscator Obfuscation Technique JBCO Reordering the load instructions above the if instructions Replacing if(non) Null Instructions with TCB Adding Dead Code Switch Stmt. Building API Buffer Methods Building Lib Buffer Classes Goto Instruction Augmentation Converting Branches to jsr Instns. Finding and Reusing Duplicate Sequences Disobeying Constructor Conventions Partially Trapping Switch Stmts. Combining Try Blocks with Catch Blocks Indirecting if Instructions Sandmark Boolean Splitter Reordering Expressions Building API Buffer Methods Split Objects Class Splitter Interleave Methods Inline Methods Convert Reducible to Irreducible Flow Graph Reordering Statements Opaque Branch Insertion Dynamic Inliner Dead Code Insertion
Table 2: List of Techniques from Tools

4.3 Jbco

JBCO [45], [49] is an obfuscator developed by Sable Group, KTH University, that proposes a set of techniques specifically for bytecode obfuscation. The tool is a part of the Soot framework. We were not able to successfully evaluate all the techniques due to dependency errors while running the tool. However, the techniques they have created have been discussed in detail with examples in [49]. One thing to note is that depending on the specification of the virtual machine in which bytecode is run, some techniques may cease to work.

  1. Reordering load instruction above if instructions. This technique moves a local variable common to both the if and else condition to outside the if-else statement, thus removing redundant code along the branches.

  2. Adding Dead Code Switch Statements. It inserts a switch statement using an opaquely false predicate.

  3. Finding and Reusing Duplicate Sequences. This technique manipulates the fact that duplicate bytecode sequences occur in various parts of the code. The code sequences can be replaced with a single switched instance, thus altering the control flow as well as reducing the code size.

  4. Partially Trapping Switch Statements. Traps are conceptually similar to exception handlers, except that they are specific to bytecode. One way to obfuscate using trap handlers is to encapsulate a sequence of bytecode instructions that may belong to disparate set of code blocks, into a single trap handler. For instance, code blocks belonging to case statements of a switch-case block can be handled by a single trap handler.

  5. Disobeying Constructor Conventions. This technique manipulates invocations to super class constructor calls. For instance, invocation to a super class constructor is placed inside a try-catch block.

  6. Combining Try Blocks with Catch Blocks. While the try block must be followed by the catch block in the source code, at the bytecode, the ordering of the blocks is irrelevant. For the same reason, multiple try-catch blocks maybe combined, or a catch block may precede a try block.

  7. Indirecting if Instructions. In this technique, the if branch is indirected through a goto instruction. To avoid the goto from being removed by the compiler, it is enclosed within a try-catch block.

  8. Building API Buffer Methods. In this technique, a buffer method is inserted between the caller and the invocation to a Java API call.

  9. Building Library Buffer Classes. The obfuscator inserts a buffer class (with methods) in between a caller and the callee library class, thus introducing another level of indirection.

  10. Goto Instruction Augmentation. It splits a method into two sequential parts, uses goto to change the control flow of the program. The parts may optionally be reordered.

  11. Converting branches to jsr instructions. jsr instruction is similar to goto the only difference being that jsr pushes a return address to the stack. In this technique, if and goto targets are replaced with jsr instructions. The return address is saved in a register for use after returning from a jsr jump. jsr is supported only in Java version 6.0 and below as per the Oracle Documentation. In our experimental evaluation, we were unable to reproduce this obfuscation technique despite lowering the Java version.

4.4 DashO

DashO extensively uses try-catch blocks for altering the control flow of a program. The number of try-catch blocks can be varied from 1 to 10. It does not obfuscate compiler-generated classes such as the default constructor. DashO also uses nop instructions to increase code size.

4.5 Jmot

JMOT supports 4 levels of obfuscation - Light, Normal, Heavy and Insane. The tool has no control flow obfuscation techniques, and only supports data sanitization to protect strings used in the program. Strings are passed through a series of function calls named as hax0, hax1 …hax7. However, this in effect alters the control flow of a program. We mention this for completeness here.

public void printResult(int
       result,boolean isPresent){
if(isPresent==false)
  System.out.println(”Element not
                      present”);
}
if (isPresent==false) {
  System.out.println(
  BinarySearch.hax0(
  BinarySearch.hax1(
  
  BinarySearch.hax7(
  ”Element not present”)))))))));
  
}

4.6 JFuscator

JFuscator is the only obfuscator that gives the developer the choice to obfuscate specific methods or the program as a whole.

4.7 Sandmark

Sandmark is an open source tool developed by University of Arizona that can obfuscate Java programs. Though the tool has been discontinued since JDK version 1.4 (in 2004), we ported the tool to support JDK 1.8. To obfuscate, Sandmark provides a list of techniques to choose from. From a total of 15 techniques, we identify 4 that are novel. Given below are the techniques previously seen in literature, but noted below due to minor implementation differences.

  1. Interleave Methods. (Called as Method Merger in Sandmark). In the tool, this is implemented as two different techniques - one that interleaves public static methods with the same signature and the other that interleaves all methods.

  2. Inline Methods. (Called as Inliner in Sandmark). In the tool only static methods are inlined.

  3. Simple Opaque Predicate. This is a composition of the techniques Extending Conditionals and Adding Redundant Operands.

  4. Static Method Bodies. This is same as Building API Buffer Methods, the difference being that it splits all non-static methods into a static helper method that will be invoked by a non-static stub.

  5. Branch Inversion. This technique is a specific implementation of the technique called Reordering Expressions, where the conditional expression of an if statement is negated.

  6. Buggy Code. Similar to Dead Code Insertion, except that a copy of a basic block is taken to introduce a bug which is then guarded by an opaquely false predicate

The following techniques are novel:

  1. Dynamic Inliner (Similar to Inline Methods) This technique uses instanceof checks to perform method inlining at runtime.

  2. Boolean Splitter. It modifies all uses and definitions of boolean variables and arrays. In the example below, the boolean isPresent is obfuscated.

    public void printResult(int result,
                boolean isPresent){
      if(isPresent==false)
      
    }
    public void printResult(int arg1,
                    boolean arg2){
      boolean bool1 = (int)
            (Math.random() * 2.0D);
      boolean arg1=false;
      if (!(arg1 ^ bool1))
      
    }
  3. Split Objects. An object of a class is split into two and are linked together by a field. In the example below, the FindElement object ob1 is split. In the obfuscated program, another object called next0 is created.

    class BinarySearch extends
                    FindElement{
     main(){
      FindElement ob1=new
                 FindElement();
      
    } }
    class FindElement{
      public FindElement0 next0;
      public FindElement(){
        if (this.next0 == null)
          this.next0 =
                new FindElement0();
    }
  4. Class Splitter. The body of a class is split in such a way that methods and fields of the class are moved into its superclass. In the example, the printResult method of BinarySearch class is moved to its superclass.

    class FindElement {
      
    }
    class BinarySearch
            extends FindElement{
      public void printResult(…){
      
    } }
    class FindElement {
      public void printResult(…){
      
    } }
    class BinarySearch
            extends FindElement{
    }\end{lstlisting}
    \end{minipage}
    \iffalse
    \item \textit{Split Classes.} There are two different implementations for this in Sandmark. It splits a class into two reorganizing its fields and methods. Below is an example of one approach. Another way to do this is to split every object into two and link it by an irrelevant field.
    \noindent\begin{minipage}{.45\textwidth}
    \begin{lstlisting}[frame=tlrb,label=code:example-21-1, numbers=none]{branchinversion}
    class FindElement{
      
    }
    class BinarySearch extends
                  FindElement{
     
     public static void main(String
                            args[]){
      
      FindElement ob1=new FindElement();
      boolean isPresent=false;
      
     }
    }
    class FindElement{
      public FindElement0 next0;
      public static boolean isPresent;
      FindElement(){
       isPresent=false;
       if (this.next0 == null){
        this.next0 = new FindElement0();
       }
       this.next0.isPresent=false;
      
      }
    class BinarySearch extends
                        FindElement{
     BinarySearch(){
       super();
     }
     
     public static void main(String
                           args[]){
     
     }
    }
    public class FindElement0{
     public static boolean isPresent;
     
    }

5 Discussion

From our study we identified 36 unique techniques from literature and 7 from tools. Three of the most popular and well-supported commercial obfuscators Zelix, Allatori and DashO implement only 13 of the 36 techniques. Thus there appears to be a gap between the theory and practice of CFO.

Component Obfuscation Technique Liter. Tools Both DR Level
1. Expression Extending Conditionals N Opaque Predicate
Adding Redundant Operands N Opaque Predicate
Reordering Expressions N Ordering
2. Statement Reordering Statements N Ordering
Remove Lib. and Program Idioms N Substitution
Instruction Substitution N Substitution
Replacing if(Non) Null Instructions With Try-Catch Block N Substitution
Converting Branches to jsr Instr. N Substitution
3. Basic Block Opaque Branch Insertion N Opaque Predicate
Dead Code Insertion N Opaque Predicate
Adding Dead Code Switch Stmts. N Opaque Predicate
Reordering Loops N Ordering
Reordering Code Blocks N Ordering
Finding and Reusing Duplicate Seq. N Ordering
Reorder Load Instrs. Above if Instr. N Ordering
Loop Fission N Loop Transf.
Loop Blocking N Loop Transf.
Loop Unrolling N Loop Transf.
Intersecting Loop Y Loop Transf.
Replace with Equivalent Codes N Substitution
Code Clone Type IV N Substitution
Basic Block Fission Y Code Insertion
Insert Dummy Loop N Code Insertion
Goto Instruction Augmentation Y Code Insertion
Irrelevant Code Insertion N Code Insertion
Control Flow Flattening N Code Insertion
Boolean Splitter N Code Insertion
Convert Reducible to Non-reducible Flowgraph Y Code Insertion
Partially Trapping Switch Stmts Y Code Insertion
Disobeying Constructor Conventions Y Code Insertion
Combining Try Blocks with Their Catch Blocks N Code Insertion
Indirecting if Instructions Y Code Insertion
4. Method Inline method N Method Transf.
Outline Method N Method Transf.
Clone Method N Method Transf.
Interleave Methods N Method Transf.
Dynamic Inliner N Method Transf.
Building API Buffer Methods N Method Transf.
Table Interpretation Y Method Transf.
Parallelizing the Code N Method Transf.
5. Class Split Objects N Class Transf.
Class Splitter N Class Transf.
Building Library Buffer Classes N Class Transf.

Table 3: Classification & Analysis Overview

Table 3 furnishes a classification of the techniques based on the component (or level) of a program at which an obfuscation transformation is applied. We identify 5 different components (or levels) of a program (column 1) - Expression (such as conditional expressions), Statement, Basic Block, Method, Class. We indicate whether the technique is proposed in the literature, tools or both (columns 3-5). In column 6, we note whether these techniques allow a program to be decompiled into a source code equivalent, thus facilitating reverse engineering. Finally, we also note the obfuscating paradigm used in column 7. The foundation for the paradigms is obtained from a prior study by Colberg et al. on taxonomy of obfuscating transformations [26].

We assess the potency of these techniques. Colberg et al. [26] defines potency as a measure of how difficult it is for a human reader to understand the obfuscated program. We augment that measure to include the ability of a decompiler to decompile an obfuscated bytecode into its equivalent source. This is because such techniques will inherently lack structure and thus will be unreadable. We determine that Table Interpretation (Virtualization) is the most potent as the knowledge of how the interpreter will execute the program is with the developer of the obfuscator itself. However, implementing such an obfuscator incurs a huge overhead. In all the other techniques, manual disassembly of the program can leak (at least partially) its proprietary knowledge.

12 techniques proposed in the literature is not available in any of the tools. 8 of those are transformations applied to a basic block. Basic block transformations are difficult to implement due to the challenge entailed in preserving inter-block dependencies. However, they are also the most potent. Any technique that generates a non-structured control flow at the bytecode level cannot be decompiled. We identify that the following 7 techniques causes a decompiler to fail - Goto Instruction Augmentation, Conversion of Reducible to Irreducible Flow Graphs, Intersecting Loops, Basic Block Fission, Partially Trapping Switch Statements, Disobeying Constructor Conventions and Indirecting if Instructions. Loop blocking, though it can be decompiled, is potent, as it makes use of nested loops. Nested loops make use of nested labels and gotos that deters the readability of a program for a human analyst.

All the other techniques that use ordering, substitution, class and method transformation paradigms are effective to create variants of a program, but does not deter the readability of a program. Literature cites method reordering as a CFO technique. However, method reordering only alters the class structurally and does not alter its control flow. Hence this cannot be classified as a CFO technique. We however list it here for completeness. Transformations on expressions and statements are the most common and easiest to implement. Opaque predicates can be potent if the predicate used to secure a conditional is potent.

None of the techniques can be deobfuscated without the original program. However, techniques that use opaque predicates such as dead code, opaque branch insertion, redundant operand etc. are common compiler optimizations that can be used to deobfuscate. A deobfuscator can also remove control flow edges that generate non-reducible flow graphs, to generate structured programs so as to enable a decompiler to generate source code.

Of the 6 tools we evaluated, Zelix, Allatori, DashO and JMOT can be used by software developers to control flow obfuscate their code. We determine that Zelix is the most potent as it is the only tool that implements language breaking transformations. Though Allatori and DashO are comparable in their implementations, programs generated by Allatori mask their obfuscation techniques well. DashO relies on try-catch blocks (TCB). They insert a varying number of TCBs depending on the level of obfuscation. This approach does not substantially mar the readability of a program. Although Sandmark provides the most number of techniques, it is not a practically viable tool for commercial purposes. This is because, the tool is no longer supported and not tested for versions of Java greater than 1.4. Each tool has a unique way of inserting dead code, which is useful to identify the obfuscator. Though JBCO has a good lineup of techniques, it has a broken code base at the time of writing this paper and hence not usable. JMOT is a purely command-line tool, does not support control flow obfuscation

The following techniques are common to most tools - Extending Conditionals, Adding Redundant Operands, Opaque Predicate Insertion, Dead Code Insertion, Irrelevant Code Insertion, Reordering Expressions, Reordering Statements, Replacing if(non)null instructions with try-catch blocks, and Goto Augmentation. Aside from the 8 techniques that are decompiler resistant, none of the other techniques provide enough security to prevent reverse engineering attacks. However, majority of the tools implement none of those techniques, allowing an adversary to defeat obfuscation.

6 Conclusion

In this research, we have surveyed existing literature and evaluated 13 tools to identify the different control flow obfuscation techniques and their implementations. We have identified 36 unique techniques from literature and 7 from tools. The three most popular commercial tools use only 13 of the 36 techniques, thus showing a lag between the theory and practice of control flow of obfuscation. We classify the 43 techniques into 5 types based on the component of a program on which obfuscation is applied. We also discuss the obfuscating paradigm used. Based on our analysis, we determine that transformations on a basic block are the most potent. We identify 8 techniques that renders a program unreadable both to a human analyst and a program decompiler. 7 of those techniques use language breaking transformations to make a program unstructured. All the other obfuscations are effective to create semantic variants and does not deter readability of a program. Most of the obfuscators do not implement potent techniques defeating the reliability of obfuscation. Each tool inserts their own proprietary dead code, which a deobfuscator can use as a feature to detect the tool. We identify the most commonly used techniques in tools and also flag those that are not implemented. When using a composition of obfuscations, there is a drastic increase in code size and complexity. Hence determining the optimal amount of obfuscation such that the run-time performance and code complexity is also optimal while preserving potency is an area that requires further work.

References

  • [1] Collberg, Christian, Clark Thomborson, and Douglas Low. ”Manufacturing cheap, resilient, and stealthy opaque constructs.” Proceedings of the 25th ACM SIGPLAN-SIGACT Symp. on Principles of Programming Languages, (1998)
  • [2] Jiang Ming, Dongpeng Xu, Li Wang, and Dinghao Wu. ”Loop: Logic-oriented opaque predicate detection in obfuscated binary code.” Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, (2015)
  • [3] Fabrizio Biondi, Sébastien Josse, Axel Legay, and Thomas Sirvent. ”Effectiveness of synthesis in concolic deobfuscation.” Computers and Security 70, (2017)
  • [4] Cappaert, Jan. ”Code obfuscation techniques for software protection.” Katholieke Universiteit Leuven, 1-112, (2012)
  • [5] Martinez, Sébastien. ”Source code obfuscation by mean of evolutionary algorithms.” Internship Report, University of Luxemborg August, (2012)
  • [6] Christodorescu, Mihai, and Somesh Jha. Static analysis of executables to detect malicious patterns. Wisconsin Univ-Madison, Dept. of Comp. Science, (2006)
  • [7] Balachandran, Vivek, and Sabu Emmanuel. ”Software code obfuscation by hiding control flow information in stack.” Information Forensics and Security (WIFS), 2011 IEEE International Workshop on. IEEE, (2011)
  • [8] Rolles, Rolf. ”Unpacking virtualization obfuscators.” 3rd USENIX Workshop on Offensive Technologies.(WOOT), (2009)
  • [9] Kinder, Johannes. ”Towards static analysis of virtualization-obfuscated binaries.” Reverse Engineering (WCRE), 19th Working Conference on. IEEE, (2012)
  • [10] Dalla Preda, Mila. ”Code obfuscation and malware detection by abstract interpretation.” PhD diss.), (2007)
  • [11] You, Ilsun, and Kangbin Yim. ”Malware obfuscation techniques: A brief survey.” Broadband, Wireless Computing, Communication and Applications (BWCCA), International Conference on. IEEE, (2010)
  • [12] Bertrand Anckaert, Matias Madou, Bjorn De Sutter, Bruno De Bus, Koen De Bosschere, and Bart Preneel. ”Program obfuscation: a quantitative approach.” Proceedings of the 2007 ACM workshop on Quality of protection. ACM, (2007)
  • [13] Matias Madou, Bertrand Anckaert, Bruno De Bus, Koen De Bosschere, Jan Cappaert, and Bart Preneel. ”On the effectiveness of source code transformations for binary obfuscation.” Proceedings of the International Conference on Software Engineering Research and Practice (SERP06). CSREA Press, (2006)
  • [14] László, Tımea, and Ákos Kiss. ”Obfuscating C++ programs via control flow flattening.” Annales Universitatis Scientarum Budapestinensis de Rolando Eötvös Nominatae, Sectio Computatorica 30, (2009)
  • [15] Buzatu, Florin. ”Methods for obfuscating Java programs.” Journal of Mobile, Embedded and Distributed Systems 4.1, 25-30, (2012)
  • [16] Sebastian Schrittwieser, Stefan Katzenbeisser, Johannes Kinder, Georg Merzdovnik, and Edgar Weippl. ”Protecting software through obfuscation: Can it keep pace with progress in code analysis?.” ACM Computing Surveys,(2016)
  • [17] Majumdar, Anirban, and Clark Thomborson. ”Manufacturing opaque predicates in distributed systems for code obfuscation.” Proceedings of the 29th Australasian Computer Science Conference-Volume 48, (2006)
  • [18] Low, Douglas. ”Protecting Java code via code obfuscation.” Crossroads 4.3 , 21-23, (1998)
  • [19] Low, Douglas. ”Java control flow obfuscation.” Diss. Univ. of Auckland, (1998)
  • [20] Borello, Jean-Marie, and Ludovic Mé. ”Code obfuscation techniques for metamorphic viruses.” Journal in Computer Virology 4.3, (2008)
  • [21] Drape, Stephen. ”Intellectual property protection using obfuscation.”, (2010)
  • [22] Udupa, Sharath K., Saumya K. Debray, and Matias Madou. ”Deobfuscation: Reverse engineering obfuscated code.” Reverse Engineering, 12th Working Conference on. IEEE, (2005)
  • [23] Coogan, Kevin, Gen Lu, and Saumya Debray. ”Deobfuscation of virtualization-obfuscated software: a semantics-based approach.” Proceedings of the 18th ACM conference on Computer and communications security, (2011)
  • [24] Balakrishnan, Arini, and Chloe Schulze. ”Code obfuscation literature survey.” CS701 Construction of compilers 19, (2005)
  • [25] Tsai, Hsin-Yi, Yu-Lun Huang, and David Wagner. ”A graph approach to quantitative analysis of control-flow obfuscating transformations.” IEEE Transactions on Information Forensics and Security 4.2, (2009)
  • [26] Collberg, Christian, Clark Thomborson, and Douglas Low. ”A taxonomy of obfuscating transformations.” Department of Computer Science, The University of Auckland, New Zealand(1997)
  • [27] Majumdar, Anirban, Clark Thomborson, and Stephen Drape. ”A survey of control-flow obfuscations.” International Conference on Information Systems Security. Springer, Berlin, Heidelberg, (2006)
  • [28] Collberg, Christian, Clark Thomborson, and Douglas Low. ”Manufacturing cheap, resilient, and stealthy opaque constructs.” Proceedings of the 25th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, (1998)
  • [29] Majumdar, Anirban, and Clark Thomborson. ”Manufacturing opaque predicates in distributed systems for code obfuscation.” Proceedings of the 29th Australasian Computer Science Conference-Volume 48, (2006)
  • [30] Myles, Ginger, and Christian Collberg. ”Software watermarking via opaque predicates: Implementation, analysis, and attacks.” Electronic Commerce Research 6.2, (2006)
  • [31] Drape, Stephen. Obfuscation of abstract data types. Diss. Univ. of Oxford, (2004)
  • [32] Hillert, Emilia. ”Obfuscate Java bytecode: an evaluation ofobfuscating transformations using JBCO.” (2014)
  • [33] Meyer, Daniel, and Dipl-Inform Sandro Schulze. ”Analyzing the Robustness of Clone Detection Tools Regarding Code Obfuscation.” Bachelor thesis, University of Magdeburg, (2012)
  • [34] Peng, Yong, Jie Liang, and Qi Li. ”A control flow obfuscation method for Android applications.” Cloud Computing and Intelligence Systems (CCIS), 4th International Conference on. IEEE, (2016)
  • [35] Balachandran, Vivek, Tan, J.J. Darell, and Thing, LL. Vrizlynn. ”Control flow obfuscation for android applications.” Computers and Security 61, (2016)
  • [36] Collberg, Christian S., and Clark Thomborson. ”Watermarking, tamper-proofing, and obfuscation-tools for software protection.” IEEE Transactions on Software Engineering, (2002)
  • [37] Balachandran, Vivek, and Sabu Emmanuel. ”Potent and stealthy control flow obfuscation by stack based self-modifying code.” IEEE Transactions on Information Forensics and Security 8.4, (2013)
  • [38] Chenxi Wang, Jack Davidson, Jonathan Hill, and John Knight. ”Protection of software-based survivability mechanisms.” Dependable Systems and Networks, 2001. DSN 2001. International Conference on. IEEE, (2001)
  • [39] Badger, Lee, D’Anna, Larry, Kilpatrick, Doug, Matt, Brian, Reisse, Andrew, and Vleck, Tom Van. ”Self-protecting mobile agents obfuscation techniques evaluation report.” Network Associates Laboratories, Report, (2002)
  • [40] Wong, Wing, and Stamp, Mark. ”Hunting for metamorphic engines.” Journal in Computer Virology 2.3, (2006)
  • [41] Haibo Chen, Liwei Yuan, Xi Wu, Binyu Zang, Bo Huang, and Pen-chung Yew. ”Control flow obfuscation with information flow tracking.” Proceedings of the 42nd Annual IEEE/ACM International Symp. on Microarchitecture, (2009)
  • [42] Banescu, Sebastian, Martín Ochoa, and Alexander Pretschner. ”A framework for measuring software obfuscation resilience against automated attacks.” Proceedings of the 1st International Workshop on Software Protection, (2015)
  • [43] Chan, Jien-Tsai, and Yang, Wuu. ”Advanced obfuscation techniques for Java bytecode.” Journal of Systems and Software, (2004)
  • [44] Y. L. Huang, F. S. Ho, H. Y. Tsai, and H. M. Kao. ”A control flow obfuscation method to discourage malicious tampering of software codes.” ACM Symp. on Information, Computer and Communications security, (2006)
  • [45] Batchelder, Michael, and Hendren, Laurie. ”Obfuscating java: The most pain for the least gain.” International Conference on Compiler Construction. Springer, Berlin, Heidelberg, (2007)
  • [46] Lim, Hyun-Il. ”Comparative Analysis of Code Obfuscation Approaches to Protect Software Products.” Intl. Journal of Computer Theory and Engineering, (2017)
  • [47] Balachandran, Vivek, Sufatrio, Darell, Tan, J.J., L.L.Thing, Vrizlynn. ”Control flow obfuscation for Android applications” Computers & Security 61, (2016)
  • [48] Xu, Hui, Zhou, Yangfan, Kang, Yu, Lyu, R. Michael. On Secure and Usable Program Obfuscation: A Survey, arxiv, (2017)
  • [49] Java Obfuscation Techniques, https://www.sable.mcgill.ca/JBCO/examples.html, Last visited Sept 2018.
  • [50] Hou, T.W., Chen, H.Y. and Tsai, M.H. ”Three control flow obfuscation methods for Java software” IEEE Proceedings-Software, Volume: 153, (2006)
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
297354
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description