blog.zynamics.com

If you have followed the development of BinNavi over the last two years you might know that we are making heavy use of something called REIL to provide features backed by advanced static code analysis. REIL is short for Reverse Engineering Intermediate Language and at its core it is a platform-independent pseudo-assembly language that can be used to emulate native assembly code.

A few years ago, Thomas spent some time thinking about making reverse engineering tools scale (check out the slides of his Black Hat Windows 2004 talk to learn more). Software today is larger and more complex than it was in the past and there are many more interesting platforms to consider as a security researcher (just think of Mac OS, iPhone, Android, Blackberry, Cisco routers, wireless devices, …). Platform-independent automation of common reverse engineering tasks seemed like the way to go. If you manage to develop powerful tools that help you find interesting parts of binary files without manual intervention you can fight growing complexity and reduce its associated costs. We created REIL as one of the key technologies for our path towards this goal.

We designed REIL with one thing in mind: Create a language that can model the effects of real assembly code but – unlike real assembly code – is very easy to analyze programmatically. To achieve this we carefully designed a minimal instruction set of just 17 different instructions and we made sure that the structure of the instruction operands is regular and comprehensible. We also made sure that all REIL instructions have exactly one obvious purpose because we wanted to avoid side effects like setting flags or implicitly accessing memory. Furthermore we have designed a very simple virtual machine (REIL VM) for REIL code to define the semantic behaviour of REIL code.

In the next few posts I will show you what the instruction set of REIL looks like, how the semantic model behind REIL interpretation works, how to use REIL in BinNavi and what the future of REIL will look like.

For now I am going to finish this post with a screenshot that shows some REIL code in BinNavi (click to enlarge). Notice the MIPS-like regularity and simplicity of the instructions. The original x86 instructions that were the source of the REIL code are shown as gray line comments.

A basic block of REIL code

If you already want to learn more about REIL you can check out the BinNavi manual. Here you can find everything that is necessary to understand the REIL language and its use in BinNavi.

Hi everyone,

this week we launched the first beta of BinNavi 3.0 to select customers. We are planning to have a beta phase of 8 weeks with the final release of BinNavi 3.0 coming May 1st 2010.

Existing customers who want to get their hands on the beta version please send an email to support@zynamics.com.

In BinNavi 3.0 we have added many valuable features that once again make it faster and easier for you to complete your reverse engineering jobs. You can find the complete list of new features in the manual on our website. It’s quite lengthy so I only want to talk about the Top 10 new features in this post.

Analyze code of MIPS-based devices

In previous versions of BinNavi it was possible to analyze x86 code, ARM code, and PowerPC code. In BinNavi 3.0 we have added support for MIPS code because MIPS was by far the platform we received most requests for.

Of course we have also added MIPS support to our static code analysis language REIL so your platform-independent analysis algorithms work on MIPS code too.

If you are a customer of our GDB Agent add-on, you can also debug MIPS-based Cisco routers like those of the 3600 family now.

Reverse Engineering MIPS code with BinNavi

Rename local and global variables to understand code

For the longest time BinNavi has had support for fancy stuff like abstract interpretation but not for basic stuff like variable renaming. In BinNavi 3.0 you can now rename local and global variables. This helps you understand code better than with previous versions of BinNavi.

Renaming variables with BinNavi

Find out where global variables are used

While improving support for variables we have also added a new view where you can see all global variables of a module and the functions that access them. This is very useful for tracking inter-function side-effects stored in global variables.

Cross-references to global variables

Quickly get back to your favourite projects, modules, and views

The ability to mark projects, modules, and views (like functions) as favorites is a really simple feature which turned out to be incredibly helpful in practice. With just two clicks you can now “star” items you consider important. Starred items have a small star next to their names and they always show up on top of tables. This makes it very simple to find functions again which you previously considered interesting.

Favorite functions are shown on top of the list

Use a faster disassembly data exporter to get started

Before BinNavi 3.0 we used a Python-based exporter to import disassembly from IDA Pro into our BinNavi MySQL databases. This exporter was really slow and required a lot of additional software packages to be installed. In BinNavi 3.0 we have switched to a C++-based exporter which is blazingly fast (we managed to export more than 80,000 functions per hour here) and does not require any additional installs. Once you realize that your exports now go more than twice as fast as they used to you will love this exporter.

Set conditional breakpoints to make debugging more efficient

Another really useful feature, conditional breakpoints were added to BinNavi 3.0 to allow you to enable or disable breakpoints depending on the current program state.

Breakpoint conditions can include checks for register, flags, and memory values as well as for thread IDs.

Configuring conditional breakpoints

Edit the target process memory to test small patches

Editing the memory of the target process was previously not possible in BinNavi. In BinNavi 3.0 you can edit the memory of whatever process you are debugging using either the GUI or the plugin API.

Editing target process memory

Isolate code quickly using the improved trace mode

I have already written two posts on this blog dedicated to this new feature (see here and here). In essence, we have found a great way to help you find relevant code while debugging.

Improved differential debugging

Quickly see where variables are used

You can now highlight instructions that use a given variable. This helps you quickly see where variables are used in a function.

Highlighting all instructions that access the _hwndNP variable

Quickly recognize special instructions

You can also highlight special instructions now. In this release “special” means either function calls, instructions that read from memory, or instructions that write to the memory. Especially the function call highlighting turned out to be really useful while reverse engineering code. We will probably extend this feature in the future.

Highlighting all function call instructions

So, to wrap this up. Once again, many new features were added and many older features were improved. It was really difficult to pick a Top 10 for this blog post and if you have looked through the list of changes in the manual you might consider other improvements to be more important than the ones presented here.

One of the big problems of static code analysis are function calls with non-static call targets. These function calls can call different target functions depending on the current program state. At first they call one function and in the next moment they might call a completely different function. Popular examples of such dynamic function calls are virtual functions (like in C++) or function pointers to callback functions.

Statically finding the set of potential call targets of a dynamic function call is very difficult. While this is an area of program analysis that has seen a lot of research in the last years, the problem is undecidable in general and can become really ugly really quickly. A simpler way to resolve the call targets of dynamic function calls is to execute the target program and log where dynamic function calls are going.

In BinNavi we have implemented a way to resolve dynamic function calls within modules as well as dynamic function calls that cross module boundaries. The general idea behind our code is this:

• Figure out where the dynamic function calls are located and put breakpoints on them
• Every time such a breakpoint is hit, execute a single step and find out where the call is going
• Keep going until enough data has been collected

You can see how it all works in the 5 minutes (13 MB) flash video you can watch when you click on the image below.

Resolved dynamic function calls to ws2_32.dll

Here is some more information about the process which I could not put into the video itself:

The whole Call Resolver functionality is not part of BinNavi itself but implemented as a plugin. This shows how easily users of BinNavi can extend the BinNavi GUI with new functionality and how powerful the debugging and graphing API of BinNavi is. In fact, you can download the code of the plugin here if you want to check it out yourself. This plugin was written in Java but it could have been written in Jython or JRuby as well.

Storing disassembly data in a MySQL database gives the plugin an enormous advantage: It is really, really simple to find the addresses of dynamic function calls. A single SQL query does the trick. In most other reverse engineering tools the plugin would need to go through all functions/basic blocks/instructions of the modules to find the dynamic function call instructions.

Setting breakpoints only on dynamic function call instructions brought a big speed improvement compared to just tracing the whole target program. As you can see in the video, the target program stays responsive enough to be used. This is very useful because it allows the user of the Call Resolver to control what functionality is executed and therefore what dynamic function calls are traced.

Of course the dynamic approach has downsides too. We have to have a way to execute the target program. If all we have is a non-executable memory dump of some suspicious file then we can not use dynamic function call analysis. Even if it is possible to execute the target program, it is easy to miss function calls that are never executed or function call targets that are never reached while the tracer is attached to the process. This is especially true if you have a heuristic like BinNavi has where you stop resolving function calls that “always” (really, more than 20 times) seem to go to the same target address.

So, what about you? I’d like to hear about your experiences with resolving dynamic function calls. Are you more of a fan of a static solution or a dynamic solution?

My earlier blog post about the improved Differential Debugging feature of BinNavi 3.0 generated a lot of interest so I have decided to write a follow-up post. Unlike last time I want you to be able to see what BinNavi can do and not just read about it. I have therefore created a short Flash video that shows how to find important code in disassembled files using the BinNavi debugger and its trace mode which is the core of Differential Debugging.

In the video I start with a disassembled IDB file of Pidgin’s liboscar.dll. The first step is to import the data from the IDB file into a BinNavi MySQL database. Afterwards I open the call graph of liboscar.dll and put the BinNavi Win32 debugger into function trace mode. In this mode trace events are generated every time a function of liboscar.dll is executed. This allows me to find the functions responsible for sending messages in just a few seconds.

You can find the video here. (5 MB Flash video with a resolution of 1280 x 1024)

Now this video shows only the most primitive use case of Differential Debugging. Nevertheless, this use case is already incredibly powerful. Finding out what code is responsible for what functionality of a program in just a few seconds is incredibly useful, no matter what you are trying to do.

However, there are situations where this simple use case is not enough. Maybe you are analyzing a daemon process where you can’t just click on some GUI element to isolate events. For these situations we provide more advanced features, like the ability to compare and connect recorded traces using set operations I mentioned in my earlier post.