IDAClang

Overview

The IDAClang plugin integrates the clang compiler frontend into IDA itself. This allows IDA to parse type information from complex C/C++/Objective-C source code and import it directly into an IDA database.

Libclang

IDAClang utilizes a specialized build of libclang - the opensource C API for the clang compiler. This custom library is also shipped with IDA alongside the plugin itself, so you do not need to worry about it. The plugin will find and load libclang automatically.

Our build of libclang is from Clang v13.0, so it can handle any Objective-C syntax and anything from C++20 and earlier.

Motivation

IDAClang was introduced as a more robust alternative to IDA’s built-in source code parser. The built-in parser can handle simple C source code, but naturally it struggles to handle complex C++ and Objective-C syntax. IDAClang solves this problem by outsourcing all the heavy lifting to a third-party library that can handle the ugly parsing operations. The plugin needs only to parse the abstract syntax tree generated by clang.

As a result, IDAClang should be much more flexible. You can even feed it complete .cpp source files. The plugin will extract whatever useful type information it can find, and ignore the rest.

VTables

One big advantage of using libclang is that we can take advantage of clang’s internal C++ VTable management. For example, when IDAClang parses a C++ class that looks like this:

class C
{
  virtual void func(void);
};

The following types will be generated in the database:

struct __cppobj C
{
  C_vtbl *__vftable /*VFT*/;
};

struct /*VFT*/ C_vtbl
{
  void (__cdecl *func)(C *this);
};

To create the C_vtbl type, IDAClang traverses clang’s internal VTableLayout data structure. This data structure is the same mechanism that the clang compiler uses during the actual code generation. Thus, we can be very confident that IDAClang is producing correct vtable types - even in much more complex situations. After all, clang knows what it’s doing in this regard.

Moreover, when using IDAClang to generate a type library (see Building Type Libraries with IDAClang below), the plugin will take advantage of clang’s name mangling to populate the symbol table:

SYMBOLS
FFFFFFFF 00000000 void __cdecl _ZN1C4funcEv(C *this);
00000018 00000000 C_vtbl_layout _ZTV1C;

TYPES
struct C_vtbl_layout
{
  __int64 thisOffset;
  void *rtti;
  void (__cdecl *func)(C *this);
};

Here IDAClang created symbols for the C::func member function, as well as the mangled VTable symbol for the C class.

Templates

Another notable advantage of using libclang is it allows us to gracefully handle C++ templates.

For example, consider the following template declarations:

template <typename T, typename V> struct S
{
  T x;
  V y;
};

typedef S<int, void *> instance_t;

When clang parses the instance_t declaration, internally it will generate a structure that represents the specialized template S<int, void *>. The IDAClang plugin will then use this internal representation to generate a valid type for S<int, void *> in IDA’s type system:

struct S<int, void *>
{
  int x;
  void *y;
};

typedef S<int, void *> instance_t;

The type with name S<int, void *> represents the fully resolved structure, with all template arguments replaced. This all happens automatically, and it is especially useful in more complex situations - such as template classes containing virtual methods that depend on template parameters, resulting in specialized VTables.

The IDAClang UI

Enabling the IDAClang Parser

To provide support for third-party parsers, IDA now has a new Source parser field in the Options>Compiler dialog:

To enable the IDAClang parser, select the clang parser from the dropdown menu:

As a quick sanity check, try saving the following declaration in a file called test.h:

typedef int myint_t;

Parse the file using menu File>Load file>Parse C header file. IDA should print this to the output window:

/private/tmp/test.h: successfully compiled

The type should now be present in the Local Types view:

Configuring IDAClang

Of course, IDAClang is capable of parsing source code that is much more complex. Often times this requires more detailed configuration of the parser invocations.

To support this, the Compiler>Options dialog provides the Arguments field:

In this field you can provide any argument you would typically provide to the clang compiler when invoking it from the command line. For example:

One of the more important clang arguments is the -target option, which specifies the target architecture and platform. This allows clang to properly configure itself to parse macOS/Windows/Linux system headers. Clang calls this the target "triple" because it is often given in the form of:

-target <arch>-<vendor>-<platform>

Some examples:

-target arm64-apple-darwin
-target x86_64-pc-win32
-target i386-pc-linux

The various combinations of supported targets is documented in more detail here.

Note that in the simple test.h example above, we did not specify a target platform. In this case clang will assume that the target platform is the same as the host machine IDA is currently running on. You can print the exact target used by clang by opening Options>Compiler>Parser specific options and enable the following option:

Now when we use IDAClang to parse the test.h file, it will print a message:

IDACLANG: triple: x86_64-apple-macosx10.15.0

Which would be the typical output when IDA is running on macOS. On Windows the default will look something like:

IDACLANG: triple: x86_64-pc-windows-msvc19.29.30137

And on Linux:

IDACLANG: triple: x86_64-unknown-linux-gnu

Such is the default behavior within libclang, but clang supports a wide variety of platforms and architectures. You can almost always specify a target that will match the input binary in the current database.

STL Example

Now let’s try invoking IDAClang on some more real-world source code.

In this example, assume we are analyzing an x64 binary that makes heavy use of the C++ Standard Template Library. Then assume that at some point we want to create a structure that looks like this:

#include <string>
#include <vector>
#include <map>
#include <set>

struct stl_example_t
{
  std::string str;
  std::vector<int> vec;
  std::map<std::string, int> map;
  std::set<char> set;
};

This is the contents of stl/stl_example.h from examples.zip. IDA’s default parser cannot handle such complex C++ syntax, so IDAClang is our only hope of importing this type. The precise configuration of IDAClang will vary between platforms, so we’ll demonstrate them all separately.

To parse stl_example.h on macOS, we’ll have to point IDAClang to the macOS SDK as well as the STL system headers:

-target x86_64-apple-darwin
-x c++
-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
-I/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1
-I/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/11.0.3/include

Copy the text above into the Options>Compiler>Arguments field.

Note that we point IDAClang to the macOS SDK with the -isysroot option and use the -I option to allow IDAClang to find the proper system headers in the Xcode toolchain. Be wary of the last option (ending with usr/lib/clang/11.0.3/include). This path contains the clang version number, so it might be different on your machine. Also make special note of the -x c++ option. This is used to inform libclang that the input source will not be plain C, which is the default syntax for .h files in libclang.

Now we can use File>Load file>Parse C header file to parse stl_example.h. This will generate a useful type for stl_example_t in our database:

On Windows the configuration is a bit different. If you’re using Visual Studio, libclang is normally able to detect common header paths automatically.

Thus you will likely only need to specify the following arguments in Options>Compiler>Arguments:

-target x86_64-pc-win32 -x c++

Ideally this will be enough to parse stl_example.h and generate some useful type info:

If for whatever reason the heuristics within libclang fail to find the headers on your system, it is very easy to specify the header paths manually. Simply open a Visual Studio x64 Command Prompt and run the following command:

echo %INCLUDE%

This will print a semicolon-separated list of the header paths used on your system:

This list can be copied directly into the Options>Compiler>Include directories field in IDA. IDAClang will automatically process this list and pass the header paths to clang upon invocation of the parser. This is likely enough to handle most Windows-based source code.

On Linux you can determine the header paths used your system by running the following command:

cpp -v

This will print something like:

#include <...> search starts here:
 /usr/lib/gcc/x86_64-linux-gnu/6/include
 /usr/local/include
 /usr/lib/gcc/x86_64-linux-gnu/6/include-fixed
 /usr/include/x86_64-linux-gnu
 /usr/include

You can then use these arguments in the Options>Compiler>Arguments field in IDA:

-target x86_64-pc-linux-gnu
-x c++
-I/usr/lib/gcc/x86_64-linux-gnu/6/include
-I/usr/local/include
-I/usr/lib/gcc/x86_64-linux-gnu/6/include-fixed
-I/usr/include/x86_64-linux-gnu
-I/usr/include

Then use File>Load file>Parse C header file to parse stl_example.h.

Invoking IDAClang from IDAPython

Like any good IDA feature, IDAClang can also be invoked from an IDAPython script.

IDA 7.7 introduced the ida_srclang module to provide simple support for invoking third-party parsers from IDAPython. Use the following IDAPython commands for an overview of this new module:

import ida_srclang
? ida_srclang
? ida_srclang.parse_decls_with_parser
? ida_srclang.set_parser_argv

The function ida_srclang.parse_decls_with_parser can notably be used to parse source code snippets:

Python>? ida_srclang.parse_decls_with_parser
Help on function parse_decls_with_parser in module ida_srclang:
parse_decls_with_parser(*args) -> 'int'
    Parse type declarations using the parser with the specified name
    @param parser_name: (C++: const char *) name of the target parser
    @param til: (C++: til_t *) type library to store the types
    @param input: (C++: const char *) input source. can be a file path or decl string
    @param is_path: (C++: bool) true if input parameter is a path to a source file, false if the
                    input is an in-memory source snippet
    @retval -1: no parser was found with the given name
    @retval else: the number of errors encountered in the input source

If the is_path argument is False, this function will assume the input argument is a string that represents a source code snippet. Otherwise it will be considered a path to a source file on disk. Also note the til parameter, which will often times be None. This ensures the parsed types are imported directly into the current database.

Examples

IMPORANT NOTE: when libclang parses in-memory strings, it makes no assumptions about the expected syntax. Thus, you must specify the -x option to tell clang which syntax to expect before invoking the parser. Here are the the known syntax directives:

-x c
-x c++
-x objective-c
-x objective-c++

For example, this is how you would use ida_srclang to parse a simple C source string with IDAClang:

import ida_srclang
# tell clang the expected syntax
ida_srclang.set_parser_argv("clang", "-x c")
# parse a type string
ida_srclang.parse_decls_with_parser("clang", None, "typedef int myint_t;", False)

STL Example Revisited

We can also handle the same STL example discussed previously, but this time parse stl_example_t as a source snippet:

import ida_srclang

clang_argv = [
  "-target x86_64-apple-darwin",
  "-x c++",
  "-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk",
  "-I/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1",
  "-I/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/11.0.3/include",
  ]
ida_srclang.set_parser_argv("clang", " ".join(clang_argv))

decl = """
#include <string>
#include <vector>
#include <map>
#include <set>

struct stl_example_t
{
  std::string str;
  std::vector<int> vec;
  std::map<std::string, int> map;
  std::set<char> set;
};
"""
ida_srclang.parse_decls_with_parser("clang", None, decl, False)

This should produce an identical result as before when we used File>Load file>Parse C header file for stl_example.h.

Boost Example

In this example we will show how IDAClang can be used in batch mode to improve the analysis of a binary compiled from Boost headers. The experiment will be performed on Debian Linux with gcc 6.3.0.

Consider the following source files from the boost/ directory in examples.zip:

  • chat_server.cpp

  • chat_message.hpp

These sources were taken directly from the Boost 1.77 examples, and we’ll use them to compile a test binary. Begin by downloading the Boost 1.77.0 headers, then compile the chat_server application:

g++ -I boost_1_77_0 -std=c++11 -o chat_server.elf chat_server.cpp -lpthread

Since Boost is a template library, it will generate a bloated binary that contains thousands of instantiated template functions. Thus, IDA’s initial analysis of chat_server.elf will likely not be very pretty. How can IDAClang help us with this? Consider boost/chat_server.py from examples.zip.

import sys
import ida_pro
import ida_auto
import ida_srclang

clang_argv = {
  "-target x86_64-pc-linux",
  "-x c++",
  "-std=c++11",
  "-I./boost_1_77_0",
  # NOTE: include paths were copied from the output of `cpp -v`. they might differ on your machine.
  "-I/usr/lib/gcc/x86_64-linux-gnu/6/include",
  "-I/usr/local/include",
  "-I/usr/lib/gcc/x86_64-linux-gnu/6/include-fixed",
  "-I/usr/include/x86_64-linux-gnu",
  "-I/usr/include",
}

# invoke the clang parser
ida_srclang.set_parser_argv("clang", " ".join(clang_argv))
ida_srclang.parse_decls_with_parser("clang", None, "./chat_server.cpp", True)

# analyze the input file
ida_auto.auto_mark_range(0, BADADDR, AU_FINAL)
ida_auto.auto_wait()

# save and exit
ida_pro.qexit(0)

This script will configure IDAClang to parse the chat_server.cpp source file and extract any type information it finds, then analyze the input with the imported type info, and saves the resulting database in chat_server.i64. You can run the script like this:

idat64 -c -A -Schat_server.py -Oidaclang:t -ochat_server.i64 -Lchat_server.log chat_server.elf

You may have noticed this option:

-Oidaclang:t

This option is passed to the IDAClang plugin and it enables CLANG_APPLY_TINFO (see idaclang.cfg for more info).

Now let’s open the resulting database chat_server.i64 in IDA, and try decompiling some functions. Immediately we see that the analysis does benefit from the imported type info. For example chat_session::do_write seems somewhat intelligible after some minor simplifications:

Since IDAClang parsed the chat_session class, we now have a correct prototype for chat_session:do_write, as well as a valid chat_session structure. Note that references to chat_session.write_msgs_ (std::deque<chat_message>) and chat_session.socket (boost::asio::ip::tcp::socket) were correctly resolved.

Granted, this is not the most realistic example. It’s not often we have access to the full source code of the target binary, but hopefully this shows that whenever any relevant source code is available, IDAClang can take full advantage.

Building Type Libraries with IDAClang

The IDAClang plugin is useful for enriching your database with complex type information, but often times the imported types are relevant to more than just one database. In this section we discuss how you can use IDAClang to generate rich, generic type libraries for IDA Pro.

Hex-Rays also provides a command-line version of IDAClang, specifically designed for building custom Type Information Libraries (TILs) that can be loaded into any IDA database.

After downloading the idaclang binary, copy it to the idabin/ directory of your IDA installation (next to the libclang dll).

For an overview of idaclang’s functionality, run:

idaclang -h

For a quick demonstration, save the following source in a file named test.h:

class C
{
  virtual void func(void);
};

You can compile this header into a type library by invoking idaclang the same way you would typically invoke the clang compiler from the command line:

idaclang -x c++ -target x86_64-pc-linux test.h

This will generate a file called test.til that contains all types that were parsed in test.h. Try dumping the TIL with the tilib utility.

tilib -l /tmp/test.til

TYPE INFORMATION LIBRARY CONTENTS
Description:
Flags      : 0107 compressed macro_table_present extended_sizeof_info sizeof_long_double
Base tils  :
Compiler   : GNU C++
sizeof(near*) = 8 sizeof(far*) = 8 near code, near data, cdecl
default_align = 0 sizeof(bool) = 1 sizeof(long)  = 8 sizeof(llong) = 8
sizeof(enum) = 4 sizeof(int) = 4 sizeof(short) = 2
sizeof(long double) = 16
SYMBOLS
FFFFFFFF 00000000 void __cdecl ZN1C4funcEv(C *__hidden this);
00000018 00000000 C_vtbl_layout ZTV1C;
TYPES
00000008 struct __cppobj C {C_vtbl *__vftable /*VFT*/;};
00000008 struct /*VFT*/ C_vtbl {void (__cdecl *func)(C *__hidden this);};
00000018 struct C_vtbl_layout {__int64 thisOffset;void *rtti;void (__cdecl *func)(C *__hidden this);};
MACROS
Total 2 symbols, 3 types, 0 macros

The tool also provides extra arguments to configure the til generation. They are given the --idaclang- prefix so they can be easily separated from the clang arguments. For example:

idaclang --idaclang-tilname /tmp/test2.til -x c++ -target x86_64-pc-linux test.h

This will create the library at /tmp/test2.til, instead of the default location.

Now let’s try building some type libraries from real-world code. The examples in this section will demonstrate the power of IDAClang by creating TILs from many different opensource C++ projects. They cover a large variety of platforms, architectures, and codebases, so it is best to unify the build system using makefiles.

At the top level of examples.zip there should be a makefile named idaclang.mak:

IDACLANG_ARGS += --idaclang-log-all
IDACLANG_ARGS += --idaclang-tilname $(TIL_NAME)
IDACLANG_ARGS += --idaclang-tildesc $(TIL_DESC)

CLANG_ARGV += -ferror-limit=50

all: $(TIL_NAME)
.PHONY: all $(TIL_NAME) clean
$(TIL_NAME): $(TIL_NAME).til

$(TIL_NAME).til: $(TIL_NAME).mak $(INPUT_FILE)
    idaclang $(IDACLANG_ARGS) $(CLANG_ARGV) $(INPUT_FILE) > $(TIL_NAME).log
    tilib64 -ls $(TIL_NAME).til > $(TIL_NAME).til.txt

clean:
    rm -rf *.til *.txt *.log

This makefile defines a simple rule for building a TIL using the idaclang command-line utility. It will be used extensively in the following examples.

IDASDK

Hex-Rays publishes an SDK for developing custom IDA plugins, which is comprised mostly of C++ header files. Thus, it is a perfect use case for IDAClang. In this example we will build a type library for IDA itself, using IDA SDK 7.7.

After downloading idasdk77.zip, unzip it into the idasdk subdirectory of examples.zip.

To build this TIL we only need to create a single header file that includes all headers from the IDA SDK, and then parse this file with idaclang. See examples/idasdk/idasdk.h, which contains include directives for all files in idasdk77/include (they happen to be in alphabetical order, but the order shouldn’t matter much):

#include <auto.hpp>
#include <bitrange.hpp>
#include <bytes.hpp>
// ... etc
#include <typeinf.hpp>
#include <ua.hpp>
#include <xref.hpp>

The IDAClang configuration required to parse idasdk.h is highly platform-dependent, so we provide separate makefiles for each of IDA’s supported platforms.

To demonstrate how we might build idasdk.h on MacOSX, see examples/idasdk/idasdk_mac_x64.mak:

TIL_NAME = idasdk_mac_x64
TIL_DESC = "IDA SDK headers for MacOSX"
INPUT_FILE = idasdk.h
SDK = /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk
TOOLCHAIN = /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain
CLANG_ARGV = -target x86_64-apple-darwin                 \
             -x objective-c++                            \
             -isysroot $(SDK)                            \
             -I$(TOOLCHAIN)/usr/include/c++/v1           \
             -I$(TOOLCHAIN)/usr/lib/clang/11.0.3/include \
             -I./idasdk77/include/                       \
             -D__MAC__                                   \
             -D__EA64__                                  \
             -Wno-nullability-completeness

include ../idaclang.mak

You can build the TIL with:

make -f idasdk_mac_x64.mak

This will generate a type library named idasdk_mac_x64.til, along with a dump of the til contents in idasdk_mac_x64.til.txt. In the text dump we might notice some familiar types:

00000010 struct __cppobj range_t
{
  ea_t start_ea;
  ea_t end_ea;
};
//  0. 0000 0008 effalign(8) fda=0 bits=0000 range_t.start_ea ea_t;
//  1. 0008 0008 effalign(8) fda=0 bits=0000 range_t.end_ea ea_t;
//          0010 effalign(8) sda=0 bits=0080 range_t struct packalign=0

00000050 struct __cppobj memory_info_t : range_t
{
  qstring name;
  qstring sclass;
  ea_t sbase;
  uchar bitness;
  uchar perm;
};
//  0. 0000 0010 effalign(8) fda=0 bits=0020 memory_info_t.range_t range_t;
//  1. 0010 0018 effalign(8) fda=0 bits=0000 memory_info_t.name qstring;
//  2. 0028 0018 effalign(8) fda=0 bits=0000 memory_info_t.sclass qstring;
//  3. 0040 0008 effalign(8) fda=0 bits=0000 memory_info_t.sbase ea_t;
//  4. 0048 0001 effalign(1) fda=0 bits=0000 memory_info_t.bitness uchar;
//  5. 0049 0001 effalign(1) fda=0 bits=0000 memory_info_t.perm uchar;
//          004A unpadded_size
//          0050 effalign(8) sda=0 bits=0080 memory_info_t struct packalign=0

It’s worth building a separate til for both x64 and arm64 macOS. IDA’s source code is not very architecture dependent, but many system headers might be. So it’s best to be as precise as possible.

To build this TIL on macOS12 for Apple Silicon, the approach is very similar:

TIL_NAME = idasdk_mac_arm64
TIL_DESC = "IDA SDK headers for arm64 macOS 12"
INPUT_FILE = idasdk.h
SDK = /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk
TOOLCHAIN = /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain
CLANG_ARGV = -target arm64-apple-darwin                  \
             -x objective-c++                            \
             -isysroot $(SDK)                            \
             -I$(TOOLCHAIN)/usr/lib/clang/13.0.0/include \
             -I./idasdk77/include/                       \
             -D__MAC__                                   \
             -D__EA64__                                  \
             -D__ARM__                                   \
             -Wno-nullability-completeness

include ../idaclang.mak

Note that we did not provide the path to the C++ STL headers like we did in idasdk_mac_x64.mak. On macOS12 the C++ headers are shipped within MacOSX12.0.sdk, so there is no need to explicitly tell idaclang where to find them.

To parse idasdk.h on Windows, use examples/idasdk/idasdk_win.mak:

TIL_NAME = idasdk_win
TIL_DESC = "IDA SDK headers for x64 Windows"
INPUT_FILE = idasdk.h
CLANG_ARGV = -target x86_64-pc-win32       \
             -x c++                        \
             -I./idasdk77/include          \
             -D__NT__                      \
             -D__EA64__                    \
             -Wno-nullability-completeness

include ../idaclang.mak

Normally we do not need to specify any include paths, since idaclang can find the Visual Studio headers automatically. If it can’t, you can always explicitly provide include paths with the -I option.

Building idasdk.h on Linux is also fairly straightforward. See idasdk_linux.mak:

TIL_NAME = idasdk_linux
TIL_DESC = "IDA SDK headers for x64 linux"
INPUT_FILE = idasdk.h
GCC_VERSION = $(shell expr `gcc -dumpversion | cut -f1 -d.`)
CLANG_ARGV = -target x86_64-pc-linux                                      \
             -x c++                                                       \
             -I/usr/lib/gcc/x86_64-linux-gnu/$(GCC_VERSION)/include       \
             -I/usr/local/include                                         \
             -I/usr/lib/gcc/x86_64-linux-gnu/$(GCC_VERSION)/include-fixed \
             -I/usr/include/x86_64-linux-gnu                              \
             -I/usr/include                                               \
             -I./idasdk77/include/                                        \
             -D__LINUX__                                                  \
             -D__EA64__                                                   \
             -Wno-nullability-completeness

include ../idaclang.mak

You can also include the decompiler types from the hexrays SDK in the type library for idasdk77. Simply copy hexrays.hpp from hexrays_sdk/ in your IDA installation to idasdk77/include/, then add this line to idasdk.h:

#include <hexrays.hpp>

Then rebuild the TIL. It will likely yield some useful decompiler types:

00000050 struct __cppobj minsn_t
{
  mcode_t opcode;
  int iprops;
</