Debugging the XNU Kernel with IDA Pro

Purpose

IDA 7.3 introduces the Remote XNU Debugger. It is designed to communicate with the GDB stub included with popular virtualization tools, namely VMWare Fusion (for OSX) and Corellium (for iOS). The debugger allows you to observe the Darwin kernel as it is running, while at the same time utilising the full power of IDA's analysis capabilities. It works equally well on Mac, Windows, and Linux.

This writeup is intended to quickly get you familiar with debugger, as well as offer some hints to make the experience as smooth as possible.

Debugging OSX with VMWare

To get started with debugging OSX, we will perform a simple experiment. This is the same experiment outlined in this great writeup by GeoSn0w, but we will be performing the equivalent in IDA - which we hope you'll find is much simpler.

Begin with the following setup:

create an OSX virtual machine with VMWare Fusion. in this example the virtual machine is OSX 10.13.6, but the experiment should work with any recent OSX version.
open Terminal in the VM and enable some basic XNU debugging options:
```
$ sudo nvram boot-args="slide=0 debug=0x100 keepsyms=1"
```
shut down the VM and add the following line to the .vmx file:
```
debugStub.listen.guest64 = "TRUE"
```
power on the virtual machine, open Terminal, and run this command:
```
 $ uname -v
 Darwin Kernel Version 17.7.0 ... root:xnu-4570.71.17~1/RELEASE_X86_64	
```
Let's use IDA to modify this version string.

Launch IDA, and when prompted with the window IDA: Quick start, choose Go to start with an empty database. Then go to menu Debugger>Attach>Remote XNU Debugger and set the following options:

Click OK, then select <attach to the process started on target>, and wait for IDA to attach. This step might take a few seconds (later we'll discuss how to speed things up). Once attached, the target is usually suspended in machine_idle:

IDA should have printed the message FFFFFF8000200000: process kernel has started, meaning it successfully detected the kernel image in memory. Now let's find the version string. Conveniently, the string appears in the kernel's symbol table, so we can simply use shortcut G and enter the name _version to jump right to it:

Use IDAPython to overwrite the bytes at this address:

idaapi.dbg_write_memory(0xFFFFFF8000AF6A00, "IDAPRO")

Resume the process and allow the VM to run freely. Go back to Terminal in the VM and run the same command as before:

$ uname -v
IDAPRO Kernel Version 17.7.0 ... root:xnu-4570.71.17~1/RELEASE_X86_64

The output should look almost the same, except Darwin has been replaced with IDAPRO. So, we have modified kernel memory without breaking anything! You can continue to explore memory, set breakpoints, pause and resume the OS as you desire.

Using the KDK

If you have installed a Kernel Development Kit from Apple, you can set KDK_PATH in dbg_xnu.cfg to enable DWARF debugging:

KDK_PATH = "/Library/Developer/KDKs/KDK_10.13.6_17G4015.kdk";

Even if there is no KDK available for your OSX version, you can still utilise the KDK_PATH option in IDA to speed up debugging. For example, in the experiment above we could have done the following:

make your own KDK directory:
```
$ mkdir ~/MyKDK
```

copy the kernelcache from your VM:

$ scp user@vm:/System/Library/PrelinkedKernels/prelinkedkernel ~/MyKDK

decompress the kernelcache:

$ kextcache -c ~/MyKDK/prelinkedkernel -uncompressed

set KDK_PATH in dbg_xnu.cfg:
```
KDK_PATH = "~/MyKDK";
```

Now whenever IDA needs to extract information from the kernel or kexts, it will parse the kernelcache file on disk instead of parsing the images in memory. This should be noticeably faster.

Debugging a Development Kernel

Our next goal is to use the KDK to create a rich database that can be used to debug XNU in greater detail. In this example we will debug the development kernel included in the Apple KDK. Let's open this file in IDA:

$ export KDK=/Library/Developer/KDKs/KDK_10.13.6_17G4015.kdk
$ export KERNELS=$KDK/System/Library/Kernels
$ ida64 -okernel.i64 $KERNELS/kernel.development

Wait for IDA to load the DWARF info and complete the autoanalysis. This may take a few minutes, but we only need to do it once.

While we wait, we can prepare the virtual machine to use the development kernel instead of the release kernel that is shipped with OSX (Note: System Integrity Protection must now be disabled in the VM). Open Terminal in the VM and run the following commands:

copy the development kernel from the KDK:

$ sudo scp user@host:"\$KERNELS/kernel.development" /System/Library/Kernels/

reconstruct the kernelcache:
```
$ sudo kextcache -i /
```
reboot:
```
$ sudo shutdown -r now
```
after rebooting, check that the development kernel was properly installed:
```
 $ uname -v
 ... root:xnu-4570.71.17~1/DEVELOPMENT_X86_64
```

The VM is now ready for debugging.

IDA Configuration

Return to IDA and use Debugger>Select debugger to select Remote XNU Debugger. Then open Debugger>Process options and set the following fields:

Now go to Debugger>Debugger options>Set specific options and make sure the KDK path field is set:

You can ignore the other options for now, and press OK.

Assembly-Level Debugging + DWARF

IDA supports source-level debugging for the XNU Kernel. However for demonstration purposes we will focus on assembly-level debugging, while taking advantage of source-level DWARF information like local variables. This is a bit more stable, and is still quite useful.

Before attaching the debugger, open Options>Source paths... and un-check the checkbox:

Then click Apply. This will prevent IDA from complaining when it can't find a source file.

Finally, select Debugger>Attach to process>attach to the process started on target. After attaching, jump to function dofileread, and use F2 to set a breakpoint. Resume the debugger and and wait for the breakpoint to be hit (typically it will be hit right away, if not try simply running a terminal command in the guest). Once XNU hits our breakpoint, open Debugger>Debugger windows>Locals:

We can now perform detailed instruction-level debugging with the assistance of DWARF. You can continue to single step, set breakpoints, and inspect or modify local variables just like any other IDA debugger.

KEXT Debugging

IDA also supports debugging kext binaries. To demonstrate this, we will debug IONetworkingFamily, a submodule of IOKit that is typically shipped with the KDK. Begin by opening the binary in IDA:

$ export KEXTS=$KDK/System/Library/Extensions
$ ida64 -onet.i64 $KEXTS/IONetworkingFamily.kext/Contents/MacOS/IONetworkingFamily

Select Remote XNU Debugger from the debugger menu. Then in Debugger>Process options, set:

Note that we provide the bundle ID of the kext (com.apple.iokit.IONetworkingFamily) as the Input file field. This allows the debugger to easily identify the target kext at runtime.

Also note that loading all kexts in kernel memory can be a slow operation, which is why it is disabled by default. Open Debugger>Debugger options>Set specific options and ensure the KDK path field is set, then set the KEXT Debugging option to KDK only:

This tells the debugger to only load kexts that are present in the KDK. Since the KDK binaries are on the local filesystem, IDA can parse the kexts in a negligible amount of time - which is ideal since we're really only interested in IONetworkingFamily.

Now power on your VM and allow it to boot up. Once it is running idle, attach the debugger. Immediately IDA should detect the kernel and all relevant kexts in memory, including IONetworkingFamily:

Double-click to bring up the debug names for this module, and search for IONetworkInterface::if_ioctl:

Now set a breakpoint at this function and resume the OS. Typically the breakpoint will be hit right away, but if it isn't try performing an action that requires a network interface (for instance, performing a google search). Once execution breaks in the kext we can use the database to debug it in detail:

Debugging a Prelinked Kernelcache

For simplicity, all of the examples up until now have dealt with a subset of the kernel, but it is also possible to load a complete prelinked kernelcache in IDA and debug it. Naturally, we have some suggestions for this.

Extending the KDK

If you're interested in debugging the entire prelinked kernel, the biggest concern is speed. IDA must create a detailed and accurate depiction of kernel memory, which could contain hundreds of kext modules. If we're not careful, this can be slow.

Fortunately there is an easy solution. Try the following:

create a writable copy of Apple's KDK:

$ cp -r /Library/Developer/KDKs/KDK_10.13.6_17G4015.kdk ~/MyKDK

copy the kernelcache from your VM to the new KDK:

$ scp user@vm:/System/Library/PrelinkedKernels/prelinkedkernel ~/MyKDK

decompress the kernelcache:

$ kextcache -c ~/MyKDK/prelinkedkernel -uncompressed

Now IDA can use both the KDK and the kernelcache to extract debugging information for almost any kext at runtime. This should be fast.

Loading the Kernelcache

When loading a kernelcache, IDA now offers more load options:

In this example we want to load everything, so choose the kernel + all kexts option and wait for IDA to load all the subfiles and finish the autoanalysis. This will take a while but there's no way around it, it's a lot of code.

IMPORTANT NOTE: Try to avoid saving the IDA database file in the KDK directory. It is important to keep irrelevant files out of the KDK since they might slow down IDA's KDK parsing algorithm.

Now we might want to improve the static analysis by loading DWARF info from the KDK. In IDA 7.3 the dwarf plugin supports batch-loading all DWARF info from a KDK into a kernelcache database. Currently this feature must be invoked manually, so we have provided this script to make it easier.

Copy kdk_utils.py to the plugins directory of your IDA installation. This plugin will create a new menu Edit>Other>KDK utils, with two new menu actions:

Load KDK: This action will automatically detect all matching DWARF files in a given KDK, then apply the DWARF info to the subfiles in the database (including the kernel itself).
Load DWARF for a prelinked KEXT: This action is useful if you have DWARF info for a prelinked kext that is not included in Apple's KDK. For a given DWARF file, the action will find a matching kext in the database and apply the DWARF info to this subfile.

Try opening Edit>Other>KDK utils>Load KDK and provide the KDK path:

Wait for IDA to scan the KDK for matching DWARF files and load them. This operation can also take a while, but it's worth it for all the extra structures, prototypes, and names that are added to the database. In the end we have a very detailed database that we are ready to use for debugging.

Now open Debugger>Process options and set the following options:

Then open Debugger>Debugger options>Set specific options and set the following fields:

Note that we set the KEXT Debugging option to all. This tells the debugger to detect every kext that has been loaded into memory and add it to the Modules list, including any non-prelinked kexts (there are likely only a handful of them, so it doesn't hurt).

Finally, power on the VM and attach to it with Debugger>Attach to process>attach to the process started on target. IDA should be able to quickly generate modules for the kernel and all loaded kexts:

You are now free to explore the entire running kernel! Try performing any of the previous demos in this writeup. They should work about the same, but now they are all possible with one single database.

Kernel ASLR + Rebasing

It is worth noting that rebasing has been heavily improved in IDA 7.3. Even large databases like the one we just created can now be rebased in just a few seconds. Previous IDA versions would take quite a bit longer. Thus, IDA should be able to quickly handle kernel ASLR, even when working with prelinked kernelcaches.

Debugging the OSX Kernel Entry Point

In this example we demonstrate how to gain control of the OS as early as possible. This task requires very specific steps, and we document them here. Before we begin, we must make an important note about a limitation in VMWare's GDB stub.

Physical Memory

Currently VMWare's 64-bit GDB stub does not allow us to debug the kernel entry point in physical memory. According to VMWare's support team, the correct approach is to use the 32-bit stub to debug the first few instructions of the kernel, then switch to a separate debugger connected to the 64-bit stub once the kernel switches to 64-bit addressing.

Since IDA's XNU debugger does not support 32-bit debugging, this approach is not really feasible (and it's not very practical anyway).