> For the complete documentation index, see [llms.txt](https://docs.hex-rays.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.hex-rays.com/add-ons/malware/how-tos/extend-the-unpacker.md).

# Extend the Unpacker

You can extend the [automated unpacker](https://docs.hex-rays.com/add-ons/malware/concepts/automated-unpacking.md) so that it unpacks new malware families and samples. For example, if your team sees a threat actor that packs many malware samples with the same techniques, you can develop an unpacking heuristic that encodes your team's knowledge. Then you can distribute the heuristic in your team with the plugin infrastructure of IDA, such as [HCLI and plugin archives](https://docs.hex-rays.com/developer/publishing-plugins/concepts/plugin-packaging-and-format.md).

Thus your team can codify their knowledge about specific packers, and give it to new team members easily. Teams and the community can build heuristics for protectors like VMProtect and Themida, and for the packers of individual threat actors. You can keep a heuristic private, publish it on the plugin repository at [plugins.hex-rays.com](https://plugins.hex-rays.com), or send it to Hex-Rays for distribution with the Malware Analysis add-on.

This how-to shows what a heuristic contains, the strategies that you can use to unpack, and how to distribute the heuristic. As an example, it builds a small heuristic for one packer stub and runs it on a malware sample.

## Prerequisites

To write a heuristic, you need:

* IDA Pro 9.5 or later with the Malware Analysis add-on installed. See [Getting Started](https://docs.hex-rays.com/add-ons/malware/getting-started.md).
* Python experience, and some experience with IDAPython.
* Knowledge of the packer: how to recognize a program that it packed, and how to get the original code back.
* Packed samples to test the heuristic with.

## What a heuristic contains

To extend the unpacker, you give it Python code that can recognize a program that is packed with a given technique, and an algorithm that can unpack that program within the IDB. This code is a Python module with some well-known exports. You add the module to the unpacker framework, as described in [Distribute the heuristic](#distribute-the-heuristic).

When the unpacker framework unpacks a file, it calls your heuristic:

* To find out if the heuristic is able to unpack at all. This is the optional `available` function in the `HEURISTIC` dictionary. For example, if the heuristic needs an API key, `available` returns `False` when the key is not set, and the unpacker does not show the heuristic.
* To find out if the heuristic must have greater priority than the other heuristics. This is the optional static method `identify(info)`. If it returns `True`, the unpacker puts the heuristic first in its list and suggests it.
* To apply the unpacking algorithm at the correct time. This is the `run()` method. It returns `True` if it unpacked the program.

```python
import ida_malware.unpacker


class CMyPackerHeuristic(ida_malware.unpacker.CUnpackingHeuristic):
    def __init__(self, info):
        super().__init__(info)
        self.is_debugger = False

    @staticmethod
    def identify(info) -> bool:
        ...

    def run(self) -> bool:
        ...


def is_available() -> bool:
    return True


HEURISTIC = {
    "name": "my-packer",
    "description": "Decrypt the code of my packer in the IDB",
    "object": CMyPackerHeuristic,
    "available": is_available,
}
```

`is_debugger` tells the unpacker if your heuristic runs the sample in the configured debugger backend. It is `True` by default. Set it to `False` if your algorithm does not use the debugger.

## Write the unpacking algorithm

For the algorithm that does the unpacking, your options are wide open.

### Unpack statically

You can unpack statically: read the bytes of the program, apply a decompression or decryption algorithm to the bytes, and then patch them back into the IDB. For example, for UPX or a similar packer, you can implement the decompression algorithm again in Python, and then update the IDB.

Your heuristic has three responsibilities:

1. Recover the code. Rewrite the IDB contents to the unpacked logic. This means that you change the segments, including their permissions and sizes, and the contents of the segments: you patch in the bytes that you recovered.
2. Find the original entry point, when possible. If the program went through a transformation from an original program to a protected program, and you can find where the original entry point is, that is ideal. This is not always possible. To give a candidate to the unpacker, call `self.add_entry_point(ea)`.
3. Rebuild the import address table (IAT), so that calls to API functions resolve, and the disassembly listing and the decompiler show which APIs the code calls. You can rebuild the IAT in place, or you can allocate a new region, add data entries to it, apply names to these entries, and fix the cross-references in the unpacked code.

You can use the existing functionality of the unpacker framework for some of this work. When `run()` returns `True`, the framework re-analyzes the database to find the code, finds the API calls and the IAT, and looks for the original entry point with IDA's startup signatures. When your heuristic has rebuilt the complete program image in IDA, these steps can do the rest of the work.

This example decrypts a section that the packer encrypted with a one-byte XOR key, and then lets the framework do the other steps:

```python
import ida_bytes
import ida_idaapi
import ida_segment

import ida_malware.unpacker

SECTION = ".xpack"
KEY = 0x5A


class CXorSection(ida_malware.unpacker.CUnpackingHeuristic):
    def __init__(self, info):
        super().__init__(info)
        self.is_debugger = False

    @staticmethod
    def identify(info) -> bool:
        return ida_segment.get_segment_ea_by_name(SECTION) != ida_idaapi.BADADDR

    def run(self) -> bool:
        seg = ida_segment.segment_info_t()
        if not ida_segment.get_segment_info(seg, ida_segment.get_segment_ea_by_name(SECTION)):
            return False

        data = ida_bytes.get_bytes(seg.start_ea, seg.size())
        ida_bytes.patch_bytes(seg.start_ea, bytes(b ^ KEY for b in data))

        seg.set_perm(seg.get_perm() | ida_segment.SEGPERM_EXEC)
        return ida_segment.set_segment_info(seg)


HEURISTIC = {
    "name": "xor-section",
    "description": "Decrypt the .xpack section with XOR key 0x5A",
    "object": CXorSection,
}
```

### Use an emulator

You can also start an emulator, such as [Unicorn](https://www.unicorn-engine.org/) or [Speakeasy](https://github.com/mandiant/speakeasy), to find how the program behaves at runtime. Take memory snapshots at the correct times, and patch them back into the IDB.

Your heuristic is responsible for starting the emulator. Set `is_debugger` to `False`, so that the unpacker does not start its configured backend for you. Then start the emulator in `run()`. To start [Speakeasy](https://docs.hex-rays.com/add-ons/malware/concepts/emulators/speakeasy.md) through the [isolated system emulator](https://docs.hex-rays.com/add-ons/malware/concepts/isolated-system-emulation.md), call the IDC function `start_emulator_with_backend()`. For more about this function, see [Emulate Code from an idalib Script](https://docs.hex-rays.com/add-ons/malware/how-tos/emulate-code-from-idalib.md).

After you start an emulator backend, use the [IDAPython debugger API](https://python.docs.hex-rays.com/ida_dbg/index.html) and the breakpoint methods of `CUnpackingHeuristic` to control it. This is because IDA connects to the emulator as a debugger backend. The sample runs only in the emulator.

Because `is_debugger` is `False`, the unpacker does not copy the process memory into the IDB, and it does not stop the emulator. Your heuristic does these steps before `run()` returns:

```python
import ida_dbg
import ida_segment
import idc

import ida_malware.unpacker

TIMEOUT = 300


class CMyEmulatorHeuristic(ida_malware.unpacker.CUnpackingHeuristic):
    def __init__(self, info):
        super().__init__(info)
        self.is_debugger = False

    def run(self) -> bool:
        if idc.eval_idc('start_emulator_with_backend("speakeasy")') != 1:
            return False

        try:
            ida_dbg.wait_for_next_event(ida_dbg.WFNE_SUSP, TIMEOUT)
            ...  # set breakpoints and run to the unpacked code

            ida_dbg.refresh_debugger_memory()
            return ida_segment.take_memory_snapshot(ida_segment.SNAP_LOAD_SEG)
        finally:
            # exit_process() only asks the emulator to stop the process
            ida_dbg.exit_process()
            ida_dbg.wait_for_next_event(ida_dbg.WFNE_ANY, 10)
```

### Use a debugger

You can also connect to a debugger to run the program in a virtual machine or on real hardware.

{% hint style="warning" %}
A debugger runs the malicious code on a real system. This can damage or destroy the system. Make this very clear to the users of your heuristic.
{% endhint %}

If `is_debugger` is `True`, which is the default, the unpacker starts the backend that the user configured in `default_debugger`, and suspends the process at the entry point before it calls `run()`. Your heuristic then controls the process with the IDAPython debugger API and the breakpoint methods of `CUnpackingHeuristic`. When `run()` returns `True`, the unpacker copies the process memory into the IDB and, if `default_stop` is `true`, stops the process. For an example, see [Example: stop at the tail jump](#example-stop-at-the-tail-jump).

The unpacker has these mechanisms to show the user what happens:

* The user selects the backend. The backend runs the sample locally or remotely, and it is the responsibility of the user to configure this correctly with the `default_debugger` and `default_remote` settings. The default backend is Bochs, which is an emulator.
* Before the unpacker uses a real debugger, it asks the user to confirm. In idalib and batch mode, it does not use a real debugger. See [Emulators and real debuggers](https://docs.hex-rays.com/add-ons/malware/concepts/automated-unpacking.md#emulators-and-real-debuggers).
* The `description` in the `HEURISTIC` dictionary is in the list of heuristics. Use it to tell the user that the heuristic runs the sample.

### Use a third-party service

You can also connect to a third-party service that dumps the memory of the sample or rebuilds it, such as VirusTotal, CAPE Sandbox, VMRay, or UnpacMe. These services can give unpacked memory regions, memory dumps, or unpacked PE files that you then write into the IDB. The built-in `vt-unpack` and `vmray-unpack` heuristics work like this.

These services can require API keys. To get the API key, use [ida-settings](https://docs.hex-rays.com/developer/publishing-plugins/concepts/plugin-packaging-and-format.md#shared-settings) in your module. The `available` function hides the heuristic when the key is not set. The unpacker calls it each time an unpack operation starts, so a key that the user sets after the database opens has an effect:

```python
import ida_settings

import ida_malware.unpacker


def get_api_key() -> str:
    return ida_settings.get_current_plugin_setting("unpacme_api_key")


class CUnpacMeHeuristic(ida_malware.unpacker.CUnpackingHeuristic):
    def __init__(self, info):
        super().__init__(info)
        self.is_debugger = False

    def run(self) -> bool:
        key = get_api_key()
        ...


HEURISTIC = {
    "name": "unpacme",
    "description": "Apply the unpacked files from UnpacMe",
    "object": CUnpacMeHeuristic,
    "available": lambda: get_api_key() != "",
}
```

Declare the `unpacme_api_key` setting in the `ida-plugin.json` file of your plugin, so that HCLI can ask the user for its value.

## Example: stop at the tail jump

In this example, you write the heuristic `popad-tail-jump` for packers with a simple stub. The stub starts with `PUSHAD`, which saves all the registers. Then it unpacks the code in memory, restores the registers with `POPAD`, and jumps to the original entry point with `JMP`. When the stub gets to this `JMP`, the unpacked code is in memory. Thus the heuristic runs the sample in the configured backend and stops the process at the `JMP`. Old versions of UPX, such as 1.24, use this stub.

To follow the steps, use a 32-bit worm that is packed with UPX 1.24, with the SHA-256 hash `3511ac944b77073be43040750a5134946d6e5c357e5acf9f39dca808b34d8d33`. You can download it from [VirusTotal](https://www.virustotal.com/gui/file/3511ac944b77073be43040750a5134946d6e5c357e5acf9f39dca808b34d8d33). Keep `default_debugger` set to `bochs`, so the sample runs in the emulator.

{% hint style="warning" %}
This sample is live malware. Open it only on a dedicated analysis machine or a VM that you can restore.
{% endhint %}

1. Find your IDA user directory. This is the first directory in `$IDAUSR`. If `IDAUSR` is not set, it is `%APPDATA%\Hex-Rays\IDA Pro` on Windows and `~/.idapro` on Linux and macOS. See [IDAUSR](https://docs.hex-rays.com/core/disassembler/concepts/environment-variables.md#idausr).
2. In the IDA user directory, create the directory `malware-analysis/unpacker/heuristics`.
3. In this directory, save the following code as `popad_tail_jump.py`:

   ```python
   import ida_bytes
   import ida_dbg
   import ida_funcs
   import ida_idaapi
   import ida_idd

   import ida_malware.unpacker

   TAIL_JUMP = "61 E9"  # POPAD; JMP rel32
   TIMEOUT = 300


   def find_tail_jump(ea: int) -> int:
       func = ida_funcs.func_entry_info_t()
       if not ida_funcs.get_func_entry_info(func, ea):
           return ida_idaapi.BADADDR
       return ida_bytes.find_bytes(TAIL_JUMP, func.start_ea, range_end=func.end_ea)


   class CPopadTailJump(ida_malware.unpacker.CUnpackingHeuristic):
       @staticmethod
       def identify(info) -> bool:
           return find_tail_jump(info.entries[0]) != ida_idaapi.BADADDR

       def run(self) -> bool:
           popad_ea = find_tail_jump(self.info.entries[0])
           if popad_ea == ida_idaapi.BADADDR:
               return False

           try:
               self.add_bpt(popad_ea + 1)
               ida_dbg.continue_process()
               if self.wait_for_event(TIMEOUT) != ida_idd.BREAKPOINT:
                   return False
               self.add_target_oep(step=True)
           finally:
               self.remove_bpts()

           return ida_dbg.dbg_can_query()


   HEURISTIC = {
       "name": "popad-tail-jump",
       "description": "Run the sample until the JMP after POPAD in the entry point function",
       "object": CPopadTailJump,
   }
   ```

   The heuristic finds the `POPAD` and `JMP` bytes in the entry point function. In the sample, the `POPAD` is at `0x41DFEE`, and the `JMP` goes to `0x4010FC` in the `UPX0` segment.

   The heuristic keeps the default `is_debugger = True`, so the unpacker starts the configured backend, Bochs, and suspends the process at the entry point. `run()` sets a breakpoint on the `JMP` and resumes the process. When the stub gets to the `JMP`, the unpacked code is in memory. `add_target_oep(step=True)` steps over the `JMP` and records the destination as the original entry point.
4. Open the packed sample. When IDA asks if you want to unpack the sample, select **Yes**. You can also select **Extensions → Malware Analysis → Unpacking → Unpack now\...** or press <kbd>Shift</kbd> + <kbd>P</kbd>.
5. In the list of heuristics, select `popad-tail-jump`. Because `identify()` returns `True` for this sample, the heuristic is at the top of the list.
6. When the unpacker finishes, it names the original entry point `OriginalEntryPoint` and jumps to it, or to `main()` if it finds it. For the sample, the original entry point is `0x4010FC`.

{% hint style="warning" %}
In idalib and in batch mode, the unpacker unpacks packed samples at database open with no dialogs. It runs the first heuristic in its list. If your `identify()` returns `True`, your heuristic runs automatically when a script opens a packed sample.
{% endhint %}

## Distribute the heuristic

There are two ways to add a heuristic to the unpacker framework.

### Put a file in the heuristics directory

Put a Python file in the directory `malware-analysis/unpacker/heuristics` in your IDA user directory, as in the example. The unpacker reads this directory one time for each IDA session, at the first unpack operation. After you change a file, restart IDA. To test a change without a restart, run a script that calls `ida_malware.unpacker.register_heuristic()`, as described in [Package the heuristic as a plugin](#package-the-heuristic-as-a-plugin).

If a file has a syntax error, raises an exception on import, or has no valid `HEURISTIC` dictionary, the unpacker does not load it. The **Output** window shows the warning `unpacker: Skipped user heuristic` with the file path and the traceback. The unpacker loads all the other heuristics.

This method has disadvantages. Each user must copy the file into the directory by hand, so the installation is not automated, and it does not scale well when you share the code with other people. Also, the heuristic is limited to one file. You cannot declare dependencies on third-party libraries or keep library code next to it, because the unpacker does not add the directory to the Python search path.

### Package the heuristic as a plugin

To distribute a heuristic, package it as a Python-based IDA Pro plugin. You can then distribute it through a public or private plugin repository, such as plugins.hex-rays.com, or share the plugin archive in your team. Users install the plugin with HCLI:

```bash
hcli plugin install ida-themida-unpacker
```

Then the heuristic is available to the unpacker framework. To do this, in the startup code of your plugin, give the module that contains the heuristic to `ida_malware.unpacker.register_heuristic()`:

```python
import ida_idaapi

import ida_malware.unpacker

import themida_heuristic


class ThemidaUnpackerPlugin(ida_idaapi.plugin_t):
    flags = ida_idaapi.PLUGIN_HIDE | ida_idaapi.PLUGIN_FIX
    comment = "Themida unpacking heuristic"
    help = ""
    wanted_name = "Themida unpacker"
    wanted_hotkey = ""

    def init(self):
        ida_malware.unpacker.register_heuristic(themida_heuristic)
        return ida_idaapi.PLUGIN_KEEP

    def run(self, arg):
        pass

    def term(self):
        pass


def PLUGIN_ENTRY():
    return ThemidaUnpackerPlugin()
```

The unpacker gets the list of heuristics each time an unpack operation starts. Thus your plugin does not need special flags, and IDA can load it before or after the unpacker.

A plugin can also declare dependencies in its `ida-plugin.json` file. List the Python packages that implement third-party logic, such as decompression or decryption, in `pythonDependencies`, and HCLI installs them with the plugin:

```json
{
  "plugin": {
    "name": "ida-themida-unpacker",
    "entryPoint": "themida_unpacker_entry.py",
    "pythonDependencies": ["pycryptodome"]
  }
}
```

For the other fields of `ida-plugin.json`, see [Packaging Format](https://docs.hex-rays.com/developer/publishing-plugins/concepts/plugin-packaging-and-format.md). To publish the plugin, see [Plugin Publishing](https://docs.hex-rays.com/developer/publishing-plugins/how-tos/plugin-publishing.md).

This gives users an easy way to get your heuristic and add it to the unpacker framework, and you can give your expertise to users who see protected malware.

`register_heuristic()` raises `ValueError` if the module has no `HEURISTIC` dictionary with the keys `name`, `description`, and `object`, or if `name` is not a string. If you call it again with a module that has the same heuristic name, the new module replaces the old one. `ida_malware.unpacker.get_registered_heuristics()` returns the registered modules. You can also call `register_heuristic()` from a script that you run with **File → Script file...**, for example to test a heuristic.

Do not call `register_heuristic()` in a file in the heuristics directory. The unpacker adds these files itself.

## Reference

The `HEURISTIC` dictionary has these keys:

| Key           | Required | Description                                                                                                                                                                                        |
| ------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`        | Yes      | A unique string. The unpacker shows it in the list of heuristics. If a built-in heuristic has the same name, the unpacker skips your heuristic and shows a warning.                                |
| `description` | Yes      | The text that the unpacker shows next to the name.                                                                                                                                                 |
| `object`      | Yes      | Your subclass of `CUnpackingHeuristic`.                                                                                                                                                            |
| `available`   | No       | A function with no arguments. If it returns `False`, the unpacker does not show the heuristic, for example when an API key is not set. The unpacker calls it each time an unpack operation starts. |

`CUnpackingHeuristic` gives your heuristic these members:

| Member                                  | Description                                                                                                                                                                                                              |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `run()`                                 | Implement this method. Return `True` when the unpacked code is in the IDB, or, if `is_debugger` is `True`, in the process memory with the process suspended.                                                             |
| `identify(info)`                        | Optional static method. Return `True` if the heuristic knows the packer of the current database. If it raises an exception, the **Output** window shows a warning, and the unpacker continues as if it returned `False`. |
| `info`                                  | The unpacker. `info.entries` is the list of entry points. `info.for_each_segment(loader=True)` yields the segments of the input file.                                                                                    |
| `is_debugger`                           | `True` by default: the unpacker starts the configured debugger backend before it calls `run()`. Set it to `False` in `__init__()` if the heuristic does not use the debugger.                                            |
| `undefine_data`                         | `True` by default: the unpacker undefines the segments before it re-analyzes them. Set it to `False` if your heuristic creates the code and data itself.                                                                 |
| `add_entry_point(ea)`                   | Records `ea` as a candidate for the original entry point.                                                                                                                                                                |
| `add_target_oep(step)`                  | Records the current instruction pointer as a candidate for the original entry point. If `step` is `True`, it first executes one instruction.                                                                             |
| `add_bpt(ea, size=0, type=BPT_DEFAULT)` | Sets a breakpoint and keeps it in the list of breakpoints of the heuristic. `type` is a `BPT_*` value from `ida_idd`.                                                                                                    |
| `add_smart_bpt(ea)`                     | Sets a hardware breakpoint if one is free, else a software breakpoint. Use it for code that the stub can change.                                                                                                         |
| `remove_bpts()`                         | Deletes all the breakpoints in the list of breakpoints of the heuristic.                                                                                                                                                 |
| `verify_bpts()`                         | Returns `True` if all the breakpoints in the list are active.                                                                                                                                                            |
| `break_at_exit_process()`               | Sets breakpoints on the functions that end the process, such as `ExitProcess`.                                                                                                                                           |
| `wait_for_event(timeout)`               | Waits up to `timeout` seconds for a debugger event and returns it, for example `ida_idd.BREAKPOINT` or `idc.DBG_TIMEOUT`.                                                                                                |
| `step_into()`                           | Executes one instruction.                                                                                                                                                                                                |
| `suspend()`                             | Suspends the process. Returns `True` if IDA can read the process memory.                                                                                                                                                 |

To see more examples, read the code of the built-in heuristics in the `heuristics` directory of the add-on.

## What's next?

If you have a question about heuristics, or you want Hex-Rays to distribute your heuristic with the add-on, email <support@hex-rays.com>.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.hex-rays.com/add-ons/malware/how-tos/extend-the-unpacker.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
