> For the complete documentation index, see [llms.txt](https://docs.hex-rays.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.hex-rays.com/add-ons/assist/how-tos/solve-crackme-with-assist.md).

# Solve a crackme with Assist

This walkthrough follows a real Assist session from the first prompt to a verified answer. The target is `kestrel-vm`, a small stripped Linux x86-64 ELF that asks for a "maintenance password". The twist is that the check is not written in C: `main` hands the password to an interpreter that runs a bytecode program embedded in the binary.

By the end of the session, Assist has reversed the VM's instruction set, recorded it as an enum in the database, disassembled the embedded program, recovered the password, and confirmed it by emulating the interpreter — and you have seen what each of those steps looks like in the panel. The same approach works on any crackme; only the prompt changes.

{% hint style="info" %}
New to Assist? See the [Overview](/add-ons/assist/overview.md) for a short recap, or [Getting Started](/add-ons/assist/getting-started.md) for installation and first-run setup.
{% endhint %}

## Prerequisites

* **IDA 9.4 SP1 or later** (IDA Home, IDA Pro) with a Hex-Rays decompiler license for the target architecture (this walkthrough leans on pseudocode; see [Requirements & Support → Decompiler requirements](/add-ons/assist/reference/requirements.md#decompiler-requirements)).
* Assist installed and an agent connected, see [Getting Started](/add-ons/assist/getting-started.md).
* A permission preset. This walkthrough assumes the recommended **Ask before changes** preset, so every database edit, script, and memory write shows up as a [capability prompt card](/add-ons/assist/reference/glossary.md#capability-prompt-card) before it runs. With **Work freely**, the same steps run without prompts for edits and scripts. See [Permissions & Privacy](/add-ons/assist/concepts/permissions-and-privacy.md#capabilities-and-defaults).
* The Assist hands-on kit, downloaded and unzipped. This walkthrough uses the `12-vm-crackme` folder.

{% file src="/files/naAH0JMRqFms5ipW1Zb2" %}

## 1. Open the binary and the panel

1. Open `kestrel-vm.i64` from the `12-vm-crackme` folder of the kit. The database is already analyzed, so there is nothing to wait for.
2. The Assist panel opens by default. If you closed it, press <kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>K</kbd> (or choose **View → Assist**) to reopen it.

With a built-in **Assist agent loop** backend, Assist offers a one-time **Triage this binary** [suggestion pill](/add-ons/assist/reference/glossary.md#suggestion-pill) above the composer. Triage is a good first step on an unknown sample, but for a crackme you usually already know what you want, so this walkthrough goes straight to a prompt.

## 2. Ask for the whole job, not just a hint

The prompt that drove the session:

> kestrel-vm checks its maintenance password with a custom bytecode VM. Reverse the VM's instruction set, define the opcodes as an enum, write a disassembler for the embedded bytecode program, recover the password from it, and confirm it by emulating the VM.

A prompt like "Tell me what this crackme does" also works and is a fine way to get oriented. The difference is that a prompt which names the deliverables — an enum, a disassembler, a recovered password, an emulation run — tells Assist what counts as done, so it keeps going until every item has evidence behind it instead of stopping at a summary.

{% hint style="info" %}
If you already know where to look, type `@` to mention a function or `#` to mention an address. The picker searches the database as you type, and the mention pins the agent's attention to that location.
{% endhint %}

## 3. Watch Assist work

Every tool call appears in the transcript as a card showing the tool, its target, and a summary of the result; expand a card to see the full output. Between calls, Assist posts one-line progress notes ("The VM is at sub\_11D0. Now let me pull the embedded bytecode…"). Here is how the session unfolded.

### Survey the binary

Assist started with three reads:

* **List functions** — 20 functions, 13 of them named from the ELF's symbols, 7 default `sub_*` names. One unnamed function, `sub_11D0`, is 690 bytes long and dwarfs everything else.
* **Search strings** — `usage: %s <maintenance password>`, `Maintenance mode unlocked. Welcome, engineer.`, `Wrong password.`, and `VM error.`
* **Decompile `main`** — which ties the two together:

```c
if ( a1 == 2 )
{
  v3 = sub_11D0((__int64)&unk_2080, 0x22Fu, a2[1]);
  if ( v3 == 1 )
  {
    puts("Maintenance mode unlocked. Welcome, engineer.");
    return 0;
  }
  v4 = "VM error.";
  if ( v3 == 0 )
    v4 = "Wrong password.";
  puts(v4);
  return 1;
}
```

So `sub_11D0` is the interpreter, the 559-byte (`0x22F`) blob at `0x2080` is the program it runs, and the return value is `1` for accept, `0` for reject, and anything else for a VM fault.

### Read the interpreter

Decompiling `sub_11D0` showed a classic dispatch loop: a 32-byte register file zeroed with `memset` (eight 32-bit slots), a program counter, a step budget of `0x186A0` (100,000), a comparison flag, and fifteen handlers arranged as an if/switch tree over the opcode byte:

```c
v8 = *(_BYTE *)(a1 + v7);          // opcode
v9 = v7 + 1;
if ( v8 == 0x76 )                  // AND  r[dst] &= imm32
{
  v20 = *(_BYTE *)(a1 + v7 + 1);
  v21 = *(_DWORD *)(a1 + v7 + 2);
  v7 += 6LL;
  *((_DWORD *)v26 + (v20 & 7)) &= v21;
  goto LABEL_15;
}
...
switch ( v8 )
{
  case 5u:                         // JNE  taken when the flag is clear
    if ( v6 )
      v7 += 3LL;
    else
      v7 += 3LL + (__int16)((*(unsigned __int8 *)(a1 + v7 + 2) << 8) | *(unsigned __int8 *)(a1 + v9));
    break;
  case 0x17u:                      // ADD  r[dst] += r[src]
    ...
```

Each handler reveals its operand layout: register selectors are single bytes masked with `& 7`, immediates are 32-bit little-endian, and branch displacements are signed 16-bit values relative to the end of the instruction. Those widths are what a disassembler needs.

### Dump and disassemble the bytecode

Assist read the 559 bytes at `0x2080`, then ran an IDAPython script (**Run Python** permission) that walks the stream with the operand widths it had just derived.

The first attempt was wrong: it reported 559 "instructions", one per byte. Assist noticed that the count equalled the byte count, found the bug in its own decoder, and re-ran. The clean pass produced 126 instructions ending exactly at `0x22AF`, with no branch target landing inside an instruction. The script also wrote the listing to a file so you can keep it.

```
2080: strlen r0        2082: cmp r0,12      2088: jne 22AE
208B: mov r7,90
2091: ldb r1,in[0]     2094: mov r2,90      209A: xor r1,r2   209D: rol r1,1
20A0: mul r1,157       20A6: mov r3,0       20AC: add r1,r3   20AF: and r1,255
20B5: cmp r1,218       20BB: jne 22AE
...  (repeated for i = 1..11)
22AD: accept           22AE: reject
```

The program is a length guard (`strlen == 12`) followed by twelve identical blocks. Each block loads one input byte, XORs it with a per-position key, rotates it, multiplies by 157, adds the position index, masks to a byte, and compares the result with a per-position target. Any mismatch jumps to `reject`; falling through all twelve reaches `accept`.

### Invert the check

With the constants for all twelve positions extracted from the listing, Assist ran a short standalone Python script (**Run Python** permission). Per byte the constraint is

```
rol8(p[i] ^ K[i], s[i]) * 157 + i  ≡  T[i]   (mod 256)
```

157 is odd, so it is invertible modulo 256 (its inverse is 181), and each byte solves directly:

```
p[i] = ror8(((T[i] - i) * 181) mod 256, s[i]) ^ K[i]
```

The result was `Kestrel#VM26`. Re-encrypting those twelve bytes reproduced all twelve targets exactly.

### Confirm by emulation

A Python reimplementation can share a misreading with the analysis that produced it, so Assist confirmed against the real code by emulating the interpreter at `0x11D0` with the actual arguments (program at `0x2080`, length `0x22F`, and a buffer holding the candidate password). See [Emulation](/add-ons/assist/concepts/emulation.md) for what the embedded emulator can do.

| Input          | Result | Guest instructions | Return    |
| -------------- | ------ | ------------------ | --------- |
| `Kestrel#VM26` | accept | 2,852              | `rax = 1` |
| `Kestrel#VM27` | reject | 2,857              | `rax = 0` |

The negative control matters: it shows the emulated interpreter actually distinguishes inputs rather than returning `1` for anything.

{% hint style="info" %}
Not every tool call succeeds on the first try, and the transcript does not hide that. In this session, several emulation runs returned after only 11 guest instructions because the arguments were sent empty; Assist said so in its progress notes ("I kept sending an empty arguments array") and corrected itself. If a turn goes in circles, press <kbd>Esc</kbd> to interrupt and redirect it.
{% endhint %}

### Record the findings in the database

Only now did Assist touch the database. Under **Ask before changes**, this is where capability prompt cards appear: **Modify this IDA database** for the edits, and **Share confirmed findings** for the memory write. Each card shows the exact change before you choose **Deny**, **Allow once**, or **Allow for this session**.

* **Create enum** `KestrelVmOpcode` with fifteen members, `VM_OP_JNE` (`0x05`) through `VM_OP_ROL` (`0xE3`).
* **Set prototype and rename** `sub_11D0` → `kestrel_vm_run`, typed as `uint32_t __fastcall kestrel_vm_run(const uint8_t *prog, uint64_t prog_len, const char *input)`.
* **Rename** `unk_2080` → `kestrel_vm_program`, with a repeatable comment describing the 559-byte, 126-instruction program.
* **Remember** a memory fact summarizing the opcode set, operand grammar, return codes, and the two addresses, so the next session on this binary starts with that knowledge. See [Memory](/add-ons/assist/concepts/memory.md).

All of the database edits are grouped into IDA undo steps, so <kbd>Ctrl</kbd>+<kbd>Z</kbd> reverts them if you disagree.

## 4. Read the answer

The final reply leads with the result and how it was confirmed, then shows its work: an **assumption register** listing each assumption (operand widths, branch semantics, rotate behavior, no hidden input opcode) alongside the test that would have falsified it, the instruction-set table, the disassembler summary with the per-position constants, the recovery math with a worked example, the emulation table, a short list of out-of-scope observations ranked by severity, and the list of database changes.

<details>

<summary>Assist's final reply (abridged)</summary>

*Tables are trimmed, and the JNE encoding below is the corrected one from the follow-up in step 5; the first pass described an extra selector byte.*

**Result.** Password: `Kestrel#VM26` (12 characters). Confirmed by emulating `kestrel_vm_run` at `0x11d0` with `prog=0x2080`, `prog_len=0x22F`, input buffer `"Kestrel#VM26"` → `rax = 1` (ACCEPT, 2,852 instructions). Negative control with last byte `'7'` → `rax = 0` (REJECT).

**VM instruction set** (decompiled at `0x11d0`, recorded as enum `KestrelVmOpcode`). Register file: 8 × 32-bit slots on the stack, zeroed via `memset(..., 0x20)`. Step budget `0x186A0` = 100,000. Every register selector is masked `& 7`; all immediates are 32-bit little-endian.

| Op     | Mnemonic | Encoding       | Semantics                                             |
| ------ | -------- | -------------- | ----------------------------------------------------- |
| `0x05` | JNE      | `05 disp16`    | if flag clear: `pc = pc + 3 + disp16`, else `pc += 3` |
| `0x17` | ADD      | `17 dst src`   | `r[dst] += r[src]`                                    |
| `0x2A` | SUB      | `2A dst src`   | `r[dst] -= r[src]`                                    |
| `0x3C` | ERR      | `3C`           | return `-1`                                           |
| `0x4D` | MUL      | `4D dst imm32` | `r[dst] *= imm32`                                     |
| `0x5E` | LDB      | `5E dst idx8`  | `r[dst] = (uint8_t)input[idx8]`                       |
| `0x6F` | REJECT   | `6F`           | return `0`                                            |
| `0x76` | AND      | `76 dst imm32` | `r[dst] &= imm32`                                     |
| `0x88` | XOR      | `88 dst src`   | `r[dst] ^= r[src]`                                    |
| `0x91` | MOV      | `91 dst imm32` | `r[dst] = imm32`                                      |
| `0xA4` | ACCEPT   | `A4`           | return `1`                                            |
| `0xB9` | CMP      | `B9 src imm32` | `flag = (r[src] == imm32)`                            |
| `0xC2` | STRLEN   | `C2 dst`       | `r[dst] = strlen(input)`                              |
| `0xD0` | JMP      | `D0 disp16`    | `pc = pc + 3 + disp16`                                |
| `0xE3` | ROL      | `E3 dst n`     | `r[dst] = rol8((uint8_t)r[dst], n & 7)`               |

Return codes: `1` accept, `0` reject, `-1` ERR, `-2` unknown opcode, `-3` empty program, program counter out of range, or step budget exhausted.

**Disassembler.** 126 instructions over 559 bytes, terminating exactly at `0x22AF`. The program is a flat 12-step chain: a `STRLEN` / `CMP r0, 12` guard, then for each `i` in `[0, 11]` the block

```
LDB r1, in[i] ; MOV r2, K[i] ; XOR r1, r2 ; ROL r1, s[i] ; MUL r1, 157
MOV r3, i     ; ADD r1, r3   ; AND r1, 0xFF ; CMP r1, T[i] ; JNE reject
```

ending at `0x22AD: ACCEPT` / `0x22AE: REJECT`. All twelve branch targets are `0x22AE`.

| i  | K\[i]  | s\[i] | T\[i]  |
| -- | ------ | ----- | ------ |
| 0  | `0x5A` | 1     | `0xDA` |
| 1  | `0x6B` | 2     | `0x59` |
| 2  | `0x7C` | 3     | `0x9A` |
| 3  | `0x8D` | 4     | `0x86` |
| 4  | `0x9E` | 5     | `0x4D` |
| 5  | `0xAF` | 6     | `0x2F` |
| 6  | `0xC0` | 7     | `0xC4` |
| 7  | `0xD1` | 1     | `0x78` |
| 8  | `0xE2` | 2     | `0xD2` |
| 9  | `0xF3` | 3     | `0x4A` |
| 10 | `0x04` | 4     | `0xC1` |
| 11 | `0x15` | 5     | `0x5F` |

The constants follow `K[i] = (i << 4) + 0x5A + i` and `s[i] = (i % 7) + 1`; the `MOV r3, i ; ADD r1, r3` pair is index whitening.

**Password recovery.** Per byte the constraint is `rol8(p[i] ⊕ K[i], s[i]) · 157 + i ≡ T[i] (mod 256)`. Since `gcd(157, 256) = 1`, `157⁻¹ mod 256 = 181` (check: `157 · 181 = 28417 = 111 · 256 + 1`), so `p[i] = ror8(((T[i] − i) mod 256 · 181) mod 256, s[i]) ⊕ K[i]`. Worked for `i = 0`: `0xDA · 181 mod 256 = 0x22`; `ror8(0x22, 1) = 0x11`; `0x11 ⊕ 0x5A = 0x4B = 'K'`. Full result: `4B 65 73 74 72 65 6C 23 56 4D 32 36` = `Kestrel#VM26`. Re-encryption reproduces all twelve targets exactly.

**Emulation confirmation.** Both runs from `0x11d0` with `rdi = 0x2080`, `rsi = 0x22F`, `rdx = <input buffer>`, stopping on return to caller: `"Kestrel#VM26"` → `rax = 1` after 2,852 instructions; `"Kestrel#VM27"` → `rax = 0` after 2,857. The reject path is five instructions longer, consistent with `JNE` taking its branch to `0x22AE` instead of falling through to `ACCEPT`.

**Database changes.** Renamed `0x11d0` → `kestrel_vm_run` (prototype `uint32_t __fastcall(const uint8_t *prog, uint64_t prog_len, const char *input)`), `0x2080` → `kestrel_vm_program`, added a repeatable comment at `0x2080`, and created enum `KestrelVmOpcode` with the fifteen members above.

</details>

## 5. Follow up in the same conversation

The conversation keeps its context, so follow-ups can be short and build on each other. Here are four worth trying.

### Make the dispatcher readable

> Apply the KestrelVmOpcode enum to the opcode bytes at each compare in kestrel\_vm\_run so the dispatcher reads as named cases.

Assist scanned the function for `cmp al, imm8` instructions whose immediate matched an enum member, found exactly fifteen (one per opcode), and applied the enum to each operand. It then verified that the instruction bytes were unchanged (this is display metadata, not a patch) and re-decompiled:

```c
if ( v8 == VM_OP_AND )
...
switch ( v8 )
{
  case VM_OP_JNE:
    ...
  case VM_OP_ADD:
    ...
  case VM_OP_SUB:
```

### Check that the job is really done

> Is this actually solved? Confirm the password end to end, and tell me what in the database is still unresolved.

Rather than answering from memory, Assist re-checked the database. It tried to run the binary natively to confirm the password end to end (**Run shell commands** permission), which failed because the sample is a Linux ELF and the host was macOS, so it emulated `main` instead with `argc = 2` and `argv[1] = "Kestrel#VM26"`: the program printed `Maintenance mode unlocked. Welcome, engineer.` and returned `0`; with `Kestrel#VM27` it printed `Wrong password.` and returned `1`. It also renamed 21 local variables in `kestrel_vm_run` (`opcode`, `pc`, `regfile`, `cmp_flag`, `step_budget`, …) so the dispatcher reads naturally, then listed the remaining work ranked by value — the register file is mistyped as `_QWORD[6]` instead of `uint32_t[8]`, the trailing `.eh_frame` bytes are decoded as code — and asked before applying the one item that changes analysis rather than presentation.

### Clean up the remaining names

Right-click a default-named function in the Pseudocode or Disassembly view and pick the rename action under **IDA Assist**, or type the request yourself:

> Give the default-named function at 0x1140 a meaningful name based on its behavior, callers, and string/import usage.

Assist read the function, its callers, and the `.init_array` / `.fini_array` entries that reference it, identified the tail call through `_ITM_registerTMCloneTable`, and renamed it `register_tm_clones`, explaining the evidence and matching the binary's existing lowercase-underscore naming. A one-line follow-up finished the job:

> Rename the remaining default-named functions too.

The remaining four defaults became `plt0`, `deregister_tm_clones`, `__do_global_dtors_aux`, and `frame_dummy`, leaving zero `sub_*` names and `main` and `start` deliberately untouched because they are real ELF symbols.

### Verify a claim you care about

> Re-derive the operand encoding of every opcode from the handler disassembly, not from your earlier listing, and tell me if anything changes.

Assist's first pass had described `JNE` as carrying a "status-register selector" byte before the displacement. Checking each handler's operand fetches in the disassembly, Assist found that the handler at `0x1378` contradicts that: the two bytes after the opcode are the low and high halves of one signed 16-bit displacement, and there is no selector byte. Assist opened its reply with the correction, re-derived the listing under the corrected grammar (same 126 instructions, same twelve targets, same password), and revised the memory fact it had stored earlier so the wrong encoding does not come back in a future session.

The lesson generalizes: when a detail matters, ask Assist to re-derive it from the bytes rather than from its own earlier output.

## 6. Undo, review, and branch

* **Undo a database edit.** Supported edits are grouped into IDA undo steps; press <kbd>Ctrl</kbd>+<kbd>Z</kbd> in IDA. See [Tools & Actions → Undo, conversation rollback, and branching](/add-ons/assist/concepts/tools-and-actions.md#undo-conversation-rollback-and-branching).
* **Review what was remembered.** Open **Settings → Memory** (or type `/memory`) to see the three facts this session saved — the instruction set, the password, and the function map — and edit or delete any of them.
* **Try a different theory.** Click **Fork** on an earlier message to branch the conversation. Both branches stay, and **Settings → Tree** (`/tree`) switches between them. Branching restores the model's context, not the database; use <kbd>Ctrl</kbd>+<kbd>Z</kbd> for that.
* **Keep the transcript.** `/export` saves the conversation as a self-contained HTML file, and **Settings → Sessions** (`/sessions`) lets you resume or review past conversations, including every database edit and script they ran.

## What's next?

* [Emulation](/add-ons/assist/concepts/emulation.md) — the CPU and whole-process emulation surfaces used to confirm the password.
* [Memory](/add-ons/assist/concepts/memory.md) — how per-binary facts are stored, injected into prompts, and revised.
* [Tools & Actions](/add-ons/assist/concepts/tools-and-actions.md) — the tool surface, right-click actions, and suggestion pills.
* [Tool Catalog](/add-ons/assist/reference/tool-catalog.md) — every tool with its permission gate and mutation behavior.
* [Permissions & Privacy](/add-ons/assist/concepts/permissions-and-privacy.md) — capability prompt cards and confidentiality rules.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.hex-rays.com/add-ons/assist/how-tos/solve-crackme-with-assist.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.