Reversing Custom CTF Virtual Machines

What is a Custom VM Challenge?

In reverse engineering and reverse/misc CTF challenges, authors frequently write a custom Virtual Machine (VM) interpreter inside an ELF binary or Windows EXE.

Rather than running standard x86 or ARM instructions, the binary loads an encrypted array of custom bytecode, and executes an internal fetch-decode-execute instruction loop:

+------------------+     Fetch Bytecode     +----------------------+
| Custom Bytecode  | ---------------------> | Interpreter Loop     |
| Array (Opcodes)  |                        | (Switch-case Dispatch)|
+------------------+                        +----------------------+
                                                       |
                                            Executes Virtual Stack /
                                            Virtual Registers

---

1. Locating the Dispatcher Loop

Open the binary in Ghidra/IDA and search for the main execution loop:

// Typical VM Interpreter Loop Signature in C
while (pc < bytecode_len) {
    opcode = bytecode[pc++];
    switch (opcode) {
        case 0x01: // V_ADD
            reg[dest] = reg[src1] + reg[src2];
            break;
        case 0x02: // V_XOR
            reg[dest] = reg[src1] ^ reg[src2];
            break;
        case 0x03: // V_CMP
            flags = (reg[src1] == reg[src2]);
            break;
        case 0x04: // V_JMP
            pc = target_addr;
            break;
        // ...
    }
}

---

2. Reconstructing the Virtual Instruction Set Architecture (ISA)

Document opcode mappings systematically in a table:

Opcode Byte	Virtual Mnemonic	Behavior / Action
0x10	PUSH imm	Push immediate integer to virtual stack.
0x20	POP reg	Pop stack top into virtual register.
0x35	XOR reg, imm	Bitwise XOR register with immediate key.
0xFF	EXIT / HALT	Terminate VM and print success/fail.

---

3. Disassembling Custom Bytecode

Write a Python script to dissemble raw bytecode using your mapped opcode table:

# Custom VM Bytecode Disassembler Script
bytecode = bytes.fromhex("1005200135aa2001ff")

pc = 0
while pc < len(bytecode):
    op = bytecode[pc]
    if op == 0x10:
        val = bytecode[pc+1]
        print(f"{pc:04x}: PUSH {val:#04x}")
        pc += 2
    elif op == 0x35:
        imm = bytecode[pc+1]
        print(f"{pc:04x}: XOR R1, {imm:#04x}")
        pc += 2
    elif op == 0xFF:
        print(f"{pc:04x}: HALT")
        break
    else:
        print(f"{pc:04x}: UNKNOWN ({op:#02x})")
        pc += 1

Once disassembled into virtual assembly, reverse the mathematical transformations applied to the input string to recover the CTF flag.