Anti-Debugging & Evasion Techniques

What is Anti-Debugging?

Anti-debugging techniques are code routines inserted into executables by malware authors or CTF challenge creators to detect whether the application is running under the control of a debugger (such as GDB, x64dbg, or IDA Pro).

If a debugger is detected, the program typically terminates, displays a fake flag, or intentionally crashes.

---

Common Linux Anti-Debugging Methods

1. ptrace(PTRACE_TRACEME) Check

In Linux, a process can only be attached to by one tracer at a time. A binary calls ptrace(PTRACE_TRACEME, 0, 1, 0) on startup:

if (ptrace(PTRACE_TRACEME, 0, 1, 0) < 0) {
    printf("Debugger detected! Exiting...\n");
    exit(1);
}

If GDB is already attached, ptrace returns -1, triggering the detection branch.

Bypass in GDB:

Catch the ptrace syscall and force its return register ($rax / $eax) to return 0:

(gdb) catch syscall ptrace
(gdb) commands
> set $rax = 0
> continue
> end

---

2. Inspecting /proc/self/status (TracerPid)

Linux exposes process status information in /proc/self/status. If no debugger is attached, TracerPid is 0. If GDB or strace is attached, TracerPid contains the PID of the tracer process.

Bypass:

Override open() or read() via LD_PRELOAD, or patch the conditional jump (jne $\rightarrow$ nop) in Ghidra/GDB.

---

Common Windows Anti-Debugging Methods

1. IsDebuggerPresent() API

Queries the BeingDebugged flag in the Process Environment Block (PEB):

if (IsDebuggerPresent()) {
    ExitProcess(0);
}

Bypass:

• x64dbg / ScyllaHide: Use the ScyllaHide plugin to automatically patch PEB flags.
• Manual GDB/x64dbg patch:

  # Set BeingDebugged byte to 0 in PEB
  set *(unsigned char*)(fs:0x30 + 2) = 0 (32-bit)
  set *(unsigned char*)(gs:0x60 + 2) = 0 (64-bit)

---

2. CheckRemoteDebuggerPresent()

Uses internal NtQueryInformationProcess to query ProcessDebugPort (0x07).

---

Timing Checks (RDTSC)

Anti-debugging routines measure the CPU cycle count elapsed between two instructions using the RDTSC (Read Time-Stamp Counter) instruction.

rdtsc
mov ebx, eax        ; Save initial timestamp
; ... suspected code ...
rdtsc
sub eax, ebx        ; Calculate delta
cmp eax, 0xFFFFF    ; If delta is unusually large (human stepping in debugger)
ja  debugger_found

Bypass:

NOP out the conditional jump instruction (cmp / ja) or patch the assembly binary using Ghidra / x64dbg.

---

Binary Patching Strategy

1. Open the binary in Ghidra.
2. Locate the conditional jump following the anti-debug check (JZ, JNZ, JNE).
3. Right-click $\rightarrow$ Patch Instruction $\rightarrow$ Change conditional branch to NOP (0x90 0x90) or unconditional JMP.
4. Export the modified binary via File $\rightarrow$ Export Program $\rightarrow$ Binary.