Integer Overflows & Underflows

What is an Integer Overflow?

An integer overflow occurs when an arithmetic operation attempts to create a numeric value that is outside the range that can be represented with a given number of bits.

In C/C++, fixed-size integer types have bounded min/max capacities:

Type	Size (32/64-bit)	Signed Range	Unsigned Range
char	1 byte (8 bits)	$-128$ to $127$	$0$ to $255$
short	2 bytes (16 bits)	$-32,768$ to $32,767$	$0$ to $65,535$
int	4 bytes (32 bits)	$-2,147,483,648$ to $2,147,483,647$	$0$ to $4,294,967,295$
size_t	4 or 8 bytes	N/A	$0$ to $2^{64}-1$

When an unsigned integer exceeds its maximum limit, it wraps around to 0: $$255 + 1 \equiv 0 \pmod{256}$$

---

Common Vulnerability Patterns

1. Integer Overflow in Allocation Size

void allocate_array(unsigned int count) {
    // Integer overflow in multiplication: count * sizeof(int)
    // If count = 0x40000001 (in 32-bit):
    // 0x40000001 * 4 = 0x100000004 -> wraps around to 4 bytes!
    int *array = (int *)malloc(count * sizeof(int));
    
    for (unsigned int i = 0; i < count; i++) {
        array[i] = read_user_input(); // Heap buffer overflow!
    }
}

The program allocates a tiny buffer of only 4 bytes, but the loop writes count ($1,073,741,825$) elements into heap memory, triggering a heap overflow.

---

2. Signedness Mismatch Bypass

void process_packet(int length) {
    char buffer[256];

    // Signed comparison check
    if (length > 256) {
        puts("Error: Packet too large!");
        return;
    }

    // memcpy takes size_t (unsigned int)
    // If length is negative (-1 / 0xFFFFFFFF):
    // (-1 > 256) evaluates to FALSE!
    // But memcpy interprets length as 4,294,967,295 bytes!
    memcpy(buffer, user_data, length); // Stack buffer overflow!
}

By supplying a negative length like -1 (0xFFFFFFFF), the signed if check passes, but memcpy treats -1 as an enormous size_t integer, causing immediate stack corruption.

---

CTF Exploitation Steps

1. Locate integer bounds checks: Inspect decompiled code in Ghidra/IDA for operations involving user-controlled sizes, counts, or array indexes.
2. Calculate wraparound values: Determine the exact input integer required to overflow the arithmetic operation: $$\text{Target Wrapped Size} = (\text{Desired Bytes}) \pmod{2^{\text{bit_width}}}$$
3. Trigger memory corruption: Pair the integer overflow with the subsequent read(), memcpy(), or array indexing operation to overwrite saved registers, function pointers, or malloc metadata.