The stack is not the only place a program writes to while it runs. Memory allocated with malloc or new lives on the heap, a region of memory that is located above the code and static data. Unlike the stack, the heap does not contain return addresses, so an initial reaction is that heap data cannot be used to hijack control. That’s not the case. The heap holds function pointers, C++ object tables, and the memory allocator’s own bookkeeping. Corrupting any of them can also redirect execution or change program behavior.

There are two kinds of memory errors:

1. A spatial error accesses memory outside the bounds of an object, which is what every overflow so far has been.

2. A temporal error accesses an object outside its lifetime, after it has been freed.

Both can leak data or redirect execution, and both can happen on the stack and on the heap. We will now explore the overflows and lifetime errors that live in the heap, then the integer arithmetic mistakes that cause undersized allocations, the off-by-one error that writes a single byte too far, and finally format-string bugs, which can read and write memory without overflowing anything.

Heap Overflows

Memory on the heap is handed out by a memory allocator, a library that requests large regions from the operating system and divides them among the program’s allocation requests. The allocator divides the space into blocks, usually called chunks, and each malloc returns a pointer to the usable part of one chunk. The allocator keeps bookkeeping information for each chunk so that it can find neighboring chunks, merge adjacent free ones, and track which chunks are free. In many traditional allocator designs, this information sits in memory right next to the data the program uses.

A common implementation is to link free chunks into a doubly-linked list. A conceptual chunk looks different depending on whether it is in use:

Allocated chunk:  [ size info ][ the program's data ...  ]
Free chunk:       [ size info ][ fd | bk | unused ...     ]

The size information in the header lets the allocator locate the chunks on either side. When a chunk is freed, the allocator reuses the space where the program’s data was to store fd and bk, forward and backward pointers that link the free chunks together in a list. The exact layout varies from one allocator to the next. In these traditional designs, where the bookkeeping sits with the data, two facts apply: the allocator trusts the bookkeeping information, and that information is reachable from the program’s data. Some modern allocators keep their metadata out of band, in a separate region, which removes the second of those.

An overflow out of one chunk’s data writes into the header of the chunk that follows it, or into the free-list pointers of a freed neighbor. The damage shows up only later, when the allocator processes that chunk, for example to merge it with a neighbor or to remove it from a free list.

At that point the allocator follows the corrupted pointers. In older allocators, a classic attack arranged them so that the allocator wrote an attacker-chosen value to an attacker-chosen address. That is an arbitrary write, among the most powerful results an attacker can get, because the chosen address can be a function pointer, a C++ object’s method table, or a saved return address in another frame. Modern allocators check their metadata before using it, so an overflow no longer provides that capability automatically, but the underlying failure is unchanged: metadata the program could overwrite is metadata the attacker can forge.

Heap corruption does not need the allocator’s help to be useful. An overflow that reaches a neighboring object can overwrite a function pointer stored there, and the next call through that pointer goes wherever the attacker chose. C++ objects with virtual methods are the common case: each holds a hidden pointer to its vtable, the table of addresses for its virtual functions, and an overwritten vtable pointer turns the next virtual call into a jump to an attacker-chosen address. Reading heap memory that holds pointers is also valuable, because it tells the attacker where libraries and other regions sit, which undermines address randomization by giving a known reference point.

Use-After-Free and Double-Free

Buffer overflows violate the boundaries of memory by writing outside an allocation. The next two bugs violate the lifetime of memory, by using it after it has been released or releasing it twice. They are even harder to find than overflows because the symptom appears far from the cause.

Use-After-Free

A use-after-free happens when a program keeps a pointer to a block, frees the block, and then uses the pointer anyway. The memory still physically exists, but the allocator now owns it and may hand it to the next allocation request at any time.

struct user {
    char name[32];
    void (*handler)(void);
};

struct user *u = malloc(sizeof(struct user));
u->handler = &process_user;
free(u);
/* ... later ... */
u->handler();   // use-after-free: u points to freed memory

After the free, one of three things is true when the stale pointer is used:

1. Nothing else has reused the memory, so the old contents are intact and the program appears to work. This is what makes the bug hard to catch in testing.

2. A later allocation has reused the memory for a different object, so the stale pointer now overlaps unrelated data, possibly data an attacker placed there.

3. The allocator has written its own free-list bookkeeping into the block, so reading through the stale pointer returns allocator internals or crashes.

The second case is the one an attacker works toward. By controlling the allocations and frees that happen between the free and the use, an attacker can arrange for the freed block to be handed back to an allocation they control, and fill it with data of their choosing. Steering the allocator’s state this way is called heap grooming. A related technique, heap spraying, fills a large part of the heap with copies of the attacker’s data so that a guessed address is likely to land on it. When the stale pointer is then followed, for example through the handler function pointer above, it leads into attacker-controlled bytes.

C++ objects with virtual methods are a frequent target, for the same reason as above: an attacker who controls what reuses a freed object controls its vtable pointer, so the next virtual call jumps to an address of their choosing. Use-after-free bugs in browser scripting engines, where objects are freed while event handlers still reference them, have been among the most heavily exploited vulnerabilities of the last decade.

Setting a pointer to null after freeing it is a common piece of advice, but it helps only with that one pointer. Any other pointer to the same object is still stale. The real fix is clear ownership of every object, so that exactly one place in the code is responsible for freeing it, which is what memory-safe languages enforce automatically.

Double-Free

A double-free passes the same block to free twice. Because many allocators store free-list pointers inside the freed block, freeing it again links a block that is already on the list, creating a cycle or an inconsistency. The consequences of this action depend on the allocator. A hardened allocator detects the second free and aborts. An allocator without that check corrupts its free list silently, and the corruption can be steered: by writing into the block between the two frees, an attacker can plant their own pointers in the free list.

The danger of this is that a block can then be returned from two separate malloc calls, so two parts of the program hold what they believe are independent objects but they occupy the same address, and a write to one changes what the other reads. On an allocator without double-free detection, the sequence can be as plain as this:

char *a = malloc(128);
char *b = malloc(128);
free(a);
free(b);
free(a);                 // a is now on the free list twice
char *x = malloc(128);   // returns a
char *y = malloc(128);   // returns b
char *z = malloc(128);   // returns a again

Now x and z point to the same memory. The exact order in which an allocator hands blocks back is not something the C language defines, so this is a picture of one possible failure rather than a guaranteed outcome, but the mechanism is the one real exploits used.

Why Lifetime Bugs Are Hard to Find

These bugs are unreliable in a way that overflows usually are not. An overflow that corrupts a return address tends to crash at the next return, which is close to the cause of the bug.

A lifetime bug depends on the allocator’s state, which varies from run to run, so whether freed memory has been reused depends on what else the program did. The free and the use are often in different functions, in different callbacks, or on different threads. And the program frequently works in testing, because the freed memory has not been reused yet, which hides the bug until a particular sequence of events triggers it.

The defenses in the last part, including allocator quarantine and hardware memory tagging, are aimed specifically at making these bugs visible.

Type Confusion

A type confusion bug makes code treat an object as a type it is not. Code that expects a pointer field finds an integer there, or a virtual call is dispatched through an object whose layout is not what the code assumed.

A use-after-free produces type confusion when the freed memory is reused for an object of a different type, but it also arises on its own, from a bad cast or a type check a language runtime got wrong.

In 2024, attackers exploited a type-confusion vulnerability in Chrome’s JavaScript engine through malicious web pages. The bug caused the engine to treat an object as the wrong type and misinterpret its contents. The signed-versus-unsigned errors described below involve converting an integer value between types. Type confusion instead causes code to interpret an object’s fields using the wrong memory layout.

Integer Overflow

Integer errors are subtle because they appear in ordinary arithmetic, not in any function a reviewer flags as unsafe. They become security problems when a miscalculated size or index leads to an overflow, an out-of-bounds access, or an allocation that is smaller than the code believes.

Fixed Width and Wraparound

On every current architecture, an integer of a given type occupies a fixed number of bits, which bounds the values it can hold.

When an operation produces a result outside the range, the extra high bit has nowhere to go. The processor does not raise an exception.

For unsigned arithmetic, C defines what happens: the result wraps around, so for a 32-bit unsigned integer, 4,294,967,295 plus 1 is 0. For signed arithmetic, C does not define the outcome. Signed overflow is undefined behavior, and that is worse than wrapping, because a compiler is allowed to assume it never happens.

A check written as “if adding 1 made the number negative, something overflowed” can be deleted by the optimizer, since by the language’s rules the condition can never be true. The hardware will usually wrap, so the examples below show the most negative value appearing where a large positive one was expected, but code cannot depend on it.

Languages differ in how they handle overflow. C, C++, and Go define unsigned arithmetic to wrap around. Java also defines wraparound for ordinary arithmetic on its signed int and long types. Rust checks for overflow by default in a debug build, where it stops the program, and wraps in a release build, though either profile can be configured otherwise. Python integers have no fixed width; they grow as needed, at some cost in performance. JavaScript’s ordinary numbers are double-precision floating point, which represents every integer precisely up to 2⁵³ but cannot tell apart some integers past it; its separate BigInt type handles arbitrary sizes. The lesson for a C programmer is that the hardware will not catch the overflow, so the code has to.

Why It Is a Memory-Safety Problem

Integer overflow is important because of what a wrapped value does to a memory operation. The common case is a size calculation before an allocation, followed by code that still uses the original count:

size_t n = get_count_from_network();
struct record *records = malloc(n * sizeof(struct record));
for (size_t i = 0; i < n; i++)
    records[i] = read_record();

If the attacker controls n and makes it large enough that n * sizeof(struct record) overflows, the product wraps to a small value, and malloc succeeds with far less space than the code believes it has. The loop then stores n records into it. On a 32-bit system with a 16-byte record, a count of 268,435,457 makes the product 4,294,967,312, which wraps to 16. The allocation holds one record, and the loop tries to write 268 million.

The buffer overflow occurs because the allocation uses the wrapped size, but the loop still uses the original record count. If the program instead used the same wrapped size for both the allocation and a subsequent read, it would allocate 16 bytes and read at most 16 bytes into them, so that read would not overflow the buffer.

The SMBGhost bug in Windows, CVE-2020-0796, was an example of this. A length field in a compressed network message was attacker-controlled, and an overflow in the kernel’s size calculation caused it to allocate a buffer that was too small, so decompressing into it overwrote kernel memory and gave an attacker the ability to write anywhere in the kernel. The bug required no authentication and was reachable over the network wherever Windows file sharing was exposed, so a successful exploit could spread from one machine to the next on its own.

The Pegasus spyware reached iPhones in 2021 through the same mistake, an integer overflow in the count of symbols in a JBIG2-compressed image inside a file that claimed to be a GIF, which produced an undersized array and let the decoder write pointers past the end of it. The file arrived over iMessage, and the GIF extension led the phone to render it automatically, with nothing the user had to do. The renderer, however, chose the decoder from the file’s contents, not its name, and ran the JBIG2 code. A file extension is no guarantee of which parser will run.

Preventing it means checking before the operation, not after. Verify that n is no larger than SIZE_MAX / sizeof(struct record) before multiplying, or use a compiler built-in such as __builtin_mul_overflow that reports whether the multiplication overflowed. An application limit on n, something like a maximum record count, belongs there too, so that a request that is representable but absurd is also refused. Keep sizes in size_t, the type meant for object sizes, and remember that it wraps like any other unsigned type.

Signed and Unsigned Mismatches

Mixing signed and unsigned integers in one expression quietly changes results, because the compiler converts between them according to rules most programmers never look up. The classic bug is a length that is negative because of an earlier error, checked against a signed limit and then handed to a function that takes an unsigned size:

int len = get_length();        // returns -1 on error
char dst[128];
if (len < 128)
    memcpy(dst, src, len);     // -1 becomes a huge size_t here

The comparison is between two signed values, and -1 is less than 128, so the test passes. memcpy takes its length as a size_t, an unsigned type, and converting -1 to it produces the largest value the type can hold, so the copy runs far past the end of dst. The fix is to reject negative values before using them as a size or an index, and then to compare against the real capacity: if (len >= 0 && (size_t)len <= sizeof(dst)).

Changing the limit’s type changes the result of the comparison. If the limit in the comparison had been a size_t rather than the literal 128, the compiler would have converted len to unsigned for the comparison, the huge value would have failed the test, and the bug would not exist. Which way a mixed comparison goes depends on the types involved, which is exactly why mixing them is dangerous: the code reads as a plain bounds check, and whether it is one depends on a declaration somewhere else.

64-Bit Does Not Make This Go Away

A 64-bit unsigned integer can hold values up to about 18 quintillion, which is more than enough to count the bytes in any file or the money in any account, so overflow can feel like a solved problem. That’s not the case. The problen does not go away. A field that an attacker can set can be set to any (possibly ridiculously high or ridiculously low) value regardless of how wide the type is.

More to the point, the default int in C on Windows, Linux, and macOS is still 32 bits even on 64-bit machines, and network protocols are full of narrow fields: an IPv4 header’s time-to-live is 8 bits, its total length is 16 bits, its fragment offset is 13, a TCP window is 16, and the week number in the legacy GPS navigation message is 10 bits. Code that handles these fields, and legacy code written to save space, still works with small integers where overflow is easy to reach.

Off-by-One Errors

An off-by-one error writes or reads one position beyond the correct boundary. The name comes from the fencepost problem: a straight fence with N sections needs N+1 posts, and confusing the two counts by one is the same mistake. The error is a single wrong boundary, but a single byte can be enough to cause damage.

char buf[16];
for (int i = 0; i <= 16; i++)
    buf[i] = src[i];

The loop should stop at i < 16. Written with <=, it writes a seventeenth byte, one past the end of a 16-byte array. That byte may land on a saved value or the low byte of a pointer, and the effect depends on the layout. String handling is a common source: copying exactly 16 characters into a 16-byte array leaves no room for the terminating null, producing either an unterminated string or a one-byte overflow of whatever follows.

A one-byte overwrite can change a nearby variable or part of a saved pointer. If it reaches a saved return address, it can change where execution resumes. If it reaches a saved frame pointer, it can cause the caller to use the wrong location for its stack frame, potentially reading a later return address from the wrong place. Either can cause a crash or, under suitable conditions, let an attacker redirect execution. The outcome depends on the compiler’s stack layout and the protections enabled.

A boundary mistake of one byte can also be the start of a much larger overwrite. In January 2021, Qualys, a security company, disclosed a heap-based buffer overflow bug in sudo, the Unix program that runs commands with elevated privilege. It had been in the code since 2011 and exploiting it gave any local user root access.

The parser in sudo skipped the character after a backslash, and with a trailing backslash, it skipped the null that terminates the string, so the copy that followed kept going and wrote past the end of its heap buffer by as much as the attacker cared to supply. The boundary error was one byte, but the overflow it enabled was not. The defense is the same as for any overflow: validate sizes, use counted operations correctly, and test with boundary inputs, the values at and just past each limit, which is where these bugs live.

Format-String Vulnerabilities

The printf family of functions, including fprintf and sprintf, formats output according to a format string that contains directives starting with %. Each directive tells the function to read and format one more argument: %d reads an integer, %s reads a pointer to a string, %x reads and prints a value in hexadecimal. The function steps through the format string and pulls one value for each directive from where it expects the arguments to be.

It’s probably the most commonly-used function for formatted output in C and C++ and the first function programmers learn about when they write their first C program that calls printf("hello, world\n").

The vulnerability is using untrusted input as the format string itself:

printf(user_input);   // unsafe

If user_input contains format directives, printf acts on them. It has no count of how many arguments it was really given; it trusts the format string to have the right number of % directives. If the format string asks for more values than were passed, the function reads whatever happens to be in the registers and stack locations where arguments would have been, and prints them. A string of %x directives therefore dumps memory, which can leak return addresses, pointers, or other data, and that leaked information can defeat address randomization by revealing where code sits.

One printf directive makes the bug even worse. The %n directive does not print anything. Instead, it writes the number of characters printed so far to the address given by its argument. When the attacker controls the format string and can arrange for that argument to be a pointer of their choosing, a read becomes a write, and values can be written to memory without overflowing any buffer at all.

The directive is obscure, which is part of the danger: most programmers do not know it exists, reviewers overlook it, and tools that scan for unsafe function names such as gets miss it because the function being misused is printf. Microsoft’s C runtime ships with %n disabled unless a program turns it on, while Linux and macOS support it for standard compliance.

The defense is simple and nearly absolute: never pass untrusted data as a format string. Supply a constant format string and pass the user’s data as an argument, so printf("%s", user_input) instead of printf(user_input), or use fputs when no formatting is needed. Compilers warn about non-constant format strings when -Wformat-security is enabled, and -D_FORTIFY_SOURCE=2 adds runtime checks in some libraries. The vulnerability exists only when input is treated as a format specification rather than as data, so keeping the two separate removes it.

Next: [Part 3 - Defenses]](memory-3.html)
Lecture 7: Part 1 | Part 2 | Part 3