Every attack in the previous parts exploits the same weakness: a program trusts the contents of memory, and that memory can be corrupted. The defenses answer it in four ways. Some keep certain memory from being executed, some make addresses hard to predict, some check that a control transfer is legitimate, and some detect corruption when it happens.

No single defense completely stops all the holes. A non-executable stack stops injected code from running, and doesn’t stop a program that overwrites its own decision variable. The parts below take the defenses roughly in the order attackers forced them, which is a useful way to see what each one defends against, not a strict timeline. The limit of each defense is as important as its purpose.

Fixing the Bugs

The most direct defense is to not write the bug.

Every memory error in the previous parts is a missing check: that an input fits the object it is copied into, that a size calculation does not overflow, that a length is not negative, that a string has room for its terminator, that no pointer outlives the object it points to.

Auditing code for those checks catches many of them, and Linux/macOS programs such as nm and grep can find calls to functions such as gets and strcpy in a binary or source tree, though a list of function names only shows you where to look and doesn’t show how the calls are used.

Static analysis tools examine code without running it and flag suspicious data flows; commercial and open-source analyzers are in wide use. Compilers and linkers now warn about the worst cases on their own, including a non-constant format string passed to printf and any use of gets.

Sanitizers

A sanitizer is a compiler option that inserts runtime checks into a program to catch memory and arithmetic errors as they happen. Several types exist:

• AddressSanitizer, developed by Google, detects out-of-bounds accesses, use-after-free, and invalid frees.

• UndefinedBehaviorSanitizer detects operations such as signed integer overflow and bad type casts.

• LeakSanitizer reports memory allocations that were never freed.

The checks catch the error when the instrumented code runs. AddressSanitizer, for example, reports where the memory was allocated, where it was freed, and where the bad access happened, and stops the program there. Whether a tool stops or keeps going depends on the tool and its options: UndefinedBehaviorSanitizer by default reports many errors and continues, while LeakSanitizer runs its check when the program exits.

All of them only see code that actually ran, so they are only as good as the tests that drive them, and they slow a program down, so the full instrumentation runs only during development and testing.

Fuzzing

Fuzzing is a testing technique: it feeds a program large numbers of malformed and semi-random inputs and watches for crashes or sanitizer reports. Its value is in reaching edge cases a human tester would never think to try, the overlong string, a negative length, or a large count that may overflow.

Modern fuzzers are coverage-guided, meaning they use compiler instrumentation to see which code paths each input exercises and then mutate inputs that reach new paths, so the fuzzer works its way into code that naive random input would never trigger.

Fuzzing combined with sanitizers is one of the most effective ways to find memory bugs before an attacker does, and a long run that finds nothing is evidence about the paths it reached, not proof of anything about the rest. The tools are mature: libFuzzer and AFL for C and C++, cargo-fuzz for Rust, and fuzzing has been built into the Go toolchain since Go 1.18 in 2022.

Using a Memory-Safe Language

The strongest fix is to not use a language that allows the bug. Java, C#, Go, and Python check the bounds of every array access and manage object lifetimes themselves, so an overflow becomes an exception and a use-after-free cannot be written.

Rust, created at Mozilla, gets the same guarantees without a garbage collector: its compiler enforces rules about which part of the program owns each object and how long a reference to it may live, and it checks array indexes at runtime in cases where it cannot prove them to be safe. The guarantees stop at the language boundary. Code marked unsafe, native libraries, and calls into C are outside them, and a safe-language program can still have logic bugs or a vulnerable C dependency underneath it.

The cost of switching can be substantial: large bodies of existing C code, performance-sensitive paths, the availability of libraries, and programmer familiarity all keep C in use. The results of switching are also documented. In September 2024, Google reported that writing new Android code in memory-safe languages had cut the share of memory-safety bugs among Android vulnerabilities from about 76 percent in 2019 to about 24 percent in 2024, because most new bugs are in new code. In February 2024, the White House Office of the National Cyber Director called on developers to move to memory-safe languages, and DARPA, the U.S. defense research agency, began a program to explore translating existing C to Rust automatically. There are also proposals to retrofit safety onto C and C++ themselves, though those are works in progress.

Non-Executable Memory

The earliest overflow exploits injected machine instructions into a buffer and set the return address to that buffer. This worked because the processor did not distinguish code from data; any readable memory could be executed.

Non-executable memory removes that. The operating system marks the stack, the heap, and other data regions as non-executable using the memory management unit’s per-page permission bits. That causes the processor to refuse to fetch instructions from those pages. AMD added the hardware support to its 64-bit processors in 2003 and Intel followed in 2004, and every major operating system uses t. The feature goes by several names. NX (no-execute) is the page permission itself, DEP (Data Execution Prevention) is what Windows calls its use of it, and W^X (write XOR execute) is the policy that no page should be writable and executable at the same time, which is what the permission is for.

Non-executable memory blocks the classic injected-shellcode attack outright. A buffer overflow can still overwrite data, but the processor will not run that data as code. It does not fix memory corruption, but it removes one of the things corruption could lead to. It does nothing against an overwritten decision variable, an information leak, or the format-string attack, none of which execute injected bytes.

Programs that generate code as they run, such as the just-in-time compilers inside browsers, live with it by writing the code to a page and then switching that page from writable to executable before jumping to it.

The effect of NX was to force attackers away from injecting code and toward reusing code that is already present and already executable.

Return-to-libc

Return-to-libc is the simplest code-reuse attack. Instead of pointing the return address at injected instructions, the attacker points it at a function that already exists in the program, typically a standard C library function, and arranges for the function to run with arguments the attacker chose.

The C library, libc, is linked into nearly every program, and it contains functions such as system, which runs a shell command. Directing control to system with an argument the attacker placed is equivalent to running a command, and because the library code is already marked executable, non-executable memory does not stop it.

On 32-bit Intel x86 architectures, the arguments to functions went on the stack, where an overflow could place them. On 64-bit systems the first arguments go in registers, so the attacker also needs a way to load the register, which is what the next technique provides. Removing dangerous functions from the library does not help much, because other useful functions remain.

Return-Oriented Programming

Return-oriented programming, or ROP, generalizes code reuse into a technique powerful enough to compute anything.

Instead of calling a whole function, such as system, the attacker finds short sequences of instructions within existing code that each end in a return. These are called gadgets. A gadget might load a value from the stack into a register, add two values, or store a register to memory, and then return.

The attacker uses the corrupted stack to supply both gadget addresses and the data the gadgets need. A return instruction takes its destination from the stack. By placing gadget addresses at the positions successive returns will read, the attacker makes each gadget return into the next.

The stack must also contain any values the gadgets read. For example, a gadget that pops a value into a register and then returns consumes two stack entries: the value to load, followed by the address of the next gadget. The stack therefore contains a mixture of addresses and data, laid out to match the instructions in the chosen gadgets.

With a sufficient set of gadgets, the chain can perform arbitrary computation using code that is already present and executable. Non-executable data pages do not stop this attack because the processor executes the existing gadgets, not the addresses and data stored on the stack.

ROP depends on two things: enough gadgets existing in the program’s executable memory, and the attacker being able to find where they are. Both are what the next defenses attack. Compilers can reduce the number of usable gadgets, and address randomization, which we cover next, makes their locations hard to predict. Building a ROP chain by hand is difficult, but many tools exist, including compilers, that search a binary for gadgets and assemble chains automatically, so the difficulty is not a defense on its own.

Address Space Layout Randomization

Each of these attacks depends on knowing where something sits in memory: the address of injected code to jump to, of a library function or gadget to reuse, or of the target a corrupted pointer is made to point to.

For years, a program’s code, its libraries, its stack, and its memory-mapped files were loaded at the same addresses on every run, so those addresses were predictable targets.

Address space layout randomization, or ASLR, removes that predictability by placing the libraries, the stack, the heap, and the program itself at different base addresses each time the program runs. A hardcoded address in an exploit then points to the right place only by luck. Randomizing the program’s own code requires build support since all references must be made as offsets rather than absolute addresses. On Linux and other ELF systems it means building a position-independent executable, one that can be loaded anywhere. This is now the default on most systems, but has not been for years. Windows instead ships relocatable images and turns randomization on with a linker option (/DYNAMICBASE).

ASLR turns a deterministic attack into a probabilistic one. To succeed, an attacker now has to learn or guess the randomized addresses, and an information leak that reveals any single valid pointer supplies the reference point that undoes the randomization for the whole region that pointer is in.

The strength of ASLR depends on how much randomness the system uses, measured as entropy. Early implementations were weak: Windows 7 randomized library placement with only 8 bits, giving 256 possible locations, few enough to try them all, and 32-bit systems in general had little room to randomize. On 64-bit Windows, high-entropy randomization of bottom-up allocations uses 24 bits, giving millions of placements. The exact figure depends on the region and the system, so what counts is the amount of randomness in a given mapping, not a single number for the whole process. ASLR is effective only when the entropy is high and no information leak exists to disclose an address. It doesn’t change anything about the bug itself, which can still corrupt data.

Stack Canaries

The stack holds both local variables and the return address, and a classic overflow works because nothing guards the boundary between them. A stack canary adds that guard.

The compiler arranges for a secret value, chosen at random when the program starts, to be copied into each protected function’s frame between its local buffers and its saved return address. Before the function returns, it checks that the copy still matches the original. A buffer that overflows to change the return address has to pass through the canary and change its value, so the check fails and the program aborts before the corrupted return address is ever used.

The name comes from the image of a canary in a coal mine, warning of danger before it reaches the miners. The idea appeared as StackGuard in 1998, and every major compiler now has it as an option, usually applied only to functions that contain arrays.

Canaries detect overflows rather than prevent them, and they protect only against the specific pattern of a contiguous overflow reaching the return address. They cannot protect against a heap corruption, a use-after-free, a write that skips over the canary, or an overflow that overwrites a nearby variable that the function uses before it returns.

Their strength also depends on the canary staying secret: an attacker who can read it, through an information leak, can include the correct value in the overflow and leave the check satisfied. A refinement from IBM’s ProPolice work, now in common compilers, reorders a function’s local variables so that arrays sit above the other locals in memory, which keeps an array overflow from reaching the function’s own scalar variables and pointers before it reaches the canary. It reduces that opportunity rather than removing it, and the runtime cost of the whole mechanism is small.

Hardened Allocators

Heap corruption exploits the allocator’s trust in metadata (its bookkeeping information) the program can reach. A hardened allocator adds defenses that make that metadata harder to forge and heap layout harder to predict:

• Integrity checks. Before using free-list pointers, the allocator confirms they are consistent, and it refuses to free a block that is already free, which stops most attacks that forge the links between chunks and most double-frees.

• Heap canaries. The allocator places a secret guard value around each block and checks it when the block is freed, so an overflow that ran into the guard is caught, though only at that later check.

• Pointer protection. The allocator can disguise free-list pointers by XORing them with a secret value and undoing the XOR before using them. Without the secret, an attacker has difficulty constructing an encoded pointer to a chosen address. However, this encoding does not detect tampering: flipping a bit in the stored pointer flips the same bit in the decoded address. Separate checks are needed to detect corruption, and an attacker who learns the secret can encode a replacement pointer.

• Delayed reuse. Recently freed blocks are held in quarantine rather than handed straight back, which breaks the predictable reallocation that use-after-free attacks depend on.

• Randomized placement and out-of-line metadata. Varying where allocations land, keeping objects of different types in separate regions, and keeping the bookkeeping outside the user-accessible blocks entirely, together make an overflow much less likely to reach anything the allocator trusts.

None of this removes the bug. It makes heap corruption unreliable instead of dependable, and an unreliable corruption is often one an attacker cannot build on.

Control-Flow Integrity

Every code-reuse attack ends in a control transfer the programmer did not write: a return to a gadget, or an indirect call through a corrupted pointer. Control-flow integrity, or CFI, checks those transfers at runtime against what the program could legitimately do. A compiler that implements it works out, for each indirect call, the set of functions that call could legitimately reach, and inserts a check that the target is in that set. In the common form, that set is every function whose type signature (not a cryptographic signature), its return type and parameter types, matches the function pointer being called, since a correct program only ever calls a pointer through its declared type.

A function pointer overwritten to point into the middle of a gadget fails the check. A pointer overwritten to point at a different function of the same type passes it, so the allowed set is the limit of the protection. Checks on function-pointer calls also say nothing about returns, which need the separate protection described next.

Software CFI has been in Windows since 2014 and in the Clang compiler, which is used to build Android and Chrome and can compile the Linux kernel with the checks enabled. Apple’s systems reach the same goal mainly through the hardware Pointer Authentication described next.

Hardware Support for Integrity

Software defenses raise the cost of an attack, but an attacker who finds a corruption path can still overwrite a return address or a function pointer. Processor vendors have added hardware that enforces control-flow and pointer integrity directly, so that tampering is caught by the silicon rather than by a check the software remembered to insert. Each of these needs the operating system and the compiler to turn it on, so a capable processor does not by itself mean a protected program.

Intel CET (Control-flow Enforcement Technology), available since the Tiger Lake generation in 2020, addresses the two things code-reuse attacks corrupt. The two mechanisms cover different transfers.

1. A shadow stack keeps a separate, protected copy of every return address, and on each return the processor compares the normal stack’s return address against the shadow copy and raises a control-protection fault that stops the program if they differ, which directly defeats a ROP chain built by overwriting return addresses.

2. Indirect branch tracking handles the forward direction: it requires every indirect jump or call to land on a special marker instruction, so an attacker cannot redirect one into the middle of existing code. Returns are the shadow stack’s job; indirect calls and jumps are indirect branch tracking’s. Windows has used the shadow stack since 2020 and Linux added support for user programs in 2023.

ARM Pointer Authentication Code, or PAC, protects pointers by adding a short keyed authentication code to each pointer, computed from the pointer value and a context such as the stack pointer, using a secret key held in registers that unprivileged application code cannot read (the operating system manages them). The code occupies the pointer’s unused high bits, the ones a 64-bit pointer does not need to hold an address, so a signed pointer is no larger than an ordinary one and cannot be used until the code is verified and stripped.

The compiler inserts the instructions that sign a pointer when it is stored and verify it before it is used, so the protection covers the pointers the compiler chose to sign, which are mainly return addresses and function pointers, rather than every pointer in the program. A pointer that was tampered with fails verification and faults. The code is a compact integrity tag, more like a message authentication code than a full cryptographic hash, and it is checked quickly. Apple has used it on every iPhone since the A12 processor in 2018.

ARM Memory Tagging Extension, or MTE, helps detect buffer overflows and use-after-free by checking that a pointer’s tag matches the tag of the memory it accesses. A tag is a small numeric label. MTE supports one 4-bit tag for each 16-byte block of memory, called a granule.

An allocator using MTE assigns tags to allocated memory and includes the matching tag in the pointer it returns. With checking enabled, the processor compares the two tags whenever the program reads or writes that memory. If a pointer tagged 3 reaches memory tagged 7, the processor detects the mismatch and faults.

This catches an overflow when a pointer runs into a neighboring allocation with a different tag. It also helps detect use-after-free: when the allocator changes the memory’s tag after freeing or reusing it, an old pointer still carries the previous tag.

MTE does not check exact object boundaries. A small overflow within the same granule can go unnoticed, as can an access to another allocation with a matching tag. Four bits allow only sixteen tag values. If tags are chosen independently and uniformly, two unrelated tags match with probability one in sixteen.

Apple Memory Integrity Enforcement, introduced with the iPhone 17 in September 2025, turns memory tagging on by default across the system, paired with a secure allocator that assigns the tags. It covers the kernel and the system processes that handle untrusted input. Earlier phones, including Google’s Pixel 8, had made tagging available; Apple’s step is making it always on for the system software, to make the memory-corruption exploit chains used by commercial spyware impractical on those devices.

Hardware Attacks the Defenses Do Not Reach

The defenses so far assume that memory faithfully holds what was written to it, but physics does not always cooperate. Rowhammer, disclosed in 2014, exploits the physical design of dense DRAM. Repeatedly reading, or hammering, one row of memory cells can disturb the electrical charge in neighboring rows and flip bits there, turning a 1 into a 0 or the reverse, without ever writing to those rows. A well-aimed bit flip can change a page table entry to gain access to memory the process should not reach, or flip a bit in a security check.

Memory makers added mitigations, and in 2021 researchers at ETH Zurich, the Swiss technical university, showed that non-uniform hammering patterns flipped bits on all 40 DDR4 modules they tested, and in 2024 their ZenHammer work reached AMD processors and one of ten DDR5 modules tested. How much any particular machine is exposed depends on its memory and its mitigations. Ordinary bounds checks do nothing about a physical bit flip, though other software mitigations can make the flips harder to aim. Rowhammer is a reminder that memory safety rests on an assumption about the hardware, and that assumption is not always safe.

How the Defenses Fit Together

Each defense sits at a different layer, and each exists because the layer above or below could not do the job alone.

• Hardware defines what the processor will execute and accept: NX, CET, PAC, and MTE.

• The operating system controls layout and permissions: ASLR, the non-executable marking of data pages, and the hardened allocator in its runtime library.

• The compiler inserts checks the source code does not control: stack canaries, safer library calls, format-string warnings, and CFI.

• Developer tools find bugs before release: sanitizers and fuzzing.

The choice of defense depends on what a given bug lets an attacker do, and every one of them leaves something open:

Which defenses an exploit must overcome depends on what it changes. An attack that corrupts a decision variable without redirecting execution can succeed despite protections for return addresses and function pointers.

That is why the layers are combined. Having to get past randomization, non-executable memory, a canary or a tag check, and the allocator’s integrity checks all at once, usually with an information leak as the first step, is what makes most memory bugs challenging to exploit.

This layering is defense in depth: no single layer is trusted to be perfect, and each one removes an easy path and forces the attacker toward rarer and less reliable conditions. Least privilege and sandboxing, which we cover later, add the last layer by limiting what a process that was compromised anyway can reach.

Lecture 7: Part 1 | Part 2 | Part 3