On the evening of November 2, 1988, a graduate student at Cornell released a program onto the Internet that copied itself from machine to machine (a self-propagating program like this is called a worm). One of the ways it got in was through fingerd, a small network service that answered queries from the finger program about who was logged in. The service read each request into a fixed-size array on its stack with gets(), a C library function that keeps reading until it sees the end of a line and never checks how much room it has. A request longer than the array overwrote what was stored next to it, including the address the service would return to when the current function finished. This behavior allowed the attacker to deliver code as data and make the program execute that code. By the next morning, roughly 6,000 machines, about a tenth of the Internet at the time, were running the worm.

Nearly four decades later, the same mistake is still being made. In June 2026, Rapid7, a security firm, published a stack buffer overflow in HP’s Poly desk phones that let an unauthenticated attacker on the network run code on the phone as root, on phones with an optional call-setup feature turned on.

In January 2026, OpenSSL, the cryptography library behind most of the world’s TLS connections, fixed a stack overflow in the code that parses encrypted and authenticated messages. (A Datadog Security Labs article.)

Microsoft reported in 2019 that about 70 percent of the vulnerabilities it had patched over the previous twelve years were memory safety bugs, and Google reported the same fraction for Chrome in 2020.

The reason is that the operating systems, browsers, network services, and device firmware we depend on are written mostly in C and C++. Those languages give a programmer direct control of memory: allocate it, take its address, do arithmetic on that address, and write through it. A pointer carries no record of how big the object it points to is or whether that object still exists. That control is what makes efficient system code possible, but it also turns a one-line mistake into a way for an outsider to run their own code.

We begin with what an attacker is trying to achieve, how a running program is laid out in memory, how writing past the end of an array on the stack changes where the program goes next, and what an attacker places in the overflow to make that useful.

The failures that live in the heap and in integer arithmetic come after that, and the defenses that operating systems, compilers, and processors have built against all of them come last.

The defenses are a separate topic because each of them was a response to an attack described here, and they make sense only once the attack is understood.

Hijacking and Injection

Most attacks on a running program fall into two categories, and the terms are used loosely enough that it is worth defining and distinguishing them before going further. The two terms overlap: an attacker can redirect a program, add new code to it, or do both at once.

Hijacking means taking control of a process by intercepting, manipulating, or redirecting what it was already going to do. Session hijacking takes over an authenticated session. An attacker who captures a web cookie, or who sits between a user and a server as a machine-in-the-middle, can act as that user without ever knowing the password. Control-flow hijacking changes the order in which a program’s instructions run, usually by corrupting a return address or a function pointer so that execution jumps somewhere the programmer never intended. The destination can be code that already exists in the program, which is how the return-to-libc and return-oriented programming techniques covered later work, or code the attacker added.

Injection means getting a process to run instructions or commands that the attacker supplied. Command injection feeds input to a program that the program then hands to a shell, so the attacker’s text is executed as a command. SQL injection does the same to a database query, by supplying data that the database reads as part of the query. Both are taken up later. Code injection places the attacker’s own machine instructions into the program’s memory and arranges for the processor to execute them. Library injection forces a program to load a different shared library than the one intended, by changing a search path or an environment variable, so that the attacker’s code runs with the program’s privileges.

The classic stack overflow does both things at once: it injects instructions into a buffer and then hijacks control flow by pointing a return address at them. We will explore that combination, and the data corruption and code reuse that are a result of it. We will cover cmmand injection and library injection later.

When These Attacks Are Worth the Effort

Running code inside another process gives the attacker whatever that process can reach. Three situations make that useful to attackers:

1. The program runs with more privilege than the attacker has. A setuid program runs with the privileges of its owner rather than the user who started it, and the dangerous case is a setuid program owned by root. A network service can also run as root because the system started it that way at boot. Either way, code the attacker gets such a program to run has root’s privileges.

2. The program runs on a machine the attacker cannot log in to. Most network servers are like this. Code that the attacker gets a server to run gives them a foothold on a machine they had no account on, which is remote code execution (RCE), and it is valuable even when the service runs as an unprivileged user.

3. The program holds something the attacker wants. A browser stores session tokens and saved passwords, and a document viewer holds the document. Taking over either one is useful without any special privilege at all.

A privileged service that listens on the network has the first two properties at once, which is why network-facing privileged services have always been the highest-value targets. Least privilege and sandboxing, covered later, exist to limit what a compromised process can reach.

Where the Bugs Come From

Behind almost every one of these attacks is a program that made an assumption about its data without checking it. An attacker studies the program to find that assumption and then supplies input that makes it false. The buffer overflows, integer overflows, and format-string bugs that follow are each a specific broken assumption, but four assumptions come up again and again. They are:

• The input will fit in the space allocated for it.

• Arithmetic on a size or a count will not overflow.

• Input provided as data will never be treated as a command.

• A file is in the format its name or header claims.

A vulnerability is a weakness that can be exploited to perform an unauthorized action. An exploit is the input or program an attacker constructs to take advantage of it. The same memory bug can cause only a crash on one build of a program but allow code execution on another, because whether it can be exploited depends on the surrounding code, the memory layout, and the defenses in place. The parts that follow are about how a broken size assumption becomes a vulnerability, and then about the defenses that make the resulting exploits unreliable.

Cryptography does not make a vulnerable parser memory-safe. Encrypting a connection or signing a file protects the data in transit, but the program that receives it still has to parse the data, often before it can check a signature, and a parser with a buffer overflow is vulnerable no matter how the bytes arrived. Cryptographic software is itself written in C, and the OpenSSL overflow mentioned above was in the code that parses a CMS message (Cryptographic Message Syntax, a standard format for signed or encrypted data such as secure email and signed software), one of the encrypted and authenticated message types the library has to take apart before the recipient can verify anything. The flaw, CVE-2025-15467, was that an oversized initialization vector carried in the message’s parameters was copied into a fixed-size buffer on the stack, so an attacker-controlled length overran a fixed buffer, and it did so before any key or authentication tag was checked.

How a Process Is Laid Out in Memory

Every running program is given its own virtual address space, a range of memory that the operating system and the processor’s memory management unit map onto physical memory.

The layout is similar on Linux, the BSD-derived systems including macOS, and Windows, though the exact addresses differ. Understanding it is necessary because almost every memory vulnerability depends on what is located next to what.

A process address space is divided into regions, each holding a different kind of data. They are listed here from the lowest addresses to the highest:

• The text segment holds the program’s machine instructions. It is marked read-only and executable so that the program cannot accidentally or deliberately modify its own code.

• The data segment holds global and static variables that the program initialized to a nonzero value.

• The BSS segment holds global and static variables that start out zero. (The name is a historical assembler term, “block started by symbol”; Linux and the other systems still use the segment, and the ELF executable format still has a .bss section that the loader fills with zeroed memory.)

• The heap holds memory the program allocates while it runs, through malloc in C or new in C++. It grows toward higher addresses as the program requests more memory.

• The stack holds the local variables, saved registers, and return addresses for each function call that is currently in progress. Each thread has its own stack. It sits at the high end of the space and grows downward, toward lower addresses, as calls nest, so it grows toward the heap while the heap grows toward it.

This is a model rather than a complete map. A process also has read-only data, the shared libraries it loads, and files it has mapped into memory. A modern allocator may manage several separate heap regions.

The regions a program writes to while running are the stack, the heap, and the global variables in the data and BSS segments, and all three are places where an overflow has something next to it worth corrupting. The text segment is read-only, so an overflow cannot rewrite the program’s instructions directly.

The Stack and the Call Stack

The stack is the structure that makes function calls work. When a function is called, the program sets aside a region of the stack for that call, called a stack frame, and releases it when the function returns. In the conventional layout on an intel x86 family processor, a frame holds four things:

1. The arguments passed to the function, for those not passed in registers.

2. The return address, the location the processor should jump to when the function finishes. The call instruction pushes it on the stack automatically.

3. Saved registers, including, on some builds, the caller’s frame pointer, which links back to the previous frame.

4. The function’s local variables, including any fixed-size arrays.

When the function returns, it first adds to the stack pointer the number of bytes it had set aside for saved registers and local variables, moving the pointer up to the saved return address. The processor then executes a return instruction, which reads that address off the stack and jumps to it. Adjusting the pointer this way frees the frame without erasing anything in it. A deeper call chain just stacks more frames on top of one another, and each return unwinds one frame.

The reason the stack is a target is in its structure. A function’s local variables, including arrays and the return address for that function are both in the same frame. These local variables are placed at a lower address than the return address. An array is filled from its start toward higher addresses. So a write that runs off the end of a local array moves toward the saved return address. The compiler decides how far away it is and what else lies in between.

What Writing Past an Array Does

Consider a function with a fixed-size local buffer that copies input into it without checking the length:

void greet() {
    char name[16];
    gets(name);
    printf("Hello, %s!\n", name);
}

The array name is a local array stored on the stack. When greet() is called, the compiler allocates space for name along with any saved registers and the return address to main(). The layout of the stack frame looks roughly like this:

The array name lives in greet’s stack frame, below the saved registers and the return address to the caller. gets() copies characters until it reaches the end of a line and then adds a terminating null (the value 0) byte, with no idea how big name is. An input of fifteen characters fits, with the null in the sixteenth byte. A longer input keeps writing past the end of name, up into whatever the compiler placed above it and then the return address.

In C, an access outside the bounds of an object is undefined behavior: the language makes no promise about what happens. What usually happens is nothing, at first. The memory just beyond the array is mapped and writable, so the processor writes the bytes without complaint and whatever was there is silently replaced. The processor checks page permissions, not array bounds, so a fault happens only if the write reaches a page that is unmapped or not writable, which is usually far past the end of a stack frame.

This silence is what makes the bug dangerous. The corruption has already happened by the time anything goes wrong, and what goes wrong happens later, when the function returns.

When greet returns, the processor reads the return address off the stack and jumps to it. If the overflow replaced that address and nothing stopped the program first, three things can happen, depending on what the new address is:

1. It points to an unmapped or non-executable location. The processor faults and the program crashes. A crash in a server is a denial-of-service attack. A crash does not by itself mean an attacker can run code, and a core dump that holds the program’s secrets is useful only to an attacker who can get at it.

2. It points into existing code, such as a library function. The processor runs that code, provided the attacker has also arranged whatever arguments and state it needs. Choosing the function and setting up its arguments is the basis of the return-to-libc and return-oriented programming attacks in the defenses section.

3. It points to an address within the overflowed buffer, and the buffer contains machine instructions the attacker supplied. The processor runs those, but only if the memory holding the buffer is executable. This is classic code injection.

Not every buffer overflow lets an attacker take control of execution. Some cause a crash, some corrupt data, and some have no visible effect in a particular run. Exploitability depends on what the overflow can overwrite, how much control the attacker has over the input, and which protections are enabled. Even without code execution, an attacker may be able to change an authorization decision or repeatedly crash a service.

Why gets() Cannot Be Used Safely

The gets() function reads a line of input into a buffer, and it was common in older code and textbooks because it is easy to call. It cannot be made safe, because the caller passes it only a pointer to the buffer and never its size, so gets() has no way to know when to stop. Any input longer than the buffer overflows it. The function was removed from the C standard library in the C11 revision. The replacement is fgets(buffer, sizeof(buffer), stdin), which reads at most one less than the size given, so that it always has room to add the terminating null. It may leave the rest of a long line unread, and the caller has to decide what to do about that.

One trap with the replacement is the size argument. sizeof(buffer) is the size of the array only where buffer was declared as an array. Inside a function that received the buffer as a char * parameter, sizeof gives the size of the pointer, usually 8, which has nothing to do with the real buffer. If the real buffer is smaller than that, the copy overflows it; if larger, the copy stops short. The capacity has to be passed in separately.

The problem is not limited to one discredited function. gets() is the clearest example because it has no safe use, but any operation that copies data into a buffer without comparing the length against the buffer’s size has the same flaw. strcpy copies a string until it hits a null terminator, strcat appends one string to another, and sprintf writes formatted output into a buffer, and none of them knows the size of the destination. The counted versions take a maximum length, but each one counts something slightly different, and the differences cause bugs of their own:

• strncpy(dst, src, n) writes exactly n bytes. If the source length is at least n, the destination gets no terminating null, and the next function that reads it as a string runs off the end.

• strncat(dst, src, n) limits how many characters are appended, not the total size of the destination. The caller still has to leave room for what is already there and for the final null.

• snprintf(dst, n, ...) does limit the total written to n bytes, including the null, and it returns the length the output would have had. A return value of n or more means the output was cut short, and a negative return means an encoding error.

The counted functions prevent the overflow only if the programmer passes the right number and handles the truncated case. A filename that was silently cut short is a different security problem, not a solved one. The vulnerability is the missing or wrong size check, not any one function name.

The Bug Is Not Always Obvious

A buffer overflow is easy to see when the input is a string literal longer than the array, as in strcpy(name, "krzyzanowski") into a 10-byte name. In real code it is usually hidden behind a value that comes from somewhere else.

A program that builds a filename by copying an environment variable into a fixed buffer with sprintf is fine until someone sets that variable to a long string. The size that decides whether it overflows is under the attacker’s control and is not visible at the point of the copy, which is why these bugs survive code review and why fuzzing, covered in the defenses section, is useful for finding them.

C and C++ are the languages where this is possible, because they allow direct access to memory and do not check array bounds. A function usually receives only a pointer to the start of an array and cannot tell how long the array is even if it wanted to check.

Memory-managed languages such as Java, C#, Go, and Python check the bounds of every array access and raise an exception on a violation, so a stray write becomes a caught error rather than silent corruption. But C remains the dominant language for operating systems, compilers, interpreters, databases, browsers, and the libraries under all of them, so the vulnerability is not going away by attrition (although there’s a strong push to move code from C to Rust).

Turning an Overflow into Running Code

Overwriting the return address is the beginning of an attack, not the end. What follows describes what an attacker does once a corrupted return address gives them control of where the processor goes next. The treatment is conceptual. The goal is to understand what each technique achieves and what constrains it, not to construct a working exploit. A working exploit would be written with specific machine architectures, operating systems, and compilers in mind.

Overwriting a Flag Instead of the Return Address

The return address is not the only useful target. Consider a function with a buffer and a decision variable:

void vulnerable() {
    char buf[16];
    int authorized = 0;
    gets(buf);
    if (authorized) grant_access();
}

If authorized sits just past the end of buf in memory, an input longer than buf overwrites it before reaching anything else, and a carefully sized input can set it to a nonzero value so the program grants access, without touching the return address at all. Whether authorized really sits there is up to the compiler. C makes no promise about the order of local variables, an optimizing compiler may keep the flag in a register or drop it entirely, and one of the defenses described later deliberately places arrays above the other locals so that an overflow cannot reach them. The same attack on fields of a structure is more reliable, because a structure’s fields are laid out in the order declared.

Here the attacker changes a decision the program makes, which is a data-only attack: the program runs only its own instructions, in the order the programmer wrote them, and still does the wrong thing. Changing the return address is control-flow hijacking. Several of the defenses in the last part stop the second and do nothing about the first.

Shellcode

Shellcode is the term for a short sequence of machine instructions an attacker places into a program’s memory to be run after a control-flow hijack. The name comes from the classic payload, which starts a command shell, giving the attacker an interactive prompt with the program’s privileges. The term now covers any small initial payload, whether it opens a shell, loads a larger second stage, or does something else.

Shellcode is written under constraints that ordinary code is not. Three constraints act on it:

1. Forbidden bytes. When the vulnerable copy treats input as a C string, a null byte ends the copy, so the shellcode cannot contain one. A line-reading function stops at a newline. The instructions have to be chosen or encoded to avoid whatever bytes the input path will not pass.

2. Size. The vulnerability and input path limit how much attacker-controlled code can be placed in usable memory. That space can extend beyond the original buffer, but input-length limits, memory boundaries, and damage to state the program still needs can restrict it. When space is tight, the initial payload must be compact.

3. Position. If the attacker can predict the address the code will sit at, the code can refer to fixed locations. When addresses are randomized, the code has to work regardless of where it lands, which is called position independence. That solves the code’s own problem but not the attacker’s: they still need a return address that jumps to the code.

In that case, shellcode often comes in two stages. A small first stage does just enough to pull in a larger second stage, which carries the real functionality. The first stage fits the cramped space, and the second stage is delivered afterward.

Landing Zones

Injecting code requires pointing the return address at it. Guessing that address exactly is fragile. A landing zone, usually built as a NOP sled, relaxes that requirement. A NOP is an instruction that does nothing and advances to the next instruction. An attacker precedes the shellcode with a long sequence of NOPs, so that the return address only has to land somewhere in that sequence. Execution then slides through the harmless instructions until it reaches the real code. A landing zone widens the range of addresses that succeed, which turns an attack that needs a precise address into one that needs only an approximate one.

Reading Past the End of a Buffer

Everything so far has been about writes. A program can also read past the end of a buffer, and an out-of-bounds read is a vulnerability in its own right even though it does not modify anything in memory.

If the program sends the bytes it read back to the attacker, it can reveal a password, a cryptographic key, or a pointer stored beyond the buffer. Reading past the buffer can also crash the program. A leaked pointer reveals a memory address inside the process. An attacker can use that address to locate code or data that address space layout randomization (ASLR) was intended to hide, a defense we examine later.

The best-known example is Heartbleed, disclosed in April 2014. The TLS protocol has a heartbeat message in which one side sends a short payload and a length, and the other side echoes the payload back. OpenSSL copied the number of bytes the sender claimed, without checking it against the number of bytes the sender actually sent. A client that sent one byte and claimed 64,000 got back one byte of its own data followed by up to 64 KB of whatever was next in the server’s memory, which over repeated requests included private keys, session cookies, and users’ passwords. About 17 percent of the web’s TLS servers were affected when the bug was announced, and it was a missing bounds check on a read, with no overflow of anything.

Next: Part 2 - The Heap, Integer Errors, and Format Strings
Lecture 7: Part 1 | Part 2 | Part 3