Part 1: Protection and Hardware Support for Access Control
Part 2: Discretionary Access Control
Part 3: Mandatory Access Control
Part 4: RBAC, ABAC, and Chinese Wall

One of the most critical responsibilities of an operating system is protection—the control of who is allowed to access which resources and what operations they can perform. Protection is the mechanism that enforces a system’s security policy. A policy defines what is permitted or prohibited; the operating system’s protection mechanisms ensure that these rules are followed.

At its core, protection means mediating all interactions between subjects (active entities such as users or processes) and objects (passive resources such as files, directories, devices, or network connections). The operating system must ensure that only authorized subjects perform operations consistent with the policy and that no subject can interfere with others in ways that compromise confidentiality, integrity, or availability.

Historically, the need for protection emerged gradually. The earliest computers were single-user batch systems. A program and its data were physically brought to the computer—often on punched cards or magnetic tape. Since only one program ran at a time, the idea of one user interfering with another simply did not exist. The system operator acted as the only gatekeeper.

When disks appeared, data became persistent and accessible even when users were absent. Now, users could store information that other users might read or modify later. This created the need for privacy and integrity controls. As multi-user and timesharing systems emerged in the 1960s and 1970s, the problem intensified: dozens or hundreds of users could share the same processor and disks simultaneously. The operating system had to ensure that one user’s process could not read, overwrite, or crash another’s. Access control became a fundamental requirement.

The personal computing era briefly obscured this need. PCs were designed for one user who had complete control over the system. But as software grew more complex and connectivity expanded, the trust model collapsed. Installing a program became an act of faith that it would not delete files, spy on the user, or transmit private data elsewhere. Today’s systems—desktops, mobile devices, and cloud services—again host code from multiple users and sources. Enforcing access control is as critical as it was on mainframes, only now on a far larger and more networked scale.

Protection and Policy Enforcement

To implement protection, the operating system must do three things:

Identify subjects — every process must be linked to a user identity.
Authenticate that identity — typically by verifying credentials such as passwords, tokens, or certificates.
Authorize actions — check each access attempt against the policy that defines what the user or process is allowed to do.

The user’s identity (or process identity) is usually represented by a numeric user ID. Each running process carries this ID in its context. A web server, database, or background system process may each have their own unique IDs. Once authenticated, every operation that the process performs is bound by the rights assigned to that ID. These rights define which resources the process can read, modify, or execute.

The operating system must protect not only user data but also itself. A compromised process must not be able to overwrite kernel code, disable interrupts, or monopolize CPU time. Hardware features such as privileged processor modes, memory management units (MMUs), and timers provide the foundation on which software protection mechanisms are built.

In this context, access control focuses on one core question: given a subject and an object, should a particular operation be allowed?

Now let's explore the mechanisms for representing, enforcing, and reasoning about such decisions.

Modeling Protection: The Access Control Matrix

Access control requires a way to represent who can do what. By the early 1970s, researchers realized that ad hoc permission checks scattered throughout code made reasoning about security impossible. Butler Lampson, one of the pioneers of computer security, proposed the Access Control Matrix as a conceptual model to describe protection in an operating system.

The access control matrix provides a structured way to capture all permissions in a system. Each row represents a subject (or domain of authority); each column represents an object; and each cell contains the rights that the subject has over that object.

	File A	File B	Printer
Alice	read, write	read	print
Bob	read	—	—
Process X	—	write	—

A cell entry such as “read, write” for Alice and File A means Alice can open and modify that file. Empty cells mean no access. This abstraction captures the essence of protection: every authorized operation corresponds to one entry in the matrix.

In practice, the operating system maintains a protection domain for each process. A protection domain defines the set of objects the process can access and the rights it has over each. The operating system checks each attempted operation against this domain.

Extended Rights and Control

Real systems need more than simple read or write permissions. The matrix model can represent richer control mechanisms:

Domain transfer: the right to switch to another protection domain (used by login programs or trusted services).
Copy right (delegation): permission to grant a right to another subject.
Ownership: the right to modify permissions on an object.
Domain control: the right to alter the rights of another domain.

These extensions demonstrate how the model can represent privilege escalation, delegation, and administration—concepts commonly found in real-world systems.

Implementation Challenges

While conceptually elegant, a literal access matrix is infeasible for implementation. Even a modest system with 1,000 users and 1 million files would require a billion entries. Most of the matrix is empty, and storing or searching it would be hopelessly inefficient. Systems therefore adopt one of two practical representations derived from the matrix:

Access Control Lists (ACLs) — Store each column with the object. Each file, device, or resource maintains a list of subjects and their allowed operations. ACLs make it easy to answer, “who can access this file?”
Capability Lists — Store each row with the subject. Each process or user holds unforgeable tokens (capabilities) that identify which objects it can access and how. Capabilities make it easy to answer, “what can this process do?”

Both are partial projections of the same conceptual matrix. The choice between them reflects system design priorities: ACLs centralize authority at the object, while capabilities decentralize it to subjects.

Example Applications

Modern systems use hybrids of these ideas. UNIX and Windows file systems store per-object ACLs. Distributed systems such as Google’s Fuchsia or the seL4 microkernel rely on capabilities for fine-grained confinement. Network authentication systems like Kerberos and OAuth issue capability-like tokens that define what a client may do, without consulting a global matrix.

Limitations

The access control matrix gives us a foundation for reasoning about protection but not a practical way to manage it.

It does not describe how rights should be created, transferred, or revoked.
It assumes static relationships, while real systems constantly add and remove users and resources.
It provides no mechanism to enforce higher-level policies, such as confidentiality or integrity, that go beyond simple permissions.

Nevertheless, the matrix remains a cornerstone of access control theory. Every modern system’s permissions structure—from UNIX file modes to cloud IAM policies—can be viewed as a projection or simplification of this model.

Discretionary Access Control

Once the access control matrix gave us a formal way to describe who could access what, operating systems needed practical mechanisms for enforcement. The earliest multi-user systems, like MIT’s CTSS in the early 1960s and later UNIX at Bell Labs, chose a user-friendly model that allowed individuals to manage access to their own files. This became known as Discretionary Access Control (DAC) because control is at the discretion of the resource owner.

In a DAC system, the user who creates a resource, such as a file, becomes its owner. The owner can decide which other users or groups may access that file and what operations they are allowed to perform. The operating system enforces these rules, but it does not dictate them.

This ownership model maps naturally to the access control matrix:

Each object’s column is stored as an Access Control List (ACL) attached to that object.
The owner may modify this list to grant or revoke rights.

DAC provides a direct and intuitive model for end-users but introduces trust and consistency problems at scale, as we’ll see later.

The UNIX (POSIX) Permission Model

The UNIX permission model is one of the earliest and most durable implementations of DAC. It originated in the early 1970s with the first versions of UNIX at Bell Labs and was standardized later through POSIX.

UNIX, POSIX, and Linux
UNIX was originally developed at Bell Labs from 1969 through the late 1980s. Over time, many variations emerged: various versions of BSD at Berkeley, System V from AT&T, and derivatives such as IBM's AIX, HP-UX, and Sun's Solaris. UNIX popularized certain elements that became core features across many operating systems, including a hierarchical file system, treating everything as a file (devices and inter-process communications), and the concept of pipes in the shell to use the output of one command as the input to another.
POSIX (Portable Operating System Interface) is an IEEE standard that unified these variants. It specifies system calls, shell utilities, and semantics for file systems, including the permission model. A POSIX-compliant system behaves like UNIX but need not be derived from it. macOS is an example of a POSIX-compliant system.
Linux is a modern, open source reimplementation of the UNIX ideas. It follows the POSIX standard but was built independently, starting in 1991. Linux is not UNIX in trademark terms, but functionally it is a POSIX UNIX-like system. Android is built on a modified Linux kernel. Microsoft's Windows Subsystem for Linux (WSL), allows you to run a Linux environment on Windows.

Design Origins

The original UNIX designers sought a compact, efficient representation of DAC permissions that could fit within the limited storage of a file system inode. Each inode contains metadata about a file—its owner, group, size, timestamps, and access rights. The permissions mechanism needed to be fast to check, easy to display, and small enough to fit in just a few bytes.

The resulting scheme—user, group, and other—became the foundation of the POSIX standard and persists today across Linux, macOS, and BSD systems.

Structure of Permissions

Every file is associated with:

An owner (user ID)
A group (group ID)
A set of permission bits organized as:
```
rwxrwxrwx
```

Each triplet corresponds to user, group, and others (everyone else), and each letter represents:

r – permission to read the file,
w – permission to write or modify it,
x – permission to execute it (for programs) or search it (for directories).

For example:

$ ls -l /bin/ls
-rwxr-xr-x  1 root  wheel  154624 Sep 25 03:03 /bin/ls

This line shows:

The file is owned by root
The owning group is wheel
The owner has read, write, and execute permission
The group and others have read and execute permission.

The leading character (-) indicates the type of file. It can be a regular file (-), a directory (d), a symbolic link (l), or a device file (b or c).

Access Evaluation

When a process attempts to open a file, the kernel checks permissions once, when the file is opened. The checks occur in a strict order:

If the user is the file’s owner, only the owner permissions apply.
Else, if the user belongs to the file’s group, only the group permissions apply.
Otherwise, the “other” permissions apply.

This order ensures predictable behavior but sometimes surprises newcomers:

For instance, even if you belong to a group with read permission, you may still be denied access if you are also the file’s owner and your owner bits exclude read permission.
Because permission checks occur only when a file is opened, a program can continue to use the file even if its permissions are later changed. This optimization improves performance but can lead to vulnerabilities if a program creates a file, changes its permissions afterward, and assumes it is now protected.

Execute Permissions

Execution rights are separate from read rights. A file may be executable but unreadable:

$ ls -l secretprog
-r-x--x--x 1 root staff 8492 Feb 4 19:22 secretprog
$ cat secretprog
cat: secretprog: Permission denied

You cannot view or copy the file’s contents, but the kernel can still load and execute it. This protects proprietary binaries from inspection.

Directory Permissions

Directories in UNIX are special files that map filenames to inode numbers. Their permissions are interpreted differently:

Permission	Meaning for Directories
r (read)	List the directory’s contents.
w (write)	Create, delete, or rename entries within the directory.
x (execute)	Traverse the directory; needed to open or access files inside.

The execute bit (x) is essential: without it, you cannot enter a directory, even if you know the name of a file inside. Conversely, if you have write permission on a directory, you can delete or rename files within it, even if you do not have write permission on those files themselves, because deletion modifies the directory entry, not the file.

A directory that has write but not read permission lets a user add or remove files without being able to list them. A directory that has read but not execute permission allows listing names but not accessing them.

Race Conditions and the umask

A common pitfall occurs when a program creates a file and then tightens its permissions afterward. Between those two steps, another process might open the file and retain access indefinitely, since permission checks occur only when the file is opened. This creates a race condition.

To avoid this, UNIX systems use a file creation mask, or umask, which removes specified permission bits at the time of file creation. The umask defines which bits are turned off by default. A typical shell command is:

$ umask 022

The three digits correspond to the bits for user, group, and others. The binary value of each number identifies the rwx bits. In this example, 2 is 10 in binary, and hence affects the write (w) permission.

This example disables write permission for group and others. If a program creates a file with mode 0666 (read/write for all), the umask removes those bits, resulting in mode 0644 (read/write for owner, read-only for others).

In most systems, the default umask is 022, which prevents users from creating files writable by others. The umask is a simple but effective safeguard against careless programs that might otherwise create world-writable files.

Changing Permissions and Ownership

Users can inspect or modify permissions and ownership with [hopefully] familiar commands:

chmod – change permissions

$ chmod u=rw,g=r,o= file.txt

This gives the owner (u for "user") read and write (rw) permission, the group (g) read-only (r) permission, and no access to others (o).

Permissions can also be specified numerically. Each bit represents read (4), write (2), and execute (1). For example:

$ chmod 640 file.txt

sets permissions to rw-r-----.

chgrp – change group ownership

$ chgrp staff file.txt

transfers ownership to another user. Only the superuser can change the file’s owner because ownership conveys the ability to modify permissions and delete the file.

User and Group Identity

User and group IDs are defined in /etc/passwd and /etc/group, respectively:

wheel:x:0:root
paul:x:1000:1000:Paul K:/home/paul:/bin/bash

Each process inherits two key identifiers:

Real user ID (ruid): identifies the account that started the process.
Effective user ID (euid): determines what the process is allowed to access.

A process also carries a group ID (gid) and may belong to supplementary groups that extend access privileges.
When you log in, the login program (running as root) authenticates your credentials, then uses system calls to configure your environment:

Reads /etc/shadow to verify your password.
Reads /etc/passwd to find your UID, GID, home directory, and shell.
Changes to your home directory with chdir().
Sets your group ID with setgid(gid).
Sets your user ID with setuid(uid).
Executes your shell with execve().

After these steps, your shell runs with your real and effective IDs set to your own account.

Privileged Execution and setuid

Some programs must perform tasks that require more authority than the invoking user possesses—such as changing passwords, mounting file systems, or opening privileged network ports. UNIX supports this through the setuid and setgid permission bits.

If a file has the setuid bit set, a process executing that program runs with the file owner’s user ID as its effective user ID, rather than the invoker’s. For example, /usr/bin/passwd runs with effective UID root so it can update the system password database. The kernel maintains both the real and effective user IDs to track accountability.

Setuid and setgid provide controlled privilege elevation but must be used carefully. Any vulnerability in a setuid program can be exploited for full system compromise. Modern systems minimize risk by replacing setuid binaries with privilege-separating daemons, code signing, or sandboxing mechanisms.

Principle of Least Privilege

A cornerstone of secure system design is the principle of least privilege:

Each user, process, or component should operate with only the permissions necessary to perform its task ... and no more.

This principle limits damage from accidents and attacks. For example:

A web server should not have permission to modify system files.
A user should not be able to kill other users’ processes.
System utilities should be carefully restricted to the minimal privileges required.

Violating this principle increases the potential attack surface.

Privilege Separation and Effective IDs

Many programs need short bursts of privilege. For example, a program may need to open a protected configuration file or bind to a low-numbered port, but can then continue safely with normal permissions. The privilege separation pattern divides a program into:

A privileged component that performs limited sensitive tasks.
An unprivileged component that handles all other work.

UNIX facilitates this design with real and effective user IDs:

ruid – real user ID: the identity of the user who started the process)
euid – effective user ID: the identity used by the kernel when checking permissions.

Normally, ruid = euid. When a process executes a setuid binary, the euid becomes that of the file’s owner. A privileged process can temporarily drop its elevated rights with seteuid(getuid()), then restore them when needed.

A secure implementation often follows this sequence:

Start as a privileged process.
Create a communication channel (for example, a pipe).
Call fork() to create a child.
The child lowers privileges with seteuid(getuid()) and performs untrusted operations.
The parent retains elevated rights only for necessary tasks.

This separation confines damage: if the unprivileged process is compromised, it cannot alter system resources.

Setuid and Security Risks

The setuid and setgid mechanisms are useful but dangerous. If an executable file has the setuid bit set, any user who runs that program temporarily gains the file owner’s privileges. While this enables necessary functions like password changes, it also creates high-value attack targets.

To reduce risk:

Keep setuid programs to a minimum.
Avoid setuid to root unless absolutely necessary.
Write such programs defensively with rigorous input validation.
Replace them where possible with privilege-separating services or sandboxed daemons.

Modern UNIX and Linux distributions increasingly avoid setuid binaries, preferring fine-grained privilege frameworks and kernel-level access controls.

The Evolution of Permissions on Windows

Early versions of DOS and Windows had essentially no access control. The system assumed a single user and trusted all software. Any program could read, modify, or delete any file. This was acceptable in the 1980s when PCs were isolated, but it became untenable in networked environments.

The introduction of Windows NT in the 1990s changed this completely. NT introduced a kernel with a full security reference monitor, a notion of user accounts, and an access control model built on Access Control Lists (ACLs). Each file, registry key, or device could specify which users or groups were allowed or denied specific rights, such as reading, writing, deleting, or changing ownership.

This model was more expressive and closer to the access control matrix concept than UNIX’s compact bitmask. It also integrated naturally with centralized authentication through Active Directory, which allowed organizations to define domain-wide policies. Over time, NT’s ACL model became the foundation for all modern Windows systems.

This difference in lineage—UNIX’s simplicity versus Windows’s enterprise-level granularity—illustrates two design philosophies: one optimized for personal ownership and collaboration, the other for centralized administration and auditing.

The Evolution of Permissions on Windows

Early versions of DOS and Windows had no meaningful access control. The system assumed a single user and trusted all software. Any program could read, modify, or delete any file. This was acceptable when personal computers were isolated, but became untenable once systems were networked and used in multi-user environments.

The introduction of Windows NT in the 1990s changed this completely. NT introduced a kernel with a security reference monitor, a formal notion of user accounts, and an access control system based on Access Control Lists (ACLs). Each securable object—files, registry keys, devices, or processes—could define exactly which users or groups were allowed or denied specific rights such as reading, writing, deleting, or changing ownership.

This model was far more expressive than UNIX’s simple owner-group-other bitmask. It resembled a full access control matrix, where each object maintains a column listing who can perform which actions. The ACL model also integrated naturally with Active Directory (AD), Microsoft’s centralized authentication and directory service. AD allowed domain administrators to manage users, groups, and policies across an organization, creating a unified enterprise security model.

Discretionary and System ACLs

Every securable object in Windows contains two optional lists:

A Discretionary Access Control List (DACL) that determines who can access the object.
A System Access Control List (SACL) that defines what events should be logged for auditing.

A DACL is composed of Access Control Entries (ACEs). Each ACE associates a Security Identifier (SID)—the unique identifier for a user or group—with a set of allowed or denied permissions such as read, write, execute, delete, change permissions, or take ownership.

For example:

Allow user Alice read and write access.
Deny group Guests any access.

When a process requests access to an object, the Windows kernel compares the process’s access token—which contains its user and group SIDs and privileges—to the object’s DACL. Access is granted only if no ACE explicitly denies the request and at least one ACE allows it.

This system supports both allow and deny rules, evaluated in order, which provides fine-grained control but can become complex to administer.

Centralized Administration and Domain Integration

Windows integrates its ACL model with Active Directory to support centralized policy and trust relationships across machines. Administrators can define group memberships, propagate permissions, and enforce domain-wide policies through Group Policy Objects (GPOs).

Although object owners retain discretion over their files, domain administrators can restrict what owners may change, blurring the line between discretionary and mandatory access control. This allows enterprises to maintain consistent, auditable security policies while preserving local flexibility.

Design Philosophy: Simplicity vs. Granularity

UNIX permissions were designed for personal ownership and collaborative simplicity. Windows permissions evolved for enterprise administration, auditing, and fine-grained delegation. The UNIX model is compact and predictable; the Windows model is detailed and expressive but requires careful management.

Both systems embody their environments: UNIX optimized for local control and trust, Windows for large-scale, policy-driven networks.

Linux Extended Attributes and Full ACLs

Traditional UNIX permissions (owner, group, and other) work well for small environments but fail to express complex sharing requirements. A research group might want to give different combinations of access rights to several collaborators without creating separate groups for each combination.

This led to the introduction of extended ACLs, standardized in POSIX.1e and widely implemented in modern Linux file systems such as ext4, XFS, and btrfs.

Access Control Entries (ACEs)

An extended ACL contains a list of Access Control Entries (ACEs), each specifying a user or group and a set of permissions. Here's a simplified example:

user::rw-
user:alice:r--
group::r--
group:students:rw-
mask::rw-
other::---

Entries are evaluated in order. The mask defines the maximum effective permissions for all named users and groups, acting as a safety limiter.

Inheritance flags can ensure that new files created in a directory automatically receive default permissions.

Storage and Implementation

Access control lists and other extended attributes do not fit within the fixed-size inode structure, so file systems store them in extended attribute blocks. The kernel retrieves these blocks when checking access permissions. On Linux, the getfacl and setfacl utilities allow inspection and modification of ACLs.

Evaluation Order

When a process opens a file, Linux evaluates permissions in this order:

If the process’s user ID matches the file’s owner, use the owner’s permissions.
Otherwise, if the user is in the file’s group, use the group permissions.
If additional ACL entries exist, search for one matching the user or group.
If no entry applies, fall back to the “other” permissions.

This preserves compatibility with traditional UNIX semantics while adding flexibility.

Practical Advantages

Extended ACLs allow administrators and collaborative teams to define per-user access without proliferating groups. They also support fine-grained control over directories; for example, granting read-only access to some files while allowing write access to others within the same project directory.

Limitations

ACLs increase administrative complexity. It can be hard to visualize who can access what, especially when multiple overlapping entries and masks exist. File managers and system utilities must provide good interfaces to inspect and audit ACLs; otherwise, misconfigurations become common.

Observations on DAC Systems

Discretionary access control works well when trust and cooperation are assumed, but it has intrinsic weaknesses:

Data propagation: once a user reads data, nothing prevents them from copying or redistributing it.
Trojan horse attacks: a malicious program executed by a user inherits that user’s rights and can misuse them: programs run with the authority of the user running them.
Inconsistent policy enforcement: each user controls their own objects; administrators cannot enforce global rules easily.

These limitations led to the development of Mandatory Access Control (MAC) models, where the system, rather than the user, dictates policy enforcement. We’ll turn to that next.

Access Control - Part 2

Protection and Policy Enforcement

Modeling Protection: The Access Control Matrix

Extended Rights and Control

Implementation Challenges

Example Applications

Limitations

Discretionary Access Control

The UNIX (POSIX) Permission Model

Design Origins

Structure of Permissions

Access Evaluation

Execute Permissions

Directory Permissions

Race Conditions and the umask

Changing Permissions and Ownership

chmod – change permissions

chgrp – change group ownership

User and Group Identity

Privileged Execution and setuid

Principle of Least Privilege

Privilege Separation and Effective IDs

Setuid and Security Risks

The Evolution of Permissions on Windows

The Evolution of Permissions on Windows

Discretionary and System ACLs

Centralized Administration and Domain Integration

Design Philosophy: Simplicity vs. Granularity

Linux Extended Attributes and Full ACLs

Access Control Entries (ACEs)

Storage and Implementation

Evaluation Order

Practical Advantages

Limitations

Observations on DAC Systems

Next: Part 3: Mandatory Access Control