mmap and the page cache

Deep dive

How it works

In: a file, an access pattern (offsets, madvise or fadvise hint) · Out: the bytes, copied by read() or visible in memory via mmap · State: page-cache pages, plus one page-table entry per mapped page · Cost: minor fault order of 0.1-1 µs; major fault order of 50-100 µs on NVMe.

read(fd, buf):  syscall → page cache hit? copy → buf      miss: block on I/O, readahead
p = mmap(fd):   load p[i] → TLB/PTE miss → page fault → page cache hit? map PTE (minor)
                                                     miss: block on I/O (major fault)

1. Readahead: on sequential read() the kernel grows an async readahead window, starting from the device's read_ahead_kb (128 KiB default). posix_fadvise(POSIX_FADV_SEQUENTIAL) doubles it. POSIX_FADV_RANDOM turns it off.
2. mmap faults: a minor fault (page already cached) costs order of 0.1-1 µs. Fault-around maps up to fault_around_bytes (64 KiB) of neighboring cached pages at once. A major fault blocks the thread for the device latency, with read-around of about the readahead size.
3. madvise on mappings: MADV_SEQUENTIAL (aggressive readahead, pages may be freed soon after access), MADV_RANDOM (no readahead), MADV_WILLNEED (start async readahead now), MADV_DONTNEED (drop the pages from this mapping), MADV_POPULATE_READ (Linux 5.14+: prefault and return an error instead of SIGBUS).
4. Writeback: dirty pages are flushed by kernel threads once they are older than dirty_expire_centisecs (3000 = 30 s), checked every dirty_writeback_centisecs (500 = 5 s). Background writeback starts at dirty_background_ratio (10 % of available memory). Writers are throttled at dirty_ratio (20 %). Durability still needs fsync/msync(MS_SYNC). See fsync and direct I/O.

Decision table

Code

A safe read-only mapping wrapper, plus a sequential scan two ways. On an M4 Pro with a warm 1 GiB file, both run at about 9-10 GB/s. Builds on macOS and Linux.

// Cargo.toml: libc = "0.2". Linux and macOS.
use std::fs::File;
use std::io::{self, Read};
use std::os::fd::AsRawFd;

/// Read-only shared mapping of a whole file. Drop unmaps it.
struct Mapped { ptr: *mut libc::c_void, len: usize }

impl Mapped {
    fn open(f: &File, advice: libc::c_int) -> io::Result<Mapped> {
        let len = f.metadata()?.len() as usize;
        if len == 0 { return Ok(Mapped { ptr: std::ptr::null_mut(), len: 0 }); } // mmap(len 0) is EINVAL
        let ptr = unsafe {
            libc::mmap(std::ptr::null_mut(), len, libc::PROT_READ, libc::MAP_SHARED, f.as_raw_fd(), 0)
        };
        if ptr == libc::MAP_FAILED { return Err(io::Error::last_os_error()); }
        // SEQUENTIAL: bigger readahead, pages behind may be dropped early. RANDOM: no readahead.
        unsafe { libc::madvise(ptr, len, advice) };
        Ok(Mapped { ptr, len })
    }
    /// Valid only while nobody truncates the file: touching a page past EOF raises SIGBUS.
    fn bytes(&self) -> &[u8] {
        if self.len == 0 { return &[]; }
        unsafe { std::slice::from_raw_parts(self.ptr as *const u8, self.len) }
    }
}

impl Drop for Mapped {
    fn drop(&mut self) {
        if self.len > 0 { unsafe { libc::munmap(self.ptr, self.len) }; } // TLB shootdown if threads share the mm
    }
}

/// Sequential-access hint for read(): doubles the readahead window on Linux.
fn advise_sequential(f: &File) {
    #[cfg(target_os = "linux")]
    unsafe { libc::posix_fadvise(f.as_raw_fd(), 0, 0, libc::POSIX_FADV_SEQUENTIAL); }
    #[cfg(target_os = "macos")]
    unsafe { libc::fcntl(f.as_raw_fd(), libc::F_RDAHEAD, 1); } // macOS has no posix_fadvise
}

fn checksum(b: &[u8]) -> u64 { b.iter().map(|&x| x as u64).sum() } // vectorizes; chunking-independent

/// read() into one reused buffer: one copy from the page cache, no faults, errors as io::Error.
fn scan_read(path: &str) -> io::Result<u64> {
    let mut f = File::open(path)?;
    advise_sequential(&f);
    let mut buf = vec![0u8; 1 << 20]; // 1 MiB: amortizes the syscall, stays in L2
    let mut sum = 0u64;
    loop {
        let n = f.read(&mut buf)?;
        if n == 0 { return Ok(sum); }
        sum += checksum(&buf[..n]);
    }
}

/// mmap: no copy, but a fault per page (or per fault-around batch), and I/O errors become SIGBUS.
fn scan_mmap(path: &str) -> io::Result<u64> {
    let f = File::open(path)?;
    let m = Mapped::open(&f, libc::MADV_SEQUENTIAL)?;
    Ok(checksum(m.bytes()))
}

Complexity and constants

• read() syscall overhead order of 0.1-1 µs, plus a copy at order of 10 GB/s per core. With 1 MiB buffers the copy dominates.
• mmap: a fault per page or fault-around batch, page-table memory (8 B per 4 KiB page), and TLB pressure. munmap in a threaded process costs a TLB shootdown.
• A major fault on NVMe costs order of 50-100 µs of blocked thread time. On a network volume it costs order of ms. The thread cannot do anything else in that time.

Efficiency

• Cost: read() costs order of 0.1-1 µs plus a copy at order of 10 GB/s per core. An mmap minor fault costs order of 0.1-1 µs, page tables take 8 B per 4 KiB page, and munmap adds a TLB shootdown. A major fault blocks the thread: order of 50-100 µs on NVMe, order of ms on a network volume. Bound: device latency when cold, copy bandwidth when warm.
• Levers:
  • read() with a reused 1 MiB buffer plus POSIX_FADV_SEQUENTIAL — same speed as mmap on warm scans — one-pass scans
  • MADV_RANDOM plus MADV_WILLNEED or MADV_POPULATE_READ — faults leave the hot path (madvise(2)) — random lookups in a file that fits in RAM
  • pread/io_uring with your own cache — async queue depth, no synchronous faults — larger-than-RAM random reads on NVMe
• Naive → standard → state of the art:
• naive: mmap with default readahead for cold random reads · one blocking major fault per page, order of 50-100 µs
• standard: reused-buffer read() for scans; mmap plus prefault for hot data · order of 10 GB/s copy; pointer speed after warmup
• state of the art: O_DIRECT, own buffer pool, io_uring · queue depth of tens of I/Os keeps the drive busy

Implementation notes

• Check residency: mincore(), fincore (util-linux), vmtouch. For cold-cache benchmarks: echo 3 > /proc/sys/vm/drop_caches on Linux, sudo purge on macOS.
• macOS: madvise supports MADV_SEQUENTIAL, MADV_RANDOM, MADV_WILLNEED. There is no posix_fadvise. Use fcntl(F_RDAHEAD), fcntl(F_RDADVISE) and fcntl(F_NOCACHE) (bypass the cache).
• Map read-only with PROT_READ + MAP_SHARED for files you never write. MAP_PRIVATE with writes creates anonymous copy-on-write pages.
• Growing files: mremap (Linux) or map a large range up front and extend the file with ftruncate. Accessing beyond EOF is SIGBUS, even inside the mapping.
• mlock pins mapped pages (no major faults later) at the cost of RLIMIT_MEMLOCK.

Pitfalls

• No write ordering: a mapped page with uncommitted changes may reach disk before the WAL record. mlock prevents eviction, not writeback.
• SIGBUS on I/O errors, disk full on a sparse file, or another process truncating the file. A Rust &[u8] over a mapping is only sound while nobody else modifies or truncates the file.
• Stalls you can't schedule: any load may block for a disk read. With async runtimes this stalls the whole worker thread.
• Eviction you don't control: the kernel's LRU doesn't know your hot set. Large scans through mmap push out the index.
• Scalability: page-table locks, a single kswapd thread per NUMA node and TLB shootdowns limit mmap throughput on multi-SSD setups (Crotty et al.).
• Benchmarking only with a warm page cache. Cold behavior is what users see after a restart.

Variants and alternatives

In production

LMDB maps the whole database read-only and writes with write() by default (MDB_WRITEMAP writes through the map). SQLite has opt-in mmap (PRAGMA mmap_size, default 0). MongoDB dropped its MMAPv1 engine in 4.2. Prometheus TSDB and many search indexes (Lucene's MMapDirectory) mmap immutable files.

Implementations

Sources

• Andrew Crotty, Viktor Leis, Andrew Pavlo, Are You Sure You Want to Use MMAP in Your Database Management System?, CIDR, 2022. paper
• Viktor Leis et al., Virtual-Memory Assisted Buffer Management, SIGMOD, 2023. paper
• Linux man pages: mmap(2), madvise(2), posix_fadvise(2), msync(2). Kernel docs: admin-guide/sysctl/vm. mmap · madvise · posix_fadvise · msync · sysctl/vm
• Howard Chu, MDB: A Memory-Mapped Database and Backend for OpenLDAP, LDAPCon, 2011. paper