fsync, O_DSYNC and O_DIRECT

Deep dive

How it works

In: a file descriptor with written data (and the directory entry for new files) · Out: return once data is on stable media, or an error · State: dirty page-cache pages and the device's volatile write cache · Cost: one device flush; order of 10-50 µs on drives with power-loss protection, about 4 ms for F_FULLFSYNC on an Apple SSD.

On Linux the cache flush happens only if the device reports a volatile write cache (/sys/block/<dev>/queue/write_cache = write back). Drives with power-loss protection report write through, and the flush is skipped. That is why their fsync is order of 10-50 µs.

Recipes

1. New or replaced file: write tmp → fdatasync(tmp) → rename(tmp, name) → fsync(dir).
2. Append-only log: preallocate and pre-zero the file, then pwrite + fdatasync (or O_DSYNC). Overwriting already-allocated blocks needs no metadata journaling, so it is a single flush. Use group commit to share each fsync among many writers.
3. Created a new log segment: fsync the segment, then fsync the directory once.

Decision table

Code

Durable sync on both OSes, crash-atomic replace, and an O_DIRECT|O_DSYNC log writer for Linux. Tested on macOS, type-checked for Linux.

// Cargo.toml: libc = "0.2". Linux and macOS.
use std::alloc::{alloc_zeroed, dealloc, Layout};
use std::fs::{self, File, OpenOptions};
use std::io::{self, Write};
use std::os::fd::AsRawFd;
use std::path::Path;

fn check(r: libc::c_int) -> io::Result<()> {
    if r == -1 { Err(io::Error::last_os_error()) } else { Ok(()) }
}

/// Data plus the metadata needed to read it back, through the drive's volatile cache.
/// Linux: fdatasync, which ends in a device cache flush (or FUA) if the cache is write-back.
/// macOS: fsync only hands data to the drive; F_FULLFSYNC also flushes the drive's cache.
/// Rust's File::sync_data does the same on both. Never retry after an error: crash and recover.
pub fn sync_data(f: &File) -> io::Result<()> {
    #[cfg(target_vendor = "apple")]
    return check(unsafe { libc::fcntl(f.as_raw_fd(), libc::F_FULLFSYNC) });
    #[cfg(not(target_vendor = "apple"))]
    return check(unsafe { libc::fdatasync(f.as_raw_fd()) });
}

/// Directory entries (create, rename, unlink) are durable only after syncing the directory.
pub fn sync_dir(dir: &Path) -> io::Result<()> {
    let d = File::open(dir)?;
    #[cfg(target_vendor = "apple")]
    return check(unsafe { libc::fcntl(d.as_raw_fd(), libc::F_FULLFSYNC) });
    #[cfg(not(target_vendor = "apple"))]
    return check(unsafe { libc::fsync(d.as_raw_fd()) });
}

/// Crash-atomic replace: after a crash, `name` holds the old or the new bytes, never a mix.
pub fn atomic_replace(dir: &Path, name: &str, data: &[u8]) -> io::Result<()> {
    let tmp = dir.join(format!(".{name}.tmp"));
    let mut f = OpenOptions::new().write(true).create(true).truncate(true).open(&tmp)?;
    f.write_all(data)?;
    sync_data(&f)?;                    // 1. contents on stable media before any name points to them
    fs::rename(&tmp, dir.join(name))?; // 2. atomic within one filesystem
    sync_dir(dir)                      // 3. the rename itself survives a crash
}

/// Block-aligned buffer for O_DIRECT: address, length and file offset must be multiples
/// of the device's logical block size (statx STATX_DIOALIGN tells; 4096 is safe).
pub struct AlignedBuf { ptr: *mut u8, len: usize }

impl AlignedBuf {
    pub fn new(len: usize) -> AlignedBuf {
        assert!(len > 0 && len % 4096 == 0);
        let ptr = unsafe { alloc_zeroed(Layout::from_size_align(len, 4096).unwrap()) };
        assert!(!ptr.is_null(), "out of memory");
        AlignedBuf { ptr, len }
    }
    pub fn bytes(&self) -> &[u8] { unsafe { std::slice::from_raw_parts(self.ptr, self.len) } }
    pub fn bytes_mut(&mut self) -> &mut [u8] { unsafe { std::slice::from_raw_parts_mut(self.ptr, self.len) } }
}

impl Drop for AlignedBuf {
    fn drop(&mut self) { unsafe { dealloc(self.ptr, Layout::from_size_align(self.len, 4096).unwrap()) } }
}

/// Log file that bypasses the page cache, and where each write returns only once durable.
/// On a device with FUA support, O_DSYNC writes go out as FUA writes instead of write + flush.
#[cfg(target_os = "linux")]
pub fn open_direct_log(path: &Path, prealloc: u64) -> io::Result<File> {
    use std::os::unix::fs::OpenOptionsExt;
    let f = OpenOptions::new().read(true).write(true).create(true)
        .custom_flags(libc::O_DIRECT | libc::O_DSYNC).open(path)?;
    // Reserve space so appends don't allocate blocks; allocation is metadata that O_DSYNC must also flush.
    check(unsafe { libc::fallocate(f.as_raw_fd(), 0, 0, prealloc as libc::off_t) })?;
    Ok(f)
}

#[cfg(target_os = "linux")]
pub fn write_block(f: &File, offset: u64, block: &AlignedBuf) -> io::Result<()> {
    use std::os::unix::fs::FileExt;
    assert!(offset % 4096 == 0);
    f.write_all_at(block.bytes(), offset) // EINVAL here usually means misalignment
}

Complexity and constants

• Commit rate without batching ≈ 1 / flush latency: 4 ms F_FULLFSYNC → 250 commits/s. 20 µs on PLP NVMe → 50 k/s.
• O_DIRECT reads skip one memcpy and the cache, but every read is a device I/O (order of 10-100 µs on NVMe). Your own cache must replace the page cache.
• Alignment granularity is the logical block size: 512 B or 4 KiB. Misaligned O_DIRECT I/O fails with EINVAL (some filesystems silently fall back to buffered I/O).

Efficiency

• Cost: commit rate without batching is 1 / flush latency: a 4 ms F_FULLFSYNC (measured on a laptop SSD) gives 250 commits/s, a 20 µs flush on power-loss-protected NVMe gives 50 k/s. O_DIRECT saves one memcpy but turns every read into a device I/O, order of 10-100 µs on NVMe. Bound: device flush latency, not bandwidth.
• Levers:
  • group commit — one flush shared by all waiting writers (see group commit) — many concurrent committers
  • fdatasync instead of fsync — skips mtime and other metadata (fsync(2)) — overwrites in a preallocated file
  • O_DIRECT + O_DSYNC, 4 KiB aligned, preallocated — one I/O per commit, often FUA (open(2)) — Linux WAL on NVMe
  • drives with power-loss protection — flush order of 10-50 µs — durability-critical writes
  • F_BARRIERFSYNC — order of 10× cheaper than F_FULLFSYNC — macOS, when only ordering matters
• Naive → standard → state of the art:
• naive: write + F_FULLFSYNC per record · 250 commits/s at 4 ms
• standard: pwrite + fdatasync per group of records · flush latency shared by the group
• state of the art: O_DIRECT+O_DSYNC on a preallocated file, group commit, PLP NVMe · one order-of-20-µs flush per group, 50 k groups/s

Implementation notes

• Query O_DIRECT alignment with statx(..., STATX_DIOALIGN) → stx_dio_mem_align, stx_dio_offset_align (Linux 6.1+). Older kernels: /sys/block/<dev>/queue/logical_block_size.
• io_uring: IORING_OP_FSYNC with IORING_FSYNC_DATASYNC, linked after the write via IOSQE_IO_LINK, makes write + sync one submission (io_uring).
• Linux fsync error reporting (errseq, since 4.13): an error is reported once to each fd open at the time. After a writeback error the pages are marked clean, so the data may exist only in memory or nowhere.
• Torn writes: fsync says nothing about atomicity. A crash during a 16 KiB page write can leave half old and half new data. Use page checksums (CRC32C) plus full-page images in the log, or atomic writes where supported (NVMe AWUPF, Linux RWF_ATOMIC on recent kernels).
• macOS has no O_DIRECT. fcntl(F_NOCACHE, 1) stops caching for that fd.
• Don't mix O_DIRECT and buffered I/O or mmap on the same file region. Cache coherency is best-effort.
• ext4 with the default data=ordered writes data before committing metadata. Don't rely on it for portable code: write the fsyncs explicitly.

Pitfalls

• Forgetting the directory fsync: after a crash the file or rename is gone even though the file itself was fsynced.
• Retrying a failed fsync (PostgreSQL "fsyncgate", 2018). PostgreSQL now PANICs instead.
• Believing O_DIRECT means "on disk". The drive's cache still holds the data.
• Benchmarking fsync on macOS with plain fsync and shipping numbers that are 100× too optimistic.
• Consumer SSDs and some virtual disks that acknowledge flushes early. Test with power-cut rigs if durability really matters.
• fallocate without writing: unwritten extents still need a metadata update on first write. Pre-zero (write once) if you want metadata-free overwrites.

Variants and alternatives

In production

PostgreSQL wal_sync_method (fdatasync default on Linux, open_datasync = O_DSYNC, fsync_writethrough = F_FULLFSYNC on macOS). InnoDB innodb_flush_method=O_DIRECT. SQLite PRAGMA fullfsync (macOS, default off). RocksDB use_direct_io_for_flush_and_compaction, bytes_per_sync (sync_file_range pacing).

Implementations

Sources

• Thanumalayan Sankaranarayana Pillai et al., All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications, OSDI, 2014. paper
• Anthony Rebello et al., Can Applications Recover from fsync Failures?, USENIX ATC, 2020. paper
• Linux man pages: fsync(2), open(2) (O_DIRECT, O_DSYNC), sync_file_range(2), statx(2). fsync · open · sync_file_range · statx
• Apple, fcntl(2) man page (F_FULLFSYNC, F_BARRIERFSYNC, F_NOCACHE). man page