digital-forensics

How it is built

Architecture overview

A digital forensics system turns a device or a machine's memory into evidence that stands up to challenge. It has to do two things that pull against each other: read as much as possible out of messy, partly deleted, sometimes encrypted data, and prove at every step that nothing was changed and nobody tampered with the result. The workflow is fixed in its order. Acquire a bit-exact image of the disk, phone or RAM and hash it; parse the image through file-system and artifact parsers into structured events and files; normalise and merge all those events onto one timeline; search and triage the content (keywords, signatures, images, text in pictures); and keep a chain of custody that records who touched what, when, and with which hash. The one idea that makes it work is evidence is read-only and hashed, and everything else is derived from it and can be redone: analysis outputs are disposable, the verified image is not. Magnet AXIOM and Cellebrite are the commercial references; Autopsy and The Sleuth Kit, Plaso, Velociraptor and Volatility are the open ones.

Parts:

• Acquisition — bit-exact images of disks, phones and memory, hashed and verified at capture — forensic-disk-imaging (no page yet), Cryptographic hashes (SHA-2, BLAKE3), seekable-compressed-stream (no page yet), aes-xts-disk-encryption (no page yet)
• File-system parsers — read partitions, file systems and their metadata, including deleted entries — inode-file-layout (no page yet), B+ tree, Write-ahead log, file-carving-signatures (no page yet)
• Memory analysis — reconstructs processes, modules and connections from a RAM image — Virtual memory and the TLB, process-tree-tracking (no page yet), executable-formats-elf-pe (no page yet)
• Network evidence — reads packet captures and rebuilds sessions — packet-capture-pcap (no page yet), tcp-stream-reassembly (no page yet)
• Timeline — every artifact becomes an event with a time, a source and a type; all sources are merged — event-schema-normalization (no page yet), k-way-merge (no page yet)
• Content search and triage — keywords, signatures and rules, images, text in images — Inverted index, Aho-Corasick multi-pattern matching, yara-rule-matching (no page yet), Perceptual hashing, image-forensics (no page yet), Optical character recognition (text recognition)
• Case integrity — chain of custody, signed reports, trusted time — Tamper-evident audit log, Digital signatures (Ed25519), trusted-timestamping (no page yet)
• Processing pipeline — runs parsing and indexing jobs over many evidence items — Durable job queue

 suspect disk / phone / RAM / pcap
        │  write blocker or live agent
        ▼
 acquisition ──► image (E01/raw/AFF4) + SHA-256, verified twice ──► evidence store (read-only)
        │                                                              │
        │ chain-of-custody log (hash-chained, signed, timestamped)     ▼
        │                                              ┌────────────────────────────┐
        │                                              │ job queue: parsers per item │
        │                                              │  fs ─ registry ─ browser ─  │
        │                                              │  logs ─ memory ─ pcap ─ carve│
        │                                              └──────────────┬─────────────┘
        │                                                  events + files + text
        ▼                                                             ▼
 case DB (B+ tree / SQLite, WAL)                      timeline (k-way merge) ─ full-text index
        ▲                                                             │ rules, hashes, YARA, OCR
        └──────────────── analyst UI, reports (signed) ◄──────────────┘

How the parts fit together

Acquisition

forensic-disk-imaging (no page yet) copies every sector of the source, including unallocated space and slack, because deleted data lives there. The source is protected from writes (a hardware or software write blocker), the image is hashed while it is written, and the source is hashed again afterwards; the two hashes must match, and the hash is recorded in the case file. SHA-256 is the safe default; older tools record MD5 and SHA-1, which are broken against deliberate collisions but still catch accidental corruption, so a second, stronger hash is added.

The image is large, so formats compress it in a way that keeps random access: the Expert Witness (E01) format stores fixed-size chunks, each compressed and checksummed separately, so a parser can read any sector without decompressing the file from the start. That is the contract seekable-compressed-stream (no page yet) names: a compressed container with an index from offset to chunk. The libewf library implements it; AFF4 is the open alternative.

Two other acquisition cases change the design. Live acquisition of a running machine follows the order of volatility of RFC 3227 (registers and cache, then memory, then disk) because RAM holds keys, connections and processes that are lost at power-off. Encrypted volumes are an acquisition problem: a disk protected by BitLocker or FileVault is ciphertext. If the key (or a recovery key) is available, parsers decrypt on the fly by implementing the volume's mode (XTS-AES, aes-xts-disk-encryption, no page yet; the full-disk-encryption system page shows the key hierarchy); otherwise memory acquired from the live machine is the best chance to recover the key, which is why RAM comes first.

File-system parsers

A parser turns sectors into a directory tree, and the interesting part is what the file system keeps beyond the tree. inode-file-layout (no page yet) is the common concept: a per-file record (NTFS master file table entry, ext4 inode, APFS record) that points to the data extents and holds timestamps, names and flags. A deleted file often keeps its record and extents until reused, so recovering it means reading the record that the tree no longer reaches. NTFS directories are indexes in a B-tree family structure (B+ tree is the concept), and stale index entries and slack in those nodes are a classic source of old file names. NTFS also keeps timestamps twice, in its standard-information and file-name attributes, and a mismatch between the two is a known sign of timestamp manipulation; times are stored as 64-bit counts of 100 ns intervals since 1601 (Windows FILETIME), a detail every parser must get right.

Journals and logs are evidence in their own right. File systems and databases use a Write-ahead log to survive crashes, and what remains of it records recent changes that the main structure has already overwritten: the NTFS journal, a database's WAL file, an ext4 journal. Parsing the log, not only the live data, recovers recent deletions and edits.

When metadata is gone, file-carving-signatures (no page yet) works from content: scan unallocated space for the known magic bytes at the start of a file type (a JPEG begins with FF D8 FF, a PNG with an 8-byte signature, a PDF with %PDF-, a ZIP with PK), then cut at the footer or the format's own length field. Scanning for hundreds of signatures in one pass over a terabyte is the job of Aho-Corasick multi-pattern matching. Carved files lose names and timestamps, and fragmented files are carved wrong, so carving output is marked lower confidence than metadata-based recovery. Open tools include TestDisk's PhotoRec and bulk_extractor.

Memory analysis

A RAM image is a flat file of physical pages. Everything an analyst wants (process list, loaded modules, network connections, command lines) lives in kernel data structures reachable only through the virtual address space: the analyzer must walk page tables to turn kernel virtual addresses into offsets in the image, which is the same translation a CPU does (Virtual memory and the TLB). Volatility needs a symbol table that matches the exact OS build to know field offsets, then walks the process list; it can also scan memory for process structures by their signatures, which finds processes a rootkit unlinked from the list (a technique to expect when the two views disagree).

process-tree-tracking (no page yet) turns what is recovered into the question investigators ask: which process started which, with what command line, and in what order. The same structure, built from event logs or an endpoint agent's records instead of RAM, lets live collection tools like Velociraptor reconstruct a parent-child chain after the fact. Recovered binaries are analysed as executable-formats-elf-pe (no page yet): headers, sections, imports and signatures tell the analyst what the file is and let it be matched against rules. Memory parsers read untrusted structures from a possibly hostile machine, so they bound every pointer chase.

Network evidence

Packet captures (packet-capture-pcap, no page yet) are evidence of what crossed the wire. tcp-stream-reassembly (no page yet) rebuilds the byte streams from out-of-order, duplicated and lost segments so that files, web sessions and mail can be extracted. Two reassembly policies that disagree, in what an endpoint would accept for overlapping segments, produce different streams; a forensic tool states which policy it used.

Timeline

A super-timeline merges every timestamp from every source: file-system times, registry keys, browser history, event logs, application databases. Each parser emits events in a common shape (event-schema-normalization, no page yet): a time, a time type (created, modified, accessed, executed), a source, a description and a pointer back to the evidence. Normalisation also fixes time: each source has its own zone and clock rules, some store local time and some UTC, and the clock of the machine may have been wrong; events keep the original value and a normalised UTC value.

Each parser produces its events in time order, so building the timeline is k-way-merge (no page yet) over k sorted streams with a heap, which is streaming and needs no sort of the whole data set. Plaso (log2timeline) follows this: parsers write events to a store and a second step sorts and filters; Timesketch is a collaborative front end for such timelines.

Content search and triage

Analysts rarely read an image; they search it. Extracted text goes into an Inverted index so a keyword or phrase query over millions of files returns in a fraction of a second; names, emails and card numbers are found by pattern scans at extraction time. YARA-style rules (yara-rule-matching, no page yet) let an analyst or a threat feed define "strings plus conditions" that flag malware or tools; YARA's scanner uses an Aho-Corasick automaton internally for its strings (source: libyara/ahocorasick.c in the YARA repository).

Images are most of the volume on phones. Perceptual hashing finds near-duplicates and known images despite resizing and recompression, where a cryptographic hash would change on a one-pixel edit. image-forensics (no page yet) asks whether an image was edited or where it came from (error-level analysis, camera sensor noise patterns). Optical character recognition makes screenshots, scans and photographs of documents searchable by pushing their text into the same index.

Case integrity

Everything above is only useful if its provenance can be shown. The Tamper-evident audit log is the chain of custody in machine form: each acquisition, transfer, access and analysis step appends an entry containing the hash of the previous entry, so a later edit or deletion breaks the chain. Digital signatures sign the acquisition record and the final report, binding them to a person or a lab key. trusted-timestamping (no page yet; RFC 3161) lets a third-party time-stamping authority sign "this hash existed at this time", which fixes the date of acquisition independent of the examiner's clock. The case database itself is usually a B+ tree store with a write-ahead log (SQLite is the common choice), and its integrity is a separate question from the evidence's.

Processing pipeline

A real case has dozens of evidence items and every parser is a separate job. A Durable job queue runs them: jobs are idempotent (re-parsing an item gives the same events), retried on failure, and ordered by dependency (decrypt before parse, parse before index). Because output is derived and disposable, a failed job is cheap: delete its output and run it again.

Data and control flow

One disk, from seizure to report:

1. The disk is attached through a write blocker; the examiner records the serial number and photographs the device (custody entry 1).
2. The imager reads sector by sector, writing chunks and hashing; read errors are logged and the bad sector range is recorded, not skipped silently.
3. The source is re-hashed; hashes match; the signed acquisition record is time-stamped and appended to the log.
4. The image is opened read-only; jobs enqueue: partition and file-system parse, registry and browser artifacts, log parsing, carving of unallocated space.
5. Parsers emit events (to the timeline), files (to the content pipeline) and text (to the index); the job queue retries failures.
6. The analyst searches, filters the timeline around the incident time, tags findings, and exports a report that lists every item with its source path and hash.
7. The report is signed and time-stamped; the log records the export.

One memory image:

1. A live acquisition tool dumps RAM first (order of volatility); the image is hashed.
2. The analyzer finds the kernel and builds page-table translation; process list and process-scan results are compared.
3. Suspicious processes get their command lines, modules and connections listed; binaries are carved out and matched against rules.
4. Keys found in memory feed the decryption of volumes acquired later.

Scaling

Limits in the order you hit them:

1. Acquisition I/O. The source device's read speed bounds it; run several imagers in parallel on separate devices and verify while writing, not after.
2. Parse and index throughput. CPU bound and parallel per evidence item and per partition; the job queue is the lever. Index incrementally so analysts can search while jobs run.
3. Storage. Images are large and kept for years; seekable compression and deduplication of known files (by hash against a known-file set) cut both storage and analysis time.
4. Timeline size. Hundreds of millions of events: keep it as sorted, indexed storage, filter by time range first.
5. Analyst attention is the true limit: triage rules, known-good hash sets, near-duplicate grouping and "first look" views matter more than a faster parser.

Failure modes

Real implementations

Sources

• RFC 3227, Guidelines for Evidence Collection and Archiving (order of volatility). RFC
• RFC 3161, Internet X.509 PKI Time-Stamp Protocol (TSP). RFC
• B. Carrier, File System Forensic Analysis, Addison-Wesley, 2005 (file systems and recovery; book, no online link).
• The Sleuth Kit, libewf (E01), Plaso, Velociraptor, Volatility 3 and YARA project documentation (links in the table).
• YARA source, libyara/ahocorasick.c. file