flow-monitoring

How it is built

Architecture overview

Flow monitoring answers "who talked to whom, how much, and when" for a network that is too fast to record packet by packet. Routers, switches and host agents summarise traffic into flow records: one record per conversation, keyed by the 5-tuple (source and destination address, ports, protocol) plus interface and a few more fields, carrying packet and byte counts and start and end times. The records are exported over UDP to a collector, which decodes, enriches and stores them; a query layer and a detector sit on top. The idea that makes it affordable is aggregation at every stage: the exporter collapses packets into flows, the collector collapses flows into per-minute rollups by dimension (customer, AS, port), and most dashboards only ever read the rollups. Sampling shrinks the load further, at the price of estimates instead of exact counts. NetFlow, IPFIX and sFlow are the exporter protocols; nfdump, pmacct, goflow2 and Akvorado are the open collectors.

Parts:

• Exporter — watch packets, keep a flow cache or sample packets, send records — ipfix-flow-export (no page yet), packet-sampling (no page yet), ebpf-xdp (no page yet)
• Collector — receive datagrams, decode templates, normalise sampled counts, tag with exporter and time — ipfix-flow-export (no page yet), Fast hash functions, Varint (LEB128)
• Enrichment — add AS number, country, customer, interface name to each record — longest-prefix-match (no page yet)
• Live summaries — top talkers, distinct counts, heavy hitters per minute — Count-min sketch, HyperLogLog, Top-k selection, d-ary heap, hash-aggregation (no page yet)
• Storage — raw flows for forensics, rollups for dashboards — Columnar storage, Rollup aggregation, Time partitioning, dictionary-encoding (no page yet), Delta encoding, Bit packing and PFOR, zone-maps (no page yet)
• Query engine — scan columns fast — Vectorized execution, SIMD programming
• Detection — notice scans, floods and traffic shifts against a baseline — Time-series anomaly detection, STL decomposition, Robust statistics, Exponential smoothing, Isolation forest

 router / switch / host
 ┌────────────────────────┐   sFlow: sampled packet headers + counters
 │ flow cache  (IPFIX/NF9)│── IPFIX / NetFlow v9: template + flow records ──┐  UDP
 │ or 1-in-N sampler      │                                                  ▼
 └────────────────────────┘                                    ┌─────────────────────────┐
                                                                 │ collector (many threads)│
 eBPF/XDP probe on a host ── same record format ───────────────►│ decode · scale by rate  │
                                                                 └───────────┬─────────────┘
                              enrich: IP → ASN/country/customer  ◄───────────┤
                                                                             ▼
                                   ┌────────── message queue / batch ────────────┐
                                   ▼                         ▼                    ▼
                           live sketches/top-k      columnar store          anomaly detector
                           (this minute)            raw + rollups by time    (baseline vs now)
                                   └───────────► query / dashboard / alerts ◄─────┘

Two properties shape every decision: the data is append-only and time-ordered, and it is lossy by nature (UDP export, sampling, caches that time out). The system never promises exact totals, only totals with known error.

How the parts fit together

Exporter

Two exporter designs coexist. In the flow cache design (NetFlow v9, IPFIX) the device keeps a table keyed by the flow key, updates counters per packet, and exports a record when the flow ends or when a timer fires. NetFlow v9 (RFC 3954) defines a flow as inactive after a period without packets and recommends exporting long flows periodically; IPFIX (RFC 7011) is the IETF standardisation of the same template idea. In the sampling design (sFlow v5) the agent does no per-flow bookkeeping: it forwards the header of every N-th packet, plus periodic interface counters, and the collector reconstructs the flows from the samples (sFlow v5 spec). The flow cache gives exact counts for the flows it sees but costs memory and CPU on the device; sFlow is cheap on the device but the collector only sees estimates. Many networks run sampled NetFlow or IPFIX, which combines the two.

Hosts without a hardware exporter use a software probe. An ebpf-xdp (no page yet) program attached at the driver or traffic-control hook sees every packet before the network stack does and can update a flow map in the kernel; a user-space agent reads the map and exports records. This is the contract: the kernel does the per-packet work, user space does the per-flow work, and the export format stays standard.

The key idea in packet-sampling (no page yet) is that one packet in N, chosen by a counter or a random process, gives an unbiased estimate when each observed count is multiplied by N. RFC 7014 describes systematic (every n-th) and random selection; sFlow requires randomisation of the interval. The relative error of an estimate with c samples is about 1/√c, so sampling is accurate for large flows and useless for small ones. It is the right tool for capacity planning, top talkers and DDoS detection and the wrong one for billing individual small customers or finding one specific session.

Collector

The collector is a UDP server. Its contracts with the exporter are narrow but easy to get wrong:

• Templates. NetFlow v9 and IPFIX describe a record's layout in a template sent beforehand, and data records only make sense once the template is known. Templates are scoped by exporter address and observation domain, so the collector keeps a table keyed by those and by template ID. Over UDP the exporter must resend templates periodically (RFC 7011 names templateRefreshTimeout), and a collector that restarts must drop data until the next refresh arrives.
• Loss. IPFIX messages carry a sequence number so a collector can count what it missed; UDP gives nothing back (RFC 7011). See Message sequence numbers.
• Sampling rate. The rate is exported as a field or in options data and must be applied to the counters on arrival, or all later totals are wrong by a factor of N.
• Throughput. A busy exporter sends thousands of datagrams per second, each holding many records. The receive path is one socket per core (SO_REUSEPORT) with a large kernel buffer, a decode that parses lengths and variable-length fields without copying, and a hash of the flow key (Fast hash functions) to route records of one flow to the same worker. Cost per record is the decode and one hash, so the limit is usually the writes downstream.

Enrichment

Raw records carry IP addresses and interface numbers. Operators want customers, ASes, countries, device names. The enrichment step maps each address to its attributes with a longest-prefix-match (no page yet) over a routing table or GeoIP prefix list, and each interface index to a name from SNMP or a configuration database. Prefix lookup is the hot loop, since it runs twice per record; multibit tries and direct-indexed tables (DIR-24-8, Poptrie) make it a few memory accesses. Enrich before storage, because the mapping changes over time and an old record should keep the attributes that were true when it was observed.

Live summaries

A minute of flows can have millions of distinct source IPs, and "who are the top 10?" must be answered while the minute is still open. Exact counting needs memory proportional to the number of distinct keys, so the live layer uses summaries:

• Count-min sketch estimates the byte count of any key in fixed memory, never undercounting; with a Top-k selection kept in a small heap (d-ary heap) it yields the heaviest hitters.
• HyperLogLog counts distinct values (sources per destination, destinations per source) in kilobytes, with an error of about one percent; this is the core of scan and DDoS detection.
• hash-aggregation (no page yet) is the exact alternative for the closed minute: a hash table of flow key to counters, spilled to disk when large. The summaries are mergeable, so each collector worker keeps its own and a merge step combines them per time window.

Storage

Flow data has a shape that suits Columnar storage: hundreds of millions of rows, a dozen to fifty columns, almost always queried by a few columns over a time range. Each column compresses well with a type-specific codec: timestamps with Delta encoding, addresses, ports, AS numbers and interface IDs with dictionary-encoding (no page yet) and Bit packing and PFOR, counters with Varint (LEB128) or bit packing. Time partitioning cuts the table into intervals, so a query over the last hour touches only those parts and retention is dropping a whole partition. zone-maps (no page yet) store min and max per block and let a filter on an address or time skip blocks.

The second piece is Rollup aggregation: a stream that folds raw rows into one mergeable partial row per minute and per chosen dimensions (exporter, interface, AS, port), so a week-long dashboard reads thousands of rows, not billions. The raw table stays for forensics but with a shorter retention. The decision to keep the rollup granularity is made at write time and cannot be reversed: a rollup without the destination port cannot answer a question about ports later.

Query engine

Scans dominate. A Vectorized execution engine processes batches of values per column, keeps data in cache between operators and lets the compiler or hand-written SIMD code compare, filter and sum many values per instruction. Filter first on the time and key columns (zone maps), aggregate second, and only then touch the wider columns.

Detection

Traffic is seasonal (daily, weekly). A fixed threshold either fires every morning or never. The detector keeps a baseline per series (bytes per exporter, flows per destination, distinct sources): Exponential smoothing or STL decomposition to predict the expected value, then a residual scaled by Robust statistics (median and MAD, not mean and standard deviation, which one attack would distort). That is the single-series case, as laid out in Time-series anomaly detection. For many series at once, an Isolation forest on feature vectors per host finds the odd one out. HyperLogLog counts feed the usual symptoms: one source with a huge number of distinct destinations is a scan, many sources to one destination is a flood.

Data and control flow

One flow, end to end:

1. A packet arrives at the exporter; the flow cache looks up its key and increments packet and byte counters, or creates an entry.
2. When the flow ends, goes idle, or hits the active timeout, the exporter packs a record with others into a datagram and sends it to the collector; templates are re-sent on a timer.
3. The collector receives the datagram, finds the template for this exporter and observation domain, decodes the records, multiplies counters by the sampling rate, and stamps the exporter.
4. Enrichment adds AS, country, customer and interface name.
5. Each record goes two ways: into the live sketches of the current minute, and into a batch for the columnar store.
6. The store writes the batch as a part in the current time partition; a background job merges small parts and refreshes the rollups.
7. A query reads the rollup for the chosen window, falling back to the raw table for forensics.

Time goes to: the decode and enrichment (CPU, per record), the write batch (disk, per batch), and query scans (disk bandwidth for cold data, memory bandwidth for warm).

One alert:

1. Every minute, the detector reads the per-series rollups and the HyperLogLog counts.
2. It updates the baseline and computes the scaled residual.
3. If the residual exceeds the threshold for several consecutive minutes, it queries the raw flows for that window to name the top sources, destinations and ports, and sends the alert with that context.

Scaling

In the order each limit bites:

1. Exporter load. Lower the sampling rate (1 in N with larger N), shorten flow-cache timeouts, or export from a dedicated probe. Accuracy drops with N.
2. Collector CPU. More receive threads, one socket per thread, batch decode, pin flows to workers by key hash. Then add collectors; they are stateless except the template cache.
3. Write throughput. Batch inserts, compress per column, partition by time, and move the live view to sketches.
4. Storage size. Rollups plus a short raw retention; coarser dimensions for old data; dictionary and delta encoding before general-purpose compression.
5. Query latency. Zone maps, partition pruning, precomputed rollups, vectorized scans; then more shards.
6. Cardinality. Per-host series explode on big networks; use sketches and top-k instead of one series per address.

Failure modes

Real implementations

Sources

• B. Claise (ed.), Cisco Systems NetFlow Services Export Version 9, RFC 3954. rfc
• B. Claise, B. Trammell, P. Aitken, Specification of the IP Flow Information Export (IPFIX) Protocol, RFC 7011. rfc
• B. Claise, B. Trammell (eds.), Information Model for IP Flow Information Export, RFC 7012. rfc; IANA IPFIX information elements.
• G. Sadasivan et al., Architecture for IP Flow Information Export, RFC 5470. rfc
• S. D'Antonio et al., Flow Selection Techniques, RFC 7014. rfc
• sFlow Version 5, sFlow.org. spec; and P. Phaal et al., RFC 3176, InMon Corporation's sFlow. rfc