continuous-profiling

How it is built

Architecture overview

Continuous profiling is a sampling profiler that is never switched off. An agent on every host interrupts the machine at a low, fixed rate, records the call stack of whatever is running, and ships the counted stacks to a central store. The store keeps them as a time series: each series is identified by a profile type and a set of labels (service, pod, node, version), and each point is a set of stacks with a count. An engineer asks "which code burned the CPU in this service between 14:00 and 14:10, and how does that differ from yesterday?" and gets a flame graph.

The idea that makes it work is that sampling cost follows the sampling rate, not the number of calls, so a profile is cheap enough to leave on (Pyroscope's docs quote about 2-5 % overhead "depending on a few factors"; the OpenTelemetry eBPF profiler states 1 % CPU and 250 MB memory as upper limits in its testing). The cost then moves to the store: stacks repeat heavily, so interning, columnar layout, delta coding and compression decide what a year of fleet-wide profiles costs.

Parts:

• Sampler — fires at a fixed rate per CPU, or every N hardware events, and captures the interrupted state — Sampling profiling with the PMU (no page yet), built on Interrupt handling; Hardware performance counters (no page yet); eBPF (no page yet)
• Unwinder — turns a captured instruction pointer and stack into a list of callers — Stack unwinding (no page yet), built on Debug info (DWARF), calling-conventions-abi (no page yet), executable-formats-elf-pe (no page yet)
• Symbolizer — maps addresses to function, file and line, on the agent or on the server — Symbolication (no page yet), built on Debug info (DWARF), Interval tree, source-maps (no page yet)
• Profile encoder — packs stacks, functions and strings into one compact profile — String interning (no page yet)
• Ingest sampling — drops or thins profiles before storage when volume is too high — Telemetry sampling (no page yet), built on hash-based-sampling (no page yet), reservoir-sampling (no page yet), Token bucket
• Store — keeps series by label and time, cheap to scan and to expire — Columnar storage, built on Bit packing and PFOR, Vectorized execution, dictionary-encoding (no page yet), zone-maps (no page yet); Delta encoding, built on Varint (LEB128); Zstandard (zstd); Time partitioning
• Viewer — merges stacks into a prefix tree and draws it — Flame graphs (no page yet)

 every host                                      central service
┌──────────────────────────────────┐   pprof    ┌───────────────────────────────────────┐
│ kernel: perf event @ N Hz / CPU  │  + labels  │ distributor ─► [ingest sampling]      │
│   └► eBPF program                │──────────► │      │                                │
│        unwind kernel+user stacks │  (10 s     │      ▼                                │
│        count (pid, ustack, kstack)│  batches)  │ writer ─► blocks: labels | time |     │
│ agent: read maps, add labels,    │            │           stacks | values  (columnar) │
│  symbolize kernel, upload        │            │      ▲  symbols (debuginfo store)     │
│  debuginfo of binaries           │            │ query ─► merge stacks ─► flame graph  │
└──────────────────────────────────┘            └───────────────────────────────────────┘

Profiles arrive in two ways. In push mode a whole-system agent sends what it sampled from outside the program; it sees kernel frames and needs no change to the program. In pull mode the server or a collector scrapes an HTTP endpoint that a runtime exposes (Go programs publish pprof endpoints; Parca and Grafana Alloy both scrape them); this sees runtime-specific profiles such as goroutines but needs the program's cooperation.

How the parts fit together

Sampler

The sampler (Sampling profiling with the PMU (no page yet)) decides when to look. On Linux the trigger is a perf event (perf_event_open): either a software clock (PERF_COUNT_SW_CPU_CLOCK, a per-CPU timer) or a hardware counter that overflows after N events such as cycles or cache misses. Counting events instead of time lets a profile say where cache misses happen, not only where time passes; that is the role of Hardware performance counters (no page yet), even when a deployment only uses the timer.

Parca Agent shows the always-on shape. It attaches an eBPF program to the CPU-clock perf event, so the kernel calls it 19 times per second on each logical CPU, whole-system, for user and kernel stacks. Each call runs from the timer interrupt (Interrupt handling). The rate is a prime on purpose: 100 Hz means every 10 ms, a period user code may share, and then a periodic task would look permanently on-CPU. A function seen in k samples has a relative error of order 1/√k, so a rarely-run path needs many samples, and fleet-wide aggregation is how continuous profiling gets them.

Why eBPF (no page yet) and not a signal handler in each process? The program runs in the kernel at the sample point, sees all processes, needs no library inside the target and no restart, and writes only to eBPF maps. The OpenTelemetry eBPF profiler states the same properties: whole-system, nothing loaded into the profiled process. A language-runtime profiler inside the process sees runtime-level facts such as goroutines; the two are complementary and the store accepts both.

Unwinder

Stack unwinding (no page yet) recovers the callers. The cheap way is the frame-pointer chain, a linked list, but compilers often omit frame pointers and system libraries are built without them. The OpenTelemetry eBPF profiler and the eBPF profiling in Grafana Alloy instead unwind native code from the .eh_frame call-frame tables (Debug info (DWARF) describes the table format), which needs neither frame pointers nor full debug info on the host (the tables live in the ELF file, see executable-formats-elf-pe (no page yet); where return addresses sit is set by calling-conventions-abi (no page yet)); both also list support for high-level runtimes (HotSpot JVM, Python, Ruby, Node.js, .NET and others) without reconfiguring or restarting them. Parca Agent's language-support page lists the requirements: ahead-of-time languages need frame pointers or DWARF unwind information; JIT languages need frame pointers plus a perf map or jitdump file.

Unwinding inside the sampler, not offline, matters because copying kilobytes of stack per sample would be far too much data; a short list of addresses is not.

Parca Agent keeps two eBPF maps. One maps a stack ID to the list of addresses. The other counts how often each key (PID, user stack ID, kernel stack ID) was seen. Every 10 seconds the agent reads and clears the counts, so a window of work becomes "stack → count" with no per-sample record.

Symbolizer

Symbolication (no page yet) turns the addresses into names. Doing it all on the agent would need debug info on every host, so Parca splits it:

• Kernel frames are symbolized at once in the agent from /proc/kallsyms, because the kernel's layout changes at run time (loaded eBPF programs, KASLR).
• Application frames stay as addresses plus a mapping (binary, build ID, offset). Resolving an address to a function is a lookup of a point in sorted ranges, which is what an Interval tree answers; the ranges come from Debug info (DWARF) and the symbol tables of the ELF file. The agent uploads the debug info of binaries it sees to the server's debuginfo store, and the server symbolizes asynchronously: by default every 10 seconds it picks up locations that are symbolizable but not yet symbolized. This works for stripped production binaries.
• JIT and interpreted frames cannot be symbolized later, since their addresses are not stable, so the sampler side must resolve them.

Profile encoder

The wire format is built on String interning (no page yet). The pprof protobuf, originally from Google, is the common format: a profile holds sample_type descriptions, samples (each a list of location IDs and values), locations (address, mapping, inlined lines), functions and a string_table where every name is stored once and entry 0 is the empty string. It is gzip-compressed on disk. The OpenTelemetry profiles signal builds on pprof and widens the dictionary idea from strings to other repeated messages, adds resource and scope context, and lets a sample carry a span and trace ID, which is how a profile links to a trace. Runtime pull endpoints and eBPF agents both produce it, so pprof is the contract between sampler and store.

Store

Profiling data is "stack trace with a value", plus time. Parca stores it in two parts, per its storage docs. A meta store (embedded key-value store) holds mappings, functions and locations, each keyed so a function or mapping seen before is stored once. The sample store holds, per sample, labels, timestamp, stack (a list of location IDs) and value, in FrostDB, a columnar embedded database written for this purpose. The data model follows Prometheus: a query is a label selector such as cpu{node="..."}.

Why Columnar storage and not a row store? Within a sorted run the labels repeat on every row (Parca's example collapses 6 x mypod1), timestamps are close together, and stacks recur across windows. Columns exploit all three, and a time-range query skips whole ranges. Parca sorts by the label columns and then by timestamp for this reason. The pieces compose in the usual way:

• Delta encoding turns the sorted timestamp (and ID) columns into small differences; Parquet, the file format Pyroscope uses for its profile tables, defines a delta encoding for integer columns (DELTA_BINARY_PACKED).
• Zstandard (zstd) is the general-purpose second stage; Parquet lists ZSTD as a page compression codec. Why not LZ4? It decodes faster at a worse ratio; profile blocks are kept long and read rarely, so ratio wins.
• Dictionary encoding (dictionary-encoding (no page yet)) in the column layer repeats the interning idea for repeated strings, and Bit packing and PFOR stores the small deltas and dictionary codes in few bits; Varint (LEB128) is the byte-oriented alternative for deltas. Per-block min/max summaries (zone-maps (no page yet)) let a query skip blocks; Vectorized execution scans the rest in batches.
• Time partitioning cuts the data into blocks by time range: Pyroscope's blocks each hold a metadata file with the time range, a TSDB-style index mapping labels to profiles, Parquet tables and a symbol file. A query reads only blocks overlapping its window, and retention drops whole blocks.

Ingest sampling

Telemetry sampling (no page yet) appears a second time on the server side. In Pyroscope v2 the distributor applies per-tenant rules that keep or drop a matching profile by a probabilistic decision, which leaves gaps in the series. Where totals matter (cost views), a dropped profile can be kept as a single totals sample tagged __sampled__="true", with stacks removed, so flame graphs for those points are empty. A hard cap per tenant can be a Token bucket that refills at the allowed rate; hash-based-sampling (no page yet) keeps every profile of a chosen subset of series, and reservoir-sampling (no page yet) keeps a bounded random sample of a stream. There are three independent rates: the profiler's rate, the share of scrape targets profiled, and the share of profiles kept at ingest.

Viewer

Flame graphs (no page yet) are the default view. Stacks are merged into a prefix tree; the width of a frame is its share of samples, vertical position is call depth. Brendan Gregg introduced them to make profiled stacks readable ("The Flame Graph", CACM 2016). Two merges over different windows can be diffed.

Data and control flow

One profile, from CPU tick to flame graph.

1. The kernel fires the perf event on a logical CPU about 19 times per second; the eBPF program reads the current PID, unwinds the user and kernel stacks, and increments the count for (PID, user stack ID, kernel stack ID).
2. Every 10 seconds the agent drains the maps, resets them, and builds a pprof profile from stacks and mappings read from /proc/PID/maps. Kernel addresses are symbolized immediately.
3. The agent attaches labels from metadata discovery (Kubernetes and systemd targets). It uploads debug info for any build ID the server has not seen, then uploads the profile.
4. The distributor validates the profile and may apply ingest sampling. In Pyroscope v2 the client waits until the data is durably in object storage and indexed; median latency is documented as under 500 ms by default.
5. In Parca, the server stores locations and functions once in the meta store and appends the samples to the columnar store, sorted by labels then time. Small segments are compacted into larger blocks; the v2 compaction workers start within a median of 15 seconds.
6. The symbolizer resolves new application locations from the uploaded debug info.
7. A query sends a label selector and a time range. Blocks overlapping the window are located (Pyroscope v2 keeps a metadata index for this), workers read only the needed columns and merge stacks, and the UI shows the flame graph or a diff.

An incident look-back. A latency alert fires from Alerting on Metrics. The engineer diffs the flame graph for the minutes around it against the same window the day before and sees which frames widened. Nothing has to be reproduced.

Scaling

In the order you meet the limits:

1. Agent overhead. The levers are the sampling rate, which profile types are collected, and the kernel-side unwinder. Moving unwinding into eBPF and sending only counts keeps the per-host cost bounded by the number of distinct stacks, not by the sample rate.
2. Cardinality of labels. Every distinct label set is a series. Use few labels (service, version, node, pod) and trace links for per-request views.
3. Ingest volume. Lower the profiler rate, profile a subset of scrape targets, or sample at the distributor. Keep totals-only profiles if cost views need exact sums.
4. Write path. Pyroscope v1 used stateful ingesters with local disks and replicated each series to three by default; v2 writes straight to object storage, so distributor and segment-writer are stateless, with a Raft-replicated metastore as the only stateful part. Objects reach millions per hour, so compaction is on the critical path.
5. Read path. A flame graph over a large window can read gigabytes and needs merging. Levers: time-partitioned blocks, column pruning, parallel query workers, fair per-tenant queues.
6. Retention. Time-partitioned blocks can be dropped whole.

Failure modes

Real implementations

The alternative: profile on demand, on one host, once someone is already investigating (perf record, a language profiler attached to a process). It costs nothing when idle and can run at high resolution, but it misses an incident that has already happened and cannot compare versions or fleets.

Sources

• G. Ren, E. Tune, T. Moseley, Y. Shi, S. Rus, R. Hundt, Google-Wide Profiling: A Continuous Profiling Infrastructure for Data Centers, IEEE Micro 2010. DOI, abstract
• B. Gregg, The Flame Graph, ACM Queue / CACM 59(6), 2016. DOI, flame graphs page
• Parca docs: overview, agent design, storage, symbolization, agent language support
• Grafana Pyroscope docs: v1 architecture, v2 architecture, write-path sampling, continuous profiling, Alloy collector
• OpenTelemetry: profiles signal, eBPF profiler
• pprof profile.proto; Linux perf_event_open(2); Apache Parquet compression and encodings; What is eBPF