agent-observability

How it is built

Architecture overview

An LLM agent is a loop: the model reasons, calls a tool, reads the result, calls the model again, sometimes hands work to a sub-agent. When it fails (wrong answer, runaway cost, a tool called ten times) the cause sits somewhere in a long chain of prompts, outputs and tool results that no single log line explains. Agent observability records that chain as a trace, a tree of timed steps with their full inputs and outputs. It stores traces cheaply, makes them searchable, rolls them up into cost, latency and quality numbers, scores them automatically with another model, and lets an engineer replay a recorded run, either to reproduce a bug or to try a new prompt or model on real inputs.

The idea that makes it work is that a trace is both a log and a dataset. As a log it answers "what happened in this run". As a dataset it is the set of real inputs, intermediate states and outputs that evaluation, regression tests and replay need. The costs are large and sensitive: prompts and tool outputs are big, repeat a lot, and contain user data and sometimes secrets, so sampling, compression and masking are part of the design from the start.

Parts:

• Instrumentation — SDK or proxy that wraps model calls, tool calls and agent steps into spans with a trace context — agent-tracing (no page yet), Distributed tracing
• Privacy filter — removes personal data and secrets before anything leaves the application — Data masking, Named entity recognition, Secret scanning (no page yet)
• Ingest and sampling — durable buffer plus the decision of what to keep — Partitioned commit log, Telemetry sampling (no page yet)
• Trace store — write-heavy span storage, compressed blobs, analytical columns, full-text search — LSM-tree, Columnar storage, Zstandard (zstd), Inverted index
• Aggregation — cost, token, latency and user metrics per window — Windowed stream aggregation, Rollup aggregation, Quantile sketches (no page yet), HyperLogLog, Usage metering pipeline
• Evaluation — scores traces with a judge model, online and offline — LLM-as-judge
• Replay — re-executes a recorded run from its stored steps — Record and replay (no page yet)
• Live view — streams a running trace to the UI as it grows — Server-sent events

 agent app ──spans (model call, tool call, step, parent id)──► mask / scan ──► ingest API
  (SDK, trace context propagated to tools and sub-agents)                          │
                                                                      commit log (partition by trace id)
                                                                                   │
                                          tail sampler: keep errors, slow, costly, low-scoring, a % of the rest
                                                                                   │
                       ┌──────────── trace store ───────────┐              rollup jobs
                       │ span rows (columnar, recent in LSM) │◄──────────  per (project, model, user, hour):
                       │ big payloads (zstd blobs)           │              tokens, cost, latency sketches
                       │ text index over prompts/outputs     │
                       └───────┬───────────┬────────────────┘
          UI: trace tree ◄─SSE─┘           │
          judge workers: score sampled traces ──► scores written back to the trace
          replay runner: stored inputs + recorded step outputs ──► new run ──► compare with the original

How the parts fit together

Instrumentation

agent-tracing (no page yet) turns an agent run into spans: one root span for the run, a child for each model call, tool call, retrieval and sub-agent, each with start and end time, status, and attributes. It builds on Distributed tracing: a trace id and parent span id travel with the work, including into tools that run as separate services and into sub-agents, so the tree is complete. The OpenTelemetry GenAI semantic conventions give the attributes a shared name: for model calls gen_ai.operation.name, gen_ai.request.model and gen_ai.usage.input_tokens, plus span types for creating and invoking an agent and executing a tool; they are marked "Development" status and have moved to their own repository (see Sources). Using them means any backend can read the data.

What differs from ordinary service tracing is the payload. A span for an LLM call carries the full prompt and response, which can be thousands of tokens, and a span can last minutes while a tool or a human is in the loop. So the SDK must not block the agent (batch and send in the background), must be able to send a span at start and again at end, and should flush on process exit so the last, usually most interesting, steps are not lost.

Privacy filter

The trace is a copy of the user's data in a second system, so it is filtered before leaving the process. Data masking replaces fields known to be sensitive (email, card number, phone) with tokens or hashes. Free text needs a detector: Named entity recognition finds names, places and organisations that a regular expression cannot. Agents also handle credentials, so Secret scanning (no page yet) (no page yet) looks for API keys and tokens in prompts and tool outputs, which agents routinely print. The filter has a trade-off to decide up front: masking in the application protects privacy but changes what replay and judges see; masking in the ingest tier is easier to update but means raw data crosses a boundary. Keep the original out of the store unless a use case needs it, and then store it separately with a short retention.

Ingest and sampling

Traces arrive in bursts and out of order, so ingestion first lands in a durable buffer: a Partitioned commit log partitioned by trace id, so that all spans of one run reach the same consumer in order. This decouples the agent from slow storage and makes reprocessing possible. Langfuse's self-hosted design shows a variant of the same idea: events are kept in blob storage, a Redis queue hands jobs to a worker, and the worker writes to the analytical database (Langfuse self-hosting docs, see Sources).

Telemetry sampling (no page yet) decides what to keep. Head sampling decides when a trace starts, which is cheap but random: it drops the rare failure along with the boring runs. Tail sampling decides after the trace is complete, so it can keep every trace that errored, was slow, was expensive or got a low score, and only a fraction of the healthy rest (OpenTelemetry sampling docs). For agents tail sampling is the better fit, at a price: spans of a trace must be buffered until it ends, which is why the commit log is keyed by trace id, and a run that lasts minutes holds its buffer that long. Metrics (counts, tokens, cost) are computed on all traffic before sampling, so sampling does not bias them.

Trace store

The store has four access patterns, which is why it is usually several structures in one system:

• Writes at high rate with updates. Spans are created, then completed, and may be amended with a score, so the ingest path favours an append-friendly structure: an LSM-tree (sorted in-memory writes flushed to immutable files, merged later). Analytical stores built on merge trees behave this way.
• Analytics over a few columns. "p95 latency by model last week" touches a handful of columns across millions of spans: Columnar storage, which compresses each column and scans only what the query names.
• Big payloads. Prompts and outputs repeat large shared prefixes (system prompt, tool schemas, history), so they compress well. Zstandard (zstd) with a trained dictionary or a high level suits this; payloads often live as compressed blobs in object storage, referenced from the span row.
• Find the needle. "Show runs where the user said refund and the tool errored" is full-text search: an Inverted index over prompt and output text, maintained alongside the rows.

Langfuse documents this split directly: PostgreSQL for transactional data, ClickHouse for observability data, Redis or Valkey for cache and queue, and S3 or blob storage for events and attachments (see Sources).

Aggregation

Dashboards and alerts must not scan raw spans. Windowed stream aggregation groups spans by event-time window and by dimensions (project, model, user, tool); Rollup aggregation keeps those windows as pre-aggregated rows at coarser grains (hour, day), so a month of data reads a few thousand rows. Each rollup holds sums and counts, plus mergeable sketches for what is not summable: Quantile sketches (no page yet) (no page yet) for latency and cost percentiles, HyperLogLog for distinct users and sessions.

Cost is the number agent teams ask for first. The Usage metering pipeline turns token counts from spans into money: tokens by model and kind (input, output, cached) times a versioned price table, grouped by customer or feature, with the same rule as metering elsewhere: the raw spans are the source of truth and the totals can be rebuilt. Price tables change, so the table version used is stored with the result.

Evaluation

Traces say what happened, not whether it was good. LLM-as-judge scores traces with a second model against a rubric (helpful, grounded, followed the instruction, used the right tool). Two modes: online, scoring a sample of live traces so quality shows up on the same dashboard as latency; offline, scoring replays of a fixed dataset before a release. The judge is a measuring instrument with its own errors: Zheng et al. (2023) document position bias, verbosity bias and self-enhancement bias, so scores are validated against a small human-labelled set and the judge prompt is versioned with the scores. Scores are written back to the trace, which is what lets the tail sampler keep low-scoring runs.

Replay

Record and replay (no page yet) uses stored traces as executable material. Two uses:

• Reproduce a failure. Re-run the agent with every model and tool result served from the recording, so the same path is taken without calling anything; the step order, prompts and arguments can be stepped through and compared. The contract is determinism by recording: the only sources of non-determinism (model sampling, tool results, clock, randomness) are captured and replayed. The same principle drives durable execution engines such as Temporal, which recover a workflow by replaying its stored event history (Temporal docs).
• What if. Take the recorded inputs (and optionally recorded tool results), call a new prompt or model live, and compare outputs and judge scores with the original. Tool results come from the recording because re-calling real tools may have side effects.

The seam to watch is divergence. As soon as the new run makes a call the recording does not contain (a different tool or different arguments), the recording cannot answer; the replay must either stop and flag it or fall back to a live call, and a replay that touches live side-effecting tools must be sandboxed.

Live view

A running agent can take minutes, so the UI should show the trace as it grows. Server-sent events is a fit: one long-lived HTTP response per viewer, the server pushes each new span or token as an event, the browser reconnects on its own with Last-Event-ID. The server side reads the same commit log (or a live buffer) as the store, so the live view and the stored view cannot disagree.

Data and control flow

One agent run.

1. The SDK opens a root span and propagates the trace id to every model call, tool call and sub-agent.
2. Each step builds a span with its inputs and outputs; the privacy filter masks and scans it; the SDK sends it in a background batch. Request latency impact: a queue append (order of microseconds), not a network call.
3. The ingest API validates and appends the span to the commit log partition of its trace id and also forwards it to the live-view channel.
4. When the root span ends, the tail sampler sees the whole trace with its status, cost and any score, and keeps or drops it. Counters and sketches were already updated for every trace.
5. Kept traces are written to the store: rows into columns, payloads into zstd blobs, text into the index.
6. A judge worker picks a sample, scores it and writes the score back, which may rescue a trace the sampler would have dropped.

One replay.

1. The user selects a stored trace and a change (new prompt, new model, or none).
2. The runner loads the original inputs and the recorded results of every step.
3. It runs the agent with calls answered from the recording where the call matches, live where the change makes them differ.
4. It stores the new trace linked to the original, runs the judge on both, and shows the diff.

Scaling

In the order the limits show up:

1. Payload volume. Prompts dominate bytes. Levers: compress with a dictionary, store shared prefixes once, deduplicate repeated system prompts, move bodies to object storage, expire bodies earlier than metadata.
2. Write rate. Spans per second grow with agents times steps. Levers: batch in the SDK, partition the commit log by trace id, scale ingest consumers horizontally.
3. Query cost. Dashboards on raw spans fall over. Levers: rollups per window and dimension, sketches for percentiles, materialised views per common question.
4. Judge cost. Every judged trace is another model call. Levers: score a sample, score only the tails (errors, long runs), use a cheaper judge for a first pass.
5. Retention and privacy. Longer retention raises both cost and risk. Levers: tiered retention (rollups long, bodies short), masking at the source, deletion by user id.

Failure modes

Real implementations

Sources

• OpenTelemetry GenAI semantic conventions: repository (spans, agent spans, metrics; the older location open-telemetry/semantic-conventions now points there), overview at opentelemetry.io.
• OpenTelemetry, Sampling (head and tail sampling).
• B. Sigelman et al., Dapper, a Large-Scale Distributed Systems Tracing Infrastructure, Google 2010. page
• L. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. paper
• Langfuse, self-hosting architecture. LangSmith, docs.
• Temporal, Workflows (replay of event history).