Design an Application Performance Monitoring System

You're asked to design an application performance monitoring system like Datadog APM, New Relic, or Honeycomb. A single user request touches a dozen microservices on its way through the system, and when it takes four seconds instead of two hundred milliseconds, an engineer needs to see exactly where that time went. The system should also aggregate across millions of such requests to answer "which service regressed after Tuesday's deploy?"

Logs show what one service did, while metrics summarize many requests. Tracing must reconstruct one request across process boundaries and show that it spent 3.4 seconds on a database call three services deep. We'll focus on context propagation, sampling, and trace assembly.

Clarify the requirements

• What's the unit of analysis? Use a trace to represent one request's full path across services. Establish this before choosing the data model.
• Do we trace every request? Full collection is expensive at scale, so ask what sampling rate is acceptable.
• How deep does instrumentation go? Service-to-service calls at minimum, and usually database queries and external API calls too, since that's where the time usually is.
• What's the query pattern? Two very different ones: find one specific slow trace, and aggregate latency percentiles across many. They need different storage.
• How fresh? A minute or two is fine for investigation. Nobody debugs a production incident on second-old trace data.

Assume: full traces including database calls, sampled rather than complete, both single-trace and aggregate queries, and minute-level freshness.

Back-of-envelope numbers

• Requests: $100\text{k requests/sec}$ across the fleet
• Spans per request: $20 \text{ services and calls}$, so $100\text{k} \times 20 = 2\text{M spans/sec}$ if you traced everything
• Span size: $\approx 500\text{ B}$, so unsampled ingest is $2\text{M} \times 500\text{ B} = 1\text{ GB/sec} \approx 86\text{ TB/day}$
• At 1% sampling: $\approx 860\text{ GB/day}$, which is a workable number
• Aggregates are computed from every request regardless of sampling, and cost a few hundred gigabytes a month

Storing every span costs roughly one hundred times more than sampling at one percent. Decide which traces to retain while computing aggregate metrics from the complete stream.

High-level architecture

Components

1. Local collector. An agent per host that batches spans, so the application never blocks on the monitoring system.
2. Span buffer. Decouples collection from the sampling and aggregation stages.
3. Sampler. Decides which complete traces are worth storing.
4. Aggregator. Computes latency and error-rate metrics from every span, sampled or not.
5. Trace store. Full span data for kept traces, indexed by trace ID and by service.
6. Metrics store. Per-service, per-endpoint latency percentiles and error rates.
7. Query service. Serves both trace lookups and aggregate dashboards.

Deep dive 1: the trace and span model

Start with the span model because every later stage depends on these identifiers.

A span is one unit of work: a service handling a request, a database query, an outbound HTTP call. It carries a start time, a duration, a name, and a set of tags. A trace is the set of spans belonging to one original request, and the structure comes from three identifiers on every span:

• trace_id, identical across every span in the request, generated once at the entry point.
• span_id, unique to this span.
• parent_span_id, pointing at the span that caused this one.

Those three fields make a tree, and the tree is what the waterfall visualization renders. A span with no parent is the root, and each child's offset from the root's start time is where it appears on the timeline.

Call out two details. Spans arrive independently and out of order, so the backend assembles the tree from identifiers rather than arrival order. Also attach the tags needed for later filters, such as service, version, host, endpoint, and customer tier. The backend cannot recover tags that producers never emitted.

Deep dive 2: propagating context across process boundaries

The tree exists only if trace_id and parent_span_id survive every hop. Explain context propagation across both synchronous and asynchronous boundaries.

Across HTTP, send the identifiers in the W3C traceparent header. Across a message queue, put them in message metadata so an asynchronous hop does not split one trace into disconnected fragments.

Within a process, the current span has to be available to code that never received it as an argument, which is what thread-local storage or an async context variable provides. This is why tracing libraries are language-specific rather than a simple HTTP client wrapper.

Deep dive 3: sampling, and why head sampling isn't enough

Storing one percent of traces is affordable. The question is which one percent, and the naive answer throws away the traces you actually needed.

Head-based sampling decides at the entry point, before the request runs, typically by hashing the trace ID against a rate. The decision propagates with the context so every service agrees, which means you always get complete traces. It's cheap and simple, and its flaw is decisive: it can't know whether the request will be slow or fail, so a 1% rate keeps 1% of your errors and discards the rest.

Tail-based sampling decides after the request finishes, when the outcome is known. Keep error traces, traces above a latency threshold, and a small random sample of normal traffic. This requires buffering spans until a trace completes. Route every span with the same trace_id to one place using consistent hashing, and set a time limit for incomplete traces.

Combine both approaches. Use head sampling to discard predictable low-value traffic such as health checks and static assets, then apply tail sampling to the remaining traces.

Deep dive 4: aggregate metrics from a sampled stream

If you store one percent of traces, you cannot compute p99 latency from stored traces. The percentile of a biased sample is not the percentile of the population, and a tail-sampled set is biased toward slow requests by construction.

The resolution is to separate the two paths. Every span contributes to aggregates, whether or not its trace is stored. The collector computes per-service, per-endpoint counts, error rates, and latency distributions from the full stream, and only the trace storage path is sampled. Aggregates are exact; traces are exemplars.

Latency distributions need a mergeable representation rather than a precomputed percentile, since percentiles from different hosts cannot be averaged. Sketches, meaning t-digest or HDR histograms, merge correctly across hosts and across time buckets, which is what makes a fleet-wide p99 meaningful.

One feature ties the two paths together: attach a few stored trace IDs as exemplars to each latency bucket, so an engineer looking at a p99 spike on a dashboard can click through to an actual slow trace that produced it. That link between aggregate and instance is what makes an APM tool useful rather than merely informative.

Common pitfalls

• No context propagation plan. Without it there are no traces, only disconnected spans.
• Breaking the trace at async boundaries. Queue hops need the context in message metadata, or the trace splits in two.
• Head sampling alone. A 1% rate keeps 1% of your errors, which are the traces you needed.
• Computing percentiles from sampled traces. The sample is biased by design, so the number is wrong.
• Averaging percentiles across hosts. Merge sketches instead; the mean of p99s is not a p99.

Leveling signals

Related lessons

Design a Metrics and Logging ServiceHard

Collect, index, and retain metrics and logs from thousands of services.

Design a Time Series Metrics StoreHard

Build the storage engine behind a monitoring system, with compression, cardinality limits, and rollups.