Design a Metrics and Logging Service

You're asked to design the observability platform every other team at the company depends on: the system that collects metrics and log lines from thousands of services, makes them searchable within seconds, powers dashboards and alerts, and keeps enough history to investigate an incident from last quarter. Engineers should be able to ask "show me every error from the checkout service in the last hour, grouped by host" and get an answer before they lose their train of thought.

Metrics and logs arrive through the same agents and the same pipeline, but they are different data with different economics. Metrics are small, numeric, and predictable. Logs are large, unstructured, and arrive in unpredictable bursts, and they dominate every cost estimate you'll make here.

Clarifying the requirements

• Metrics, logs, or both? Confirm both are in scope, then say early that they need separate storage. Treating them as one dataset is the most common way this design goes wrong.
• How fresh does the data need to be? Seconds for alerting, and minutes is acceptable for dashboards. That difference is what justifies two paths through the pipeline.
• How long is the retention? Days at full fidelity and months in cheaper storage is typical, and the answer drives most of the cost.
• Who queries it, and how? Free-text search over logs, aggregation over metrics, and alert rules evaluated continuously. Three different query shapes.
• What happens if the pipeline falls behind? Dropping data during an incident is exactly when you least want to, so ask what the buffering expectation is.

Assume: both data types, seconds of freshness for alerts, 30 days hot and 13 months cold, and buffering that survives a multi-hour outage.

Back-of-envelope numbers

• Log volume: $10\text{k hosts} \times 100 \text{ lines/sec} = 1\text{M log lines/sec}$
• Log bytes: $1\text{M/sec} \times 500\text{ B} = 500\text{ MB/sec} \approx 43\text{ TB/day}$ raw
• After compression at roughly $10\times$: $\approx 4\text{ TB/day}$, or $120\text{ TB}$ for 30 days hot
• Metrics: $10\text{k hosts} \times 200 \text{ series} \div 15\text{ sec} \approx 130\text{k points/sec}$, which compresses to a few hundred gigabytes a month
• Buffer for a 4-hour outage: $500\text{ MB/sec} \times 14{,}400\text{ sec} \approx 7\text{ TB}$

The gap between those two volumes is the whole design. Logs are roughly two orders of magnitude more expensive than metrics, so most of the effort goes into reducing how much of the log stream you index and how long you keep it.

High-level architecture

Components

1. Ingest tier. Authenticates agents, validates payloads, and enforces per-tenant quotas before anything is accepted.
2. Kafka buffer. Absorbs bursts and decouples ingestion from processing, so a slow index never rejects incoming data.
3. Metrics processor. Aggregates and writes numeric series.
4. Log processor. Parses, enriches, samples, and routes log lines.
5. Time series store. Compressed numeric points, partitioned by series and time.
6. Search index. Inverted index over recent logs, which is what makes free-text queries fast.
7. Object storage. The raw compressed log bodies, cheap and long-lived.
8. Alert evaluator. Runs rules continuously against recent metrics.
9. Query service. Fans a user query to whichever store can answer it.

Deep dive 1: why the buffer is the most important component

Everything upstream of the buffer is a firehose you don't control, and everything downstream is a system with finite capacity. Putting a durable log between them is what keeps a slow consumer from turning into data loss.

Concretely, the buffer gives you four things:

• Burst absorption. An incident produces a log spike exactly when the data matters most, and the queue takes it while processing catches up.
• Independent consumers. Metrics and logs read the same stream at their own pace, and adding a third consumer later costs nothing upstream.
• Replay. A bug in the log parser is fixable by resetting the consumer offset and reprocessing, rather than by losing a day of data.
• Backpressure that stops at the right place. When the index is overwhelmed, consumers fall behind and the queue grows. Agents keep shipping successfully, which is what you want, because the alternative is agents buffering on production hosts and eventually filling their disks.

Size the buffer from the outage you're willing to survive, which is the seven-terabyte figure above. That number belongs in the interview because it turns "we'll use Kafka" into a capacity decision.

Deep dive 2: storing metrics and logs differently

The single largest mistake in this design is putting both data types in one store. They differ on every axis that matters.

Metrics go to a time series store, where points for one series are contiguous and compress to a byte or two each. The mechanics of that store are their own design question, covered in Design a Time Series Metrics Store.

Logs need full-text search, which means an inverted index mapping terms to the documents containing them. The index is what costs money, so the design lever is what you put in it: index the structured fields of service, host, level, trace ID, and timestamp, plus the message body, but keep the raw line in object storage rather than duplicating everything into the index.

That split lets you tier honestly. Recent logs are indexed and instantly searchable. Older logs live only in object storage, where a query is a slower scan rather than an index lookup, at a small fraction of the cost.

Deep dive 3: controlling what you keep

At a terabyte an hour, the design question isn't how to store everything, it's how to decide what's worth storing. Four levers, in the order you'd apply them:

• Sampling by level. Keep every error and warning, and sample info and debug at some percentage. Most log volume is routine success messages that nobody has ever searched for.
• Aggregation at the agent. A thousand identical lines become one line with a count. This is enormously effective on tight loops and retry storms, which are exactly what produce volume spikes.
• Tiered retention. Full-fidelity indexed logs for days, then index-free object storage for months, then delete. Each tier is roughly an order of magnitude cheaper than the one above it.
• Per-tenant quotas. One team turning on debug logging shouldn't degrade observability for everyone else, so quotas belong at the ingest tier where they can reject rather than downstream where the damage is already done.

Structured logging is what makes all of these possible. A line emitted as JSON with consistent field names can be sampled by level, aggregated by message template, and queried by field. A free-text line can only be grep'd. Recommending structured logging as a platform requirement rather than a nice-to-have is a strong signal, because it's a decision that has to be made before the volume arrives.

Deep dive 4: alerting on a stream that can lag

Alerts are the part of this system that has to work when everything else is on fire, and that constraint shapes their design.

Evaluate alerts on the streaming path, not on the stored data. Reading from the buffer directly keeps alert latency at seconds even when the indexing pipeline is minutes behind, which is the state the system will be in during exactly the incident the alert exists to catch.

Absence of data is itself an alert condition. A service that stops reporting looks identical to a healthy service with nothing to say, and the difference matters enormously. Alert rules need a "no data received" case rather than only thresholds on values.

Deduplicate and group before notifying. One bad deploy across five hundred hosts produces five hundred firing rules, and paging someone five hundred times is the same as paging them zero times. Group by the alert rule and the smallest common dimension, and send one notification with a count.

State the pipeline's own health as a first-class metric. Consumer lag on the buffer, ingest rejection rate, and indexing latency should be monitored by something outside this system, since an observability platform that goes down silently takes every other team's visibility with it.

Common pitfalls

• One store for metrics and logs. Their volume, shape, and query patterns all differ by orders of magnitude.
• No buffer between ingestion and processing. A slow index becomes dropped data during the incident you most need to see.
• Indexing every field of every log line. The index becomes larger than the data and dominates the cost.
• Alerting off the indexed store. Alert latency inherits pipeline lag, which is worst exactly when it matters.
• No per-tenant quota. One team's debug flag degrades observability for the entire company.

Leveling the answer

Related lessons

Design a Time Series Metrics StoreHard

Build the storage engine behind a monitoring system, with compression, cardinality limits, and rollups.

Design an Application Performance Monitoring SystemHard

Trace a single request across dozens of services and find where the latency went.