Batch Processing and Data Pipelines

Your product generates billions of clicks, orders, page views, and errors each day. How do systems like Datadog or a large analytics platform turn those raw events into useful dashboards, metrics, and models? The answer is a data pipeline, a system that produces derived data rather than a response to one user request.

Most pipelines follow the same stages: ingest, buffer, process, store, and serve. We still need to choose how quickly each consumer needs its result. Batch processing handles bounded chunks at high throughput with minutes or hours of latency. Stream processing handles an unbounded flow continuously and can produce results within seconds, but it is harder to operate correctly. Good pipelines use replayable stages, distinguish event time from processing time, support backfills, and avoid paying for streaming where a scheduled batch is enough.

The core idea

Transactional databases, often called OLTP systems for online transaction processing, handle many small, targeted reads and writes. Analytics has the opposite shape: fewer queries that may scan billions of rows. Running GROUP BY over last month's events on the production database can overwhelm the same database serving checkout. Give analytics its own path, with columnar, append-only storage partitioned by time and parallel compute that runs near the data.

Make two decisions early. First, choose batch or streaming per consumer. Batch processes a bounded input such as yesterday's events. It is cheaper and easier to replay. Streaming processes an unbounded input continuously and provides fresher results, but adds state, event-time handling, and exactly-once processing concerns. A fraud model may need results within seconds, while a finance report only needs a correct daily result.

Second, decide where the source of truth lives. Keep an immutable log or data lake of raw events and treat downstream tables as derived, rebuildable data. If a bug corrupts an aggregate, we can correct the logic and replay the raw input.

Anatomy of a pipeline

Event time and processing time

Stream aggregations operate over tumbling, sliding, or session-based windows. We need to choose which timestamp assigns an event to a window. Event time records when the event happened, but events may arrive late or out of order because clients go offline, retries reorder messages, and clocks drift. Processing time records when the pipeline received the event. It is simpler, but a delayed event lands in the wrong bucket.

If you use event time, you face a question that has no perfect answer: when do you stop waiting? Say you're counting events in the 2:00 to 2:05 window. At 2:05 some of those events are still in transit, so closing the window immediately undercounts, and waiting forever never produces a result.

Use a watermark, a heuristic estimate that the pipeline has probably received every event up to time T. The watermark closes and emits a window. Events that arrive afterward need an explicit policy: drop them or publish a correction, which requires downstream consumers to accept updated results.

Getting correct results from repeated work

Workers crash mid-batch and streams replay from checkpoints, so the default everywhere is at-least-once delivery: some input will be processed twice. There are two honest ways to still get correct results.

• Idempotent, deterministic outputs. A batch job that atomically overwrites its whole output partition is naturally safe to re-run, which is why "recompute the entire daily partition" beats clever incremental updates for reliability. Streaming sinks get the same property by upserting on a key or deduplicating on event ID.
• Transactional sinks. Engine checkpoints plus transactional writes give you effective exactly-once, but only inside the framework's boundary. The moment you write to an external system, you're back to needing idempotency.

In practice, exactly-once processing means at-least-once delivery combined with idempotent effects. The same rule applies to event-driven and pub/sub.

Running pipelines in production

Orchestration. Batch pipelines are DAGs of dependent jobs ("sessionize after ingest completes, aggregate after sessionize") run by a scheduler that owns retries, dependency gating, SLAs ("daily revenue ready by 6am"), and alerting. This is async jobs and workers's job machinery specialized for data dependencies.

Design for backfills. Logic changes, bugs, and late corrections require reprocessing historical data. Parameterize jobs by date range, retain raw data long enough, make output partitions safe to overwrite, and rate-limit backfill compute so it does not starve current runs.

Data quality. Validate schema, null rates, volume, and distributions at stage boundaries. Quarantine bad records in a dead-letter path instead of failing the entire pipeline or silently accepting invalid data. Use a schema registry so an incompatible upstream change fails at publish time rather than corrupting downstream tables.

Skew and cardinality. Parallelism comes from partitioning by key, so the failure mode is distributed storage's hot key: one whale customer makes one reducer run for hours while thousands idle. Salt the hot key and aggregate in two phases, or handle known whales in a dedicated path. The metrics-specific killer is cardinality explosion: metrics keyed by unbounded labels (user ID, request ID) create a time series per value and blow up index and memory. Enforce label allow-lists and cardinality budgets at ingest.

When to use it, and when not to

Use pipelines when data volume makes per-request computation impractical, such as analytics over billions of events, log and metric aggregation, search indexing, and ML feature generation. They also fit derived outputs such as reports, rollups, indexes, and training sets, especially when freshness can range from seconds to hours and historical data may need reprocessing.

Keep pipelines off the request path. A pipeline analyzes or derives data from traffic; it should not block an individual request. It should also not own transactional side effects. Observing an event must not be the mechanism that charges a card, which belongs in transactional workflows. For a few GB of data, a Postgres query or cron job may be enough. Distributed engines add coordination overhead that small datasets do not justify.

Common pitfalls

• Running analytics on the production transactional database. Large scans compete with the small reads and writes serving the product.
• No backfill story. Every bug becomes an incident when you can't re-run history.
• Processing-time windows. They quietly miscount every user who was offline.
• Unbounded cardinality. Metrics keyed by user or request ID create a time series per value and overwhelm the store.
• Ignoring skew. One whale customer makes one worker run for hours while the rest sit idle.

Leveling signals

Practice this pattern