Async Jobs and Workers

When a user requests a data export or uploads a large video, the work may take minutes or hours. How can the application respond immediately and still finish that work reliably? The answer is the asynchronous jobs and workers pattern.

Persist the request as a job, return its ID, and let background workers execute it. The initial request finishes in milliseconds, while the job can report progress and survive retries or worker crashes. We now need to design job state, leases, idempotency, retry policies, and worker capacity.

The core idea

Request/response works when the operation is short and the user needs its result immediately. Move work into a job when it takes a long time, depends on an unreliable external service, must run later, or needs progress tracking. Video transcoding, data exports, ML inference, email delivery, and scheduled reports all fit this shape.

Instead of finishing the work before responding, persist the user's intent and make the background execution reliable.

Take a file upload in an AI chat app. The API should not synchronously scan, parse, OCR, chunk, embed, and index a document before responding. Instead it stores the file durably, creates a process_uploaded_file job, and returns 202 Accepted with a job_id. Workers run the pipeline in the background, and the user polls GET /jobs/{job_id} or gets notified on completion. The request path stays fast, and the heavy work becomes something the system can retry, scale, and observe.

How this relates to event-driven systems

These two patterns look nearly identical on a whiteboard. Both put a queue between a producer and a pool of consumers, both run work outside the request path, and both need retries, idempotency, and a dead-letter path. In practice they often run on the same broker.

The difference is what the message means. In event-driven systems the unit is an event, a statement of fact like FileUploaded, broadcast to whoever cares. Here the unit is a job, an instruction to do a specific piece of work like process_uploaded_file, addressed to whoever is free. An event says something happened; a job says something needs doing.

That shapes what each pattern worries about. Event-driven design is mostly concerned with communication between services: who subscribes, how the schema evolves, whether ordering holds. Async job design is mostly concerned with execution: tracking a job's state, keeping it alive when a worker dies, bounding retries, and scaling the fleet.

They compose naturally, and most real systems use both: FileUploaded (event) → process_file (job) → FileIndexed (event).

Core architecture

Components

1. API. Validates, writes a durable job record, returns a job_id immediately. Does no work itself.
2. Metadata store. The source of truth for job state (status, ownership, progress, attempts, result). The client never reads it directly; status polls go back through the API.
3. Queue or job table. The dispatch mechanism, deciding which job a worker picks up next.
4. Scheduler. Decides when delayed or recurring jobs become eligible. Executes nothing.
5. Worker pool. Stateless processes that claim leases, execute, heartbeat, and write terminal status.

The metadata store holds the fields that make jobs observable and recoverable. The ones that carry the design:

Alongside these sit the obvious identifiers and payload (job_id, job_type, the input, a result pointer, and progress). The queue answers "which job next?" The metadata store answers everything else: what it is, what state it's in, who owns it, and what happened.

Job lifecycle

A job is not a queue message. It has a lifecycle, and a failed attempt is not a failed job.

Timeouts and cancellation add more transitions. Explicit state lets users see progress, gives retries their attempt counts and error codes, and allows the system to recover jobs with expired leases. Operators can also find stuck jobs and identify the affected tenants.

Leases and reliable execution

A worker should never own a job forever. It claims a lease for a limited period, and if it crashes, the lease expires and another worker retries. Managed queues call this a visibility timeout. In a job table it looks like:

SQL
UPDATE job_runs
SET status = 'running', claimed_by = 'worker-17',
    lease_expires_at = now() + interval '60 seconds'
WHERE run_id = 'run_123' AND status = 'queued';

For long jobs, the worker heartbeats to extend the lease and update progress. Final writes should check ownership or use a fencing token, so a stale worker whose lease already expired can't overwrite the status written by its replacement.

A worker can also crash after performing a side effect but before acknowledging completion, so the job runs again. Job execution is therefore at-least-once, and handlers have to make repeated execution safe. Pass an idempotency key derived from the job ID to external calls (charge_customer(amount, idempotency_key=job_id)), write results with UPSERT / ON CONFLICT DO NOTHING, store outputs at deterministic paths and check before recomputing, and break long jobs into checkpointed steps. Deduplicate at creation too: a unique constraint on (tenant_id, idempotency_key) prevents duplicate jobs when a client retries POST /exports.

Retries and failure handling

This section is the canonical reference for retry mechanics; event-driven and pub/sub links here.

Retries must be explicit, capped, and observable. A policy includes max_attempts, initial and max backoff, a multiplier, jitter, and, most importantly, error classification. Retryable errors are network timeouts, 429s, 5xxs, and worker crashes. Non-retryable errors are invalid input, authorization failures, and malformed payloads. Retrying those wastes capacity and delays the DLQ signal.

The standard formula:

delay = min(max_backoff, initial * 2^attempt) + random_jitter

Without backoff, a downstream outage triggers a retry storm that makes the outage worse. Without jitter, thousands of jobs retry at the same instant. After the budget is exhausted, the job moves to a dead-letter queue with its error context, which is an operational tool for inspection, alerting, and controlled replay after a fix, not a trash can. Watch for poison jobs: a malformed input that fails identically every attempt should be classified non-retryable early rather than burning through the full retry budget.

For every failed attempt, define the error classification, backoff, retry budget, dead-letter behavior, and replay path. "Failures are retried" is not enough to operate the system.

Scheduling and delayed work

Many jobs should run later rather than immediately, including reminders, webhook retries, daily reports, cleanup jobs, and billing cycles. This is the scheduling part of the pattern.

The simplest model is a run-at timestamp. Set available_at in the future, and workers only take jobs where available_at <= now(). Delay queues can implement short delays, but long-term schedules need a durable database record.

For recurring work, separate what should run from one execution of it. A JobDefinition holds the job type, payload, cron expression, timezone, next_fire_at, retry policy, and enabled flag. A JobRun is one concrete execution with its own status, attempts, timestamps, and result. The scheduler scans for due definitions in small fixed-size batches, creates a run, enqueues it, and advances next_fire_at. It decides when; workers decide how.

Two scheduler instances may scan the same due job. Leader election helps but can split-brain, so the data model should make duplicates harmless:

SQL
INSERT INTO job_runs (job_definition_id, scheduled_at, status)
VALUES (?, ?, 'queued')
ON CONFLICT (job_definition_id, scheduled_at) DO NOTHING;

The unique constraint on (job_definition_id, scheduled_at) turns duplicate materialization attempts into no-ops. We also need a policy for schedules missed during an outage. Catch up by creating every missed run for billing, skip to the next future run for cache cleanup, or coalesce the gap into one run for metrics aggregation.

Worker scaling

The API creates jobs and the scheduler makes them ready, but the worker fleet determines whether work actually gets done on time.

Pull, not push. Default to workers pulling when they have capacity: natural backpressure, easy horizontal scaling, and crashed workers simply stop pulling. Push (a dispatcher assigning work) only wins when you need specialized placement.

Size by throughput, not queue depth. A thousand ten-millisecond jobs and a thousand ten-minute jobs are different worlds. Approximate instead:

required_concurrency ≈ arrival_rate × avg_runtime × retry_factor × headroom
e.g. 500 jobs/sec × 2s × 1.3 ≈ 1,300 concurrent executions

Autoscale on oldest-job age. The age of the oldest ready job, or equivalently p95 job-start latency, measures user impact more directly than queue depth. Scale up when that age crosses a threshold. Scale down only after the backlog remains low and workers stay underutilized for a sustained period. This is the job-system equivalent of consumer lag in event-driven and pub/sub.

Separate pools by job type. One pool for everything lets video transcoding starve password-reset emails. Separate pools get their own instance types, autoscaling policies, timeouts, rate limits, and deploy cadence.

Per-tenant fairness. One tenant submitting 10 million indexing jobs must not delay another tenant's password-reset email. Use per-tenant concurrency caps, weighted fair scheduling, or tenant-level token buckets. Track oldest-job age per tenant rather than only across the whole fleet. The same noisy-neighbor problem appears as per-domain politeness in Design a Web Crawler.

Respect downstream rate limits. Workers call email providers, payment APIs, webhook endpoints, and LLM services. Respect downstream capacity with per-provider token buckets, max in-flight caps, circuit breakers, and Retry-After handling. For webhook delivery this is the whole game: one customer's 429ing endpoint should slow that customer's jobs, not the platform.

Applying the pattern

Design a Job Scheduler. The pattern is the question. Lead with definition vs. run, the scheduler scan with idempotent materialization, leases, retry budgets, and the status API. Deep dives cover missed-schedule policy, duplicate schedulers, and fairness.

Design Webhook Delivery. Each delivery is a job: retry with backoff per endpoint, per-endpoint rate limits and isolation, a DLQ after the budget, delivery-attempt history, and an idempotency key per delivery.

Design Facebook Data Export. A long-running parent job with fanned-out child jobs per data type or shard: return an export_id immediately, track progress across children, retry failed shards only, assemble the archive, and expire the result.

File Uploader for an AI Chat App. A staged pipeline as jobs (scan, parse, chunk, embed, index), with a checkpoint per stage, independent retries per stage, separate CPU/GPU pools, and progress reported by stage.

A practical design sequence

1. Identify which work leaves the request path.
2. The API persists job metadata and returns a job_id.
3. A queue or job table dispatches ready work; the scheduler handles run-at and cron.
4. Workers pull, claim leases, heartbeat, and write progress and terminal status.
5. Retry transient failures with backoff and jitter under a budget; DLQ the rest.
6. Make handlers idempotent and dedupe at creation.
7. Scale on oldest-job-age, separate pools, and enforce tenant fairness.
8. Expose a status API and operator controls (pause, replay, cancel).

Leveling signals

Practice this pattern

Design Facebook Data ExportMediumAsked at Meta

Let a user request a full export of their account data and download it when it's ready.

Design a Web CrawlerHardAsked at Google

Crawl and download a large set of web pages while respecting per-site rate limits.