Design a Job Scheduler

You're asked to design a distributed job scheduler that runs one-time and recurring jobs across a worker fleet. The prompt may sound like "cron, but distributed," but it has to handle duplicate schedulers, worker failures, retries, missed schedules, and large bursts of jobs due at the same time. You'll apply async jobs and workers throughout the design.

Clarify the requirements

Users register jobs to run once at a future time or on a recurring schedule using cron expressions and timezones. The system runs each due job at least once, retries failures, and exposes its status. Clarify the following before you design it:

• Who executes the work? Does the scheduler invoke customer code (webhooks, container images), or enqueue to the customer's own workers? Assume it dispatches HTTP calls or queue messages; execution semantics still matter.
• Scale? Assume 10M job definitions with due times clustered around midnight and the top of the hour.
• Guarantees? Use at-least-once firing with idempotent handlers. Exactly-once execution is not practical once workers can crash between running a job and recording the result.
• Precision? Assume precision within seconds. Subsecond scheduling changes the design, so confirm whether it is out of scope.

Back-of-envelope numbers

Assume 10M job definitions, each firing on average once an hour.

• Average load: $10\text{M definitions} / 3{,}600\text{ sec} \approx 2{,}800\text{ fires/sec}$
• Top-of-the-hour spike: if 20% of jobs fire hourly on the hour, $2\text{M fires}$ land in the same minute, roughly $33\text{k fires/sec}$ for that minute
• Definition store: $10\text{M rows} \times 1\text{ KB} \approx 10\text{ GB}$, which is small
• Run history: $240\text{M runs/day} \times 500\text{ B} \approx 120\text{ GB/day}$, which is the real storage cost

The definition store is small compared with the run history. The top-of-the-hour spike, rather than the average rate, determines how much dispatch capacity you need. Use that burst to size the queue and worker fleet.

High-level architecture

Components

1. API. CRUD on job definitions; validates cron expressions and timezones.
2. Definition store. A relational store fits well here (SQL vs. NoSQL). Each job has a row containing its schedule, payload, retry policy, enabled flag, and an indexed next_fire_at column.
3. Scheduler. Scans for due definitions in small, fixed-size batches, materializes runs, and advances next_fire_at. It decides when a job runs but does not execute it.
4. Run store. One row per execution with status, attempts, lease, and result. The source of truth workers write to.
5. Dispatch queue. Holds ready runs while they wait for a worker.
6. Workers. Pull runs, claim leases, execute an HTTP call or enqueue operation, send heartbeats, and write the terminal status.

Separate JobDefinition, which describes what should run and when, from JobRun, which represents one execution with its own lifecycle. This follows the model in async jobs and workers and keeps scheduling state distinct from execution state.

Deep dive 1: the scan, and surviving duplicate schedulers

The scheduler repeatedly runs SELECT * FROM definitions WHERE next_fire_at <= now() LIMIT batch. Process the results in small, fixed-size batches. For each definition, materialize a run, enqueue it, advance next_fire_at to the next cron occurrence, and commit.

More than one scheduler instance is needed for availability, so two instances may scan the same job. This creates a small CAP theorem problem: leader election can split, and an old leader can resume after a pause and run a scan it no longer owns. Make duplicate materialization harmless instead of relying on coordination to prevent it:

SQL
INSERT INTO runs (definition_id, scheduled_for, status)
VALUES (?, ?, 'queued')
ON CONFLICT (definition_id, scheduled_for) DO NOTHING;

The unique constraint on (definition_id, scheduled_for) turns duplicate materialization into a no-op. Two schedulers can scan the same due job, but only one run exists afterward. The data model now enforces correctness even if coordination fails. Leader election can still reduce redundant scans, but correctness no longer depends on it.

Deep dive 2: missed schedules and the recovery policy

The scheduler was down for an hour. A job that fires every minute has 60 missed occurrences. What now?

Use a per-job recovery policy because the correct behavior depends on what the job does. Catch up by materializing every missed run for billing and ledger jobs where each occurrence moves money. Skip to the next future occurrence for cache warming and cleanup jobs whose old runs have no value. Coalesce the gap into one run for aggregation jobs that can cover all missed intervals together. Store the policy on the definition, scan with a lookback window, and alert on scheduling lag.

Cron evaluation must account for timezones and daylight saving time. A 2:30 AM job does not exist on the spring-forward night and occurs twice on the fall-back night. Store the timezone with the definition, let the cron library resolve occurrences in that timezone, and document how the system handles both cases.

Deep dive 3: execution semantics and worker failure

A worker takes a run, calls the customer's endpoint, and dies before recording the result. The run's lease expires, another worker claims it, and the endpoint gets called twice. This is why firing is at-least-once, and why the platform passes an idempotency key (the run ID) on every dispatch so customer handlers can dedupe, the same discipline as the execution model.

Classify errors, retry with exponential backoff and jitter under a per-definition budget, and expose a dead-letter state in the dashboard. A retry of run N may overlap with the scheduled start of run N+1. Give each definition a concurrency policy: allow overlapping runs, forbid the new run while the old one remains active, or replace the old run with the new one. Kubernetes CronJobs use the same vocabulary.

Deep dive 4: scaling the spike

The top-of-the-hour burst stresses each tier differently. The indexed range scan is cheap, but materializing 2M runs in one minute requires sharding. Partition definitions by a hash of the definition ID and assign one scheduler to each partition, applying distributed storage to the scheduler itself. The queue absorbs the burst while workers drain it at their own rate. Scale workers using the age of the oldest ready run rather than queue depth. You can also add per-definition jitter, such as "hourly, ±90 seconds," to spread the burst at its source.

Leveling signals

Related lessons

Design a Web CrawlerHard

Applies the same scheduling ideas to recrawl intervals, with an adversarial set of targets.

Design Webhook DeliveryHard

What the dispatch edge looks like when the endpoint being called belongs to a customer.