Design a Web Crawler

A web crawler is the cleanest possible expression of background work at scale: no user is waiting on any single page fetch, the work never ends, and the system's job is to execute billions of small tasks reliably, politely, and without repeating itself. That makes it one of the best problems for demonstrating that you can design an execution platform rather than just draw a queue.

Clarifying the requirements

A crawler takes seed URLs, fetches those pages, extracts the links inside them, and follows those links outward until it has covered as much of the web as its owner cares about. The classic consumer is a search engine's indexing pipeline. Before designing, pin down what your interviewer actually wants:

• What happens to fetched pages? Usually they're stored for downstream indexing and ranking, which means you need a storage story, not just a fetching story.
• One-shot or continuous? Real crawlers run on a schedule, recrawling pages as they change. This single answer transforms the design from a batch script into a scheduling system.
• Are all sites equal? No. A news homepage changes hourly; an old blog post changes never. The crawler should prioritize accordingly.
• Duplicates? The web is full of URLs you've seen and content you've already stored under a different URL. Both kinds of duplication waste your budget and pollute the index.
• How hard can you hit a single site? Crawlers are expected to be polite, meaning they limit how fast they request pages from any one website and follow the rules that site publishes in its robots.txt file. Hammering someone's server is somewhere between rude and a denial-of-service attack, and it gets your crawler blocked.

Non-functional requirements: scalable to a large fraction of the web, highly available, and efficient, in the specific sense that the system should be smart enough not to fetch what it has already seen.

Back-of-envelope numbers

Assume 1 billion pages at roughly 100 KB per page, recrawled on average every 7 days.

• Storage per full pass: $10^9 \text{ pages} \times 100\text{ KB} \approx 100\text{ TB}$. With extracted text, metadata, and retained versions, plan on a petabyte-class store over time.
• Throughput: $10^9 \text{ pages} / 7\text{ days} \approx 1{,}650\text{ pages/sec}$ sustained
• Bandwidth: $1{,}650\text{ pages/sec} \times 100\text{ KB} \approx 165\text{ MB/s}$ of fetch traffic

The math points to two conclusions. First, page content belongs in object storage (S3-style), not in a database. Second, no single machine fetches 1,600 pages a second politely; this is a large worker fleet coordinated through a shared frontier, which is exactly the shape of async jobs and workers.

High-level architecture

Components

1. Scheduler. Finds URLs whose next_crawl_at has passed and materializes them into the frontier. Never fetches anything itself.
2. Crawl frontier. The dispatch queue, partitioned by domain for politeness, with priority classes so urgent recrawls skip the backlog.
3. Fetcher workers. Stateless pull-based pool. Lease a URL, fetch it, report the result.
4. Extractor. Parses the page, stores the content, and hands discovered links to the dedupe path.
5. Object storage. Fetched pages, written once per unique content checksum.
6. URL metadata store. One row per URL with priority, next_crawl_at, attempt count, and last status. The job store and the source of truth.

The loop is the whole system: the scheduler decides which URLs are due, the frontier dispatches them, workers fetch and parse, the extractor stores content and feeds newly discovered URLs back to the metadata store, and the cycle continues forever.

Mapping each component to the pattern vocabulary makes the design easy to narrate. The URL metadata store is the job metadata store: one row per URL with priority, next_crawl_at, attempt counts, and last status. The frontier is the dispatch queue, and only dispatch: it answers "what does a worker fetch next," while the metadata store answers everything else. The scheduler materializes due work into the frontier and never fetches anything itself. Fetcher workers are a stateless pull-based pool. A crawl of one URL is a job, and "recrawl news sites hourly" is a recurring job definition.

Deep dive 1: scheduling and prioritization

Not all pages deserve the same attention, so the metadata store carries a priority and a next_crawl_at per URL. News and other fast-changing sites get short recrawl intervals; static reference pages get long ones. The scheduler scans for due URLs (next_crawl_at <= now()) in small fixed-size batches and pushes them into the frontier, which can be a set of priority queues (one per priority class) so urgent recrawls never wait behind a backlog of low-value pages.

Two refinements earn senior credit here. First, adapt the interval per page: if the content checksum is unchanged across several crawls, back the frequency off; if it changes every visit, tighten it. The system learns each page's change rate instead of relying on a hand-set constant. Second, handle scheduler failure the same way a job scheduler does: schedule state lives durably in the metadata store, not in scheduler memory, so a restarted scheduler resumes from next_crawl_at values. Duplicate materialization by two scheduler instances is made harmless with a uniqueness guarantee per (URL, scheduled time), the idempotent-materialization trick from async jobs and workers. For a missed window, coalesce: crawl the page once now rather than replaying every missed slot.

Strict priority also brings the classic starvation problem: if news URLs saturate the fleet, the long tail never gets crawled. Reserve a slice of capacity for low-priority work or age priorities upward over time.

Deep dive 2: deduplication, twice

Duplication attacks a crawler from two directions, and conflating them is a common interview stumble. Keep them separate.

URL-level dedupe: have I seen this URL before? With billions of known URLs, checking a candidate against a database on every discovery is a heavy read path. A hash-set lookup is O(1) on paper, but the table is enormous, lives partly on disk, and every discovered link on every page hits it.

A Bloom filter is the standard answer. It's a compact probabilistic structure that answers set membership with one-sided error: "definitely not seen" is always correct, while "probably seen" has a small false-positive rate. Membership tests stay O(1) with a handful of hash functions over a bit array that fits in memory even for billions of entries. The false-positive cost is benign here: rarely skipping a genuinely new URL is acceptable at web scale, and if you can't tolerate even that, treat the filter as a cheap first pass and confirm positives against the authoritative store. Normalize URLs before checking (case, trailing slashes, tracking parameters, fragments), or trivial variants slip past.

Content-level dedupe: have I stored this page before, under another URL? Mirrors, session-ID URLs, and print views serve identical content at different addresses. Catch them by computing a checksum or fingerprint of fetched content (MD5-style hashing works; fingerprints like SimHash also catch near-duplicates) and checking a checksum store before writing to object storage. On a hit, record the URL-to-content mapping and skip the store.

Both checks are forms of idempotency, the property async jobs and workers demands of any at-least-once system: fetching or processing the same thing twice must not corrupt the result or double the cost.

Deep dive 3: politeness, traps, and fairness

Politeness is the constraint that makes crawling unusual: the capacity you have to protect isn't yours. Most systems ration their own capacity between their own users. Here you're rationing your requests against servers run by strangers who never agreed to host your traffic. A crawler that ignores this is indistinguishable from an attack, and gets blocked accordingly.

Respect robots.txt. Fetch and cache it per domain, honor disallow rules and Crawl-delay, and re-fetch it periodically.

Rate-limit per domain. Cap concurrent connections (often to one) and enforce a delay between requests to the same domain. The clean implementation partitions the frontier by domain, with a per-domain queue and its own token bucket. This is exactly the per-tenant token bucket from async jobs and workers, pointed outward: one slow or strict domain throttles only its own queue while the fleet stays busy elsewhere.

Cap pages per domain. Some sites have millions of auto-generated pages, and crawler traps (calendars that link forward forever, infinitely paginated listings) generate URLs without end. Set a per-domain page budget, cap URL depth, and watch for URL patterns that keep growing without new content checksums. Without these limits, one pathological site can eat a meaningful fraction of your fleet.

DNS at this rate is its own dependency. Millions of lookups per hour will rate-limit you at public resolvers and add latency to every fetch. Run a caching resolver near the workers; at extreme scale, crawlers often build their own resolution layer for control over caching and timeouts.

Deep dive 4: failure handling and worker scaling

Fetches fail constantly: timeouts, 5xxs, connection resets, rate-limit responses. Classify before retrying, the same discipline as any job system. Transient errors (timeouts, 429s, 5xxs) retry with exponential backoff and jitter under a small per-URL budget, and a 429 or Retry-After should also slow the whole domain's bucket, not just that URL. Permanent errors (404, 410, DNS non-existence) don't retry; mark the URL and move on. URLs that exhaust their budget get parked in a dead-letter state with their error context for inspection, useful for spotting a systemic problem like your crawler's IP range being blocked.

Workers themselves crash mid-fetch. Claims on frontier entries carry a lease (a visibility timeout), so a died worker's URL simply reappears for another worker, and content-checksum dedupe makes any resulting double-fetch harmless.

Scale the fleet on oldest-due-URL age, the crawler's version of oldest-job-age: if high-priority URLs are sitting due for too long, add fetchers. Raw frontier depth misleads, since a deep backlog of week-interval pages is fine while an hour of overdue news pages is not. Separate pools help too, since fetching is network-bound while parsing and extraction are CPU-bound, and the two scale on different signals. One deliberate architectural choice: fetchers stay stateless, with all coordination state living in the frontier and metadata store, so scaling is purely horizontal and any worker can process any eligible URL.

Follow-up questions to expect

• How and when do you recrawl? Per-URL next_crawl_at driven by observed change rate; the extractor feeding discovered URLs back into the metadata store keeps the loop alive.
• Can someone attack or exhaust the crawler? Crawler traps and giant auto-generated sites; defend with per-domain budgets, depth caps, and trap detection.
• Why a Bloom filter instead of a hash table? Constant-time membership at a fraction of the memory, with the false-positive direction being the safe one for this use case.
• Two different URLs return identical pages. What happens? Content checksum hits the dedupe store; store the mapping, skip the write.
• A domain starts returning 429s. What changes? That domain's token bucket slows, its retries back off with jitter, and the rest of the fleet is unaffected.
• A fetcher dies holding 50 URLs. What happens? Leases expire, URLs reappear in the frontier, dedupe absorbs any double-processing.

Leveling the answer

Related lessons

Design a Job SchedulerHard

The recurring-work sibling: definitions materialized into runs, dispatched to a worker fleet.

Design Webhook DeliveryHard

Retry budgets and per-target politeness, applied to endpoints you deliver to rather than fetch from.