Web Crawler

You're asked to build a small web crawler from scratch. Starting from one page on a toy website, follow its links and report every unique page you can reach.

Companies that build search, retrieval, and agent products use versions of this problem. The traversal algorithm is simple. Most of the work lies in defining URL identity, handling failures, structuring the code cleanly, and later adding concurrency without duplicating work.

What's in the codebase

• crawler.py. The harness you extend, with crawl, normalize_url, extract_links, same_domain, a LinkExtractor, and main. Filling in its TODOs is the exercise.
• site/. A local copy of the toy website's HTML. The same pages are hosted live, which is the crawler's default seed.
• serve.py. Optional. Serves site/ locally if you'd rather crawl offline.
• tests/test_web_crawler.py. Optional self-checks. They serve the local copy on their own background server, so they run without network access.

Out of the box the crawler raises NotImplementedError. That's the expected starting state.

Your task

Starting from the seed, fetch each page, extract its <a href> links, and follow them. Return and print the set of unique URLs you crawled.

The toy site is built to punish assumptions. Pages link back to each other and repeat the same navigation on every page. The same page appears under query-string and fragment variants that should collapse to one entry. Links are a mix of relative and absolute. Some point off-domain.

What to focus on

• Defining the contract before coding. What counts as "the same URL" is the central design decision here, and it has no single right answer. Pick one and say why.
• Structure. Extraction, normalization, the same-domain check, and the traversal loop want to be separable. The concurrency follow-up is much easier if they are.
• Robustness. A 404, a non-HTML response, or a timeout on one page should not end the crawl.

The concurrency follow-up

If you're asked to parallelize the crawler, identify the shared state, the race condition, and the stopping condition before you write code. Workers can add more work while other workers are running, so an empty queue does not necessarily mean the crawl is finished.

Do not hold a lock across network I/O, because that serializes the crawler while making it look concurrent. Mark each URL as seen before fetching it so two workers cannot fetch the same page.

Using AI on this problem

AI can help with the URL parsing API or a threading skeleton. Do not accept a normalization function you cannot explain. If you use AI for the concurrent version, inspect where it acquires locks and which operations happen while the lock is held.

Leveling signals