Skip to main content

Find Duplicate Files

Premium

You're asked to write a tool that finds duplicate files across a directory tree. The copies have different names and live in different folders, the way they accumulate in practice: downloads, backups, final_v2 (1). Your tool should group files whose contents are identical so someone can review the extras and reclaim the space.

You're given a filesystem abstraction and an empty function. Everything else is yours to design.

Starter code and solution
Python 3.11+ (standard library)
Download code

What's in the codebase

  • filesystem.py. A small filesystem abstraction with walk, size, and read_chunks, plus a real implementation and an in-memory one. The deduper talks to this, never to the OS directly, so it stays testable and swappable.
  • dedupe.py. Where find_duplicates(root, fs=...) lives. Currently a stub.
  • fixtures/sample_tree/. A committed sample tree with known duplicates. python3 fixtures/generate_fixtures.py recreates it if it goes missing.
  • tests/test_find_duplicates.py. Self-check tests that fail until you implement the stub.

Use the filesystem abstraction deliberately. It lets you test against a tree you construct instead of depending on whatever happens to exist on disk.

Your task

Traverse the tree under root and group files that are duplicates by content. Return a list of groups, where each group is the set of paths holding identical bytes. Files with no duplicate are not reported.

Before you code, get specific about what "duplicate" means and note your assumptions.

What to focus on

  • Pinning the definition. Are two empty files duplicates of each other? Does a symlink to a file count as a copy of it? What about two directory entries that are the same file on disk? None of these have obvious answers, and all of them will come up.
  • Cost awareness. The naive approach works on the sample tree and falls over on a real one. The interviewer will ask what happens at a million files, and a good answer already has the structure to accommodate it.
  • Testing that proves the result. A test with only two identical files will also pass for an incorrect implementation that groups by size alone.

Using AI on this problem

AI can produce a reasonable deduplication routine quickly, but it may silently choose policies for empty files, symlinks, and unreadable paths. Ask it to list those assumptions, then decide whether each one matches the contract you want.

Leveling signals

Mid-levelCorrect content-based grouping and a test that builds real files. Needs a nudge toward cost and toward the edge cases.
SeniorDesigns to avoid unnecessary work unprompted, streams rather than loading whole files, uses the filesystem seam deliberately, and raises the empty-file and symlink ambiguities themselves.
Staff+All of that, plus a crisp cost model at each tier, the distinction between a hash match and a proof of equality, identity by inode where it matters, failure modes like races and permissions as a matter of course, and a clear sense of when to stop optimizing.
Next Lesson