Design App that Downloads User Data

You're asked to design the "download your information" feature: a user requests a copy of everything the platform holds about them, and some time later they get a link to a downloadable archive. It should gather data from many different services, handle accounts ranging from a few megabytes to hundreds of gigabytes, survive failures partway through, and show the user meaningful progress along the way.

The request itself takes milliseconds. The work behind it can take hours, which means the entire design is about what happens after you've already told the user "we'll email you when it's ready."

Clarifying the requirements

• What's included? Posts, photos, messages, comments, likes, ad interactions, login history. Each lives in a different service with a different API and a different volume profile.
• What's the deadline? Regulations typically require fulfillment within days, not minutes. That generous SLA is what makes a fully asynchronous design acceptable.
• What format? A zip containing JSON or HTML per category is typical. Format matters less than the fact that it's one archive assembled from many sources.
• How large can an export get? Ask explicitly. The answer, anywhere from megabytes to hundreds of gigabytes for a heavy user with fifteen years of photos, drives the chunking and storage decisions.
• How sensitive is this? Extremely. An export is every piece of data about one person in a single file, so access control and link expiry are functional requirements, not afterthoughts.

Back-of-envelope numbers

• Request rate: $3\text{B users} \times 0.01\% \approx 300\text{k exports/month} \approx 0.1/\text{sec}$, which is trivial
• Archive volume: $300\text{k exports} \times 1\text{ GB avg} \approx 300\text{ TB/month}$, with heavy accounts well above the average
• At a 30-day retention window, steady-state storage is roughly 300 TB

The request rate being negligible is itself a finding worth stating. Nothing about this system needs to be fast; everything about it needs to be reliable and resumable.

High-level architecture

Deep dive 1: parent and child jobs

The central design decision is not to treat an export as one job. A single job that gathers everything is unresumable: a failure at 90% means starting over, and a heavy account might never complete before something goes wrong.

Instead, the parent job fans out into one child job per data category, and the parent's state is derived from its children. That gives you three things at once. A failure in the photos extraction retries only photos. Progress is computable, since twelve of fifteen children complete is a real percentage to show the user. And categories extract in parallel, so total time is set by the slowest category rather than the sum of all of them.

For very large categories, children subdivide further. Fifteen years of photos becomes one child job per time range, each writing its own chunk to object storage.

Deep dive 2: surviving failures partway through

Every child job must be independently retryable, which means each must be idempotent. A child that re-runs should overwrite its output chunk rather than appending to it, so a retry produces the same archive as a clean first run. Keying each chunk by (job_id, category, range) makes overwriting the natural behavior.

Beyond that, the standard machinery from the pattern applies:

• Leases with heartbeats, so a worker that dies mid-extraction releases its child job for another worker instead of stranding it.
• Classified retries. A rate-limit response from a source service is retryable with backoff; a permanently deleted account is not.
• A checkpoint per child, so a category that's 80% extracted resumes near where it stopped rather than from zero.
• Per-source rate limiting. The photo service was not built to serve one customer's entire history as fast as possible. Throttle extraction so an export never degrades the live product.

Deep dive 3: assembly and delivery

Once every child succeeds, the finalizer packages the chunks. The important property is that it should never load the archive into memory. It streams chunks from object storage into a zip written back to object storage, so a 200 GB export costs the same memory as a 200 MB one.

Delivery has its own requirements, and they're mostly about security:

• Time-limited signed URLs. The link expires, typically in a few days, and is single-use or tied to an authenticated session.
• Re-authentication before download. The archive is every piece of data about a person, so requiring a fresh login before handing it over is proportionate.
• Automatic deletion. Archives are deleted after the retention window, which caps both storage cost and the blast radius if a link leaks.
• Encryption at rest, and optionally a password-protected archive.

Deep dive 4: Reporting progress to the user

Progress reporting is what users actually experience, and it comes for free from the parent-child structure. The parent's completion percentage is the fraction of children done, optionally weighted by expected size, so photos count for more than login history. Store progress in the job record and let clients poll, rather than pushing.

What breaks first, in order: a very heavy account can generate an export large enough that the finalizer's packaging step becomes the bottleneck, which is why chunk streaming matters; a source service can rate-limit you into hours of backoff, which is why per-source throttling is deliberate rather than reactive; and abandoned exports accumulate storage, which is why deletion is scheduled rather than manual.

Rate-limit export requests per account. A user requesting exports repeatedly consumes real resources. Rate-limit export requests per account, and deduplicate by returning the existing archive if one was generated recently.

Common pitfalls

• Treating the export as a single job. A failure anywhere means starting from zero, and progress becomes unreportable.
• Holding the archive in memory. Works for a test account, fails for a real one.
• No idempotency on child jobs. A retry produces a corrupted archive with duplicated content.
• Ignoring source rate limits. One user's export degrades the live product for everyone else.
• Permanent download links. An unexpiring URL to a complete personal data archive is a serious exposure.

Leveling the answer

Related lessons