File Uploader for AI Chat App
You're asked to design file uploads for an AI chat app. A user uploads a 200-page PDF and expects to ask questions about it soon afterward. Before that can happen, the system must store and scan the file, parse its contents, split the text into chunks, create embeddings, index them, and report when the file is ready. Let's model that work as a durable, staged pipeline.
Clarify the requirements
- Which formats and sizes? Consider PDFs, Office documents, images that require OCR, and plain text. Set a concrete limit such as 512 MB and design the upload path around it.
- When can the user query the file? Show visible progress while the file is "processing," then mark it "ready." Ask whether users should be able to query the first 50 pages while the rest are still processing because partial availability changes the retrieval contract.
- Is the system multi-tenant? Assume yes. One tenant's thousand-file import must not delay another tenant's single memo, and retrieval must never leak data across tenants.
- Retention and deletion? Users delete files, and deletion must reach every derived artifact: chunks, embeddings, index entries, caches.
The system must not lose an acknowledged upload. Processing may take seconds or minutes, but users need visible progress, and the request path must never wait for the processing work to finish.
Back-of-envelope numbers
Assume 1M uploads/day averaging 5 MB per file.
- Upload rate: average, spiky enough on bulk imports to plan for 20×
- Raw storage:
- Chunking: a 200-page PDF yields roughly 600 chunks, so at 100 chunks per file on average, sustained into the vector index
- Embedding capacity: at per GPU instance that's a handful of instances, but bulk imports demand a queue rather than synchronous embedding
The upload rate is modest, while embedding creates the main throughput constraint. Keep the API thin and scale the processing pipeline independently.
High-level architecture
- Client. Requests an upload, receives a presigned URL, and sends resumable parts directly to object storage. The API tier never handles the file bytes.
- Object storage. The durable source of truth for originals. An upload-complete event starts the pipeline.
- Stage queues. Give each stage its own queue so it can scale and retry independently.
- Pipeline workers. Use separate pools for malware scanning, parsing or OCR, chunking, GPU embedding, and indexing. Each stage records a checkpoint in metadata.
- Vector index. Stores chunk embeddings and a lexical index for hybrid retrieval (what is RAG?). Every entry includes tenant and file IDs.
- Job metadata. Stores each file's status, stage progress, attempt count, and error codes for the progress UI.
This is the ingest half of a retrieval-augmented generation system. On the query side, the retriever embeds the user's question, searches the vector index for similar chunks belonging to that tenant, and passes the results to the model as context. The model can then answer from the uploaded text and cite its document and page.
An agent can use retrieval as a tool: it decides when to search the user's files, issues a query, reads the results, and continues its loop (agentic AI architectures). Chunk size, the embedding model, and tenant filters all affect what that tool can return. What is RAG? covers the query side in more depth. This design focuses on building the index with the machinery from async jobs and workers.
Deep dive 1: the pipeline as checkpointed stages
Do not process the entire file as one large job. If embedding fails near the end, a monolithic job would repeat parsing and OCR. Give each stage its own job, queue, retry policy, and checkpoint. A crash then repeats only the failed stage. Store parsed text in object storage and chunks in a table so downstream stages always have durable inputs.
Each stage fails differently. A corrupted PDF is not retryable, so mark the file as failed and show the user why. An OCR timeout on a 500-page scan can retry with a larger budget on a bigger instance. Retry an embedding-service 429 with backoff and apply downstream rate limits. A poison file that repeatedly crashes the parser should exhaust its budget quickly and move to a dead-letter queue with diagnostics. Sandbox parsers so a malformed file cannot crash or exploit the worker pool.
Deep dive 2: idempotency and deduplication
Every stage runs at least once, so every stage must be safe to repeat. Chunking uses deterministic IDs such as (file_id, chunk_seq) and upserts. Embedding checks for an existing vector before recomputing, and indexing upserts by chunk ID. Replaying a stage then converges on the same result instead of creating duplicates. This applies reliable execution to each stage.
Deduplicate whole files by hashing their contents during upload. If ten teammates upload the same handbook, store and index one copy, then give each user a reference. At 5 TB per day, content addressing can materially reduce cost. The content hash also helps with updates: compare chunks in a modified document and re-embed only the ones that changed instead of rebuilding the full index.
Deep dive 3: progress, partial availability, and the user contract
Report progress for each stage instead of showing only "Processing...". The metadata store already tracks pages parsed and chunks embedded, so expose those checkpoints to the chat UI through a status endpoint or push channel.
Decide whether a file becomes usable before processing finishes. Chunks are searchable as soon as they reach the index, so the user could query the first pages while the rest continue processing. If you allow this, retrieval must include only chunks with indexed status, answers must disclose that coverage is partial, and the completion event must remove that warning.
Deep dive 4: tenancy, deletion, and the model-upgrade backfill
Tenancy affects both pipeline fairness and retrieval isolation. Use per-tenant concurrency caps and queue weights so a bulk import cannot starve everyone else. Tag every chunk with tenant and ACL metadata, then filter by the requesting user's current permissions at query time because permissions stored only at indexing time can become stale.
Deletion must remove the original, parsed text, chunks, vectors, index entries, and any semantic-cache entries derived from the file. Run deletion as its own at-least-once pipeline and use a periodic sweep to find orphaned artifacts. Deleting only the metadata row leaves user data behind.
An embedding-model upgrade needs a versioned index and a backfill pipeline. Embeddings from different models occupy different vector spaces, so mixing old and new vectors quietly damages retrieval quality. Re-embed the corpus into a new index version, move each tenant to it after that tenant's backfill completes, and keep the old version available until the cutover. This follows the backfill discipline used in larger data pipelines.
Leveling signals
Related lessons
The query side this pipeline feeds, including retrieval quality and vector store maintenance.
The same async job machinery pointed at large user-initiated exports rather than ingestion.