Latency, Cost, and Quality Tradeoffs

When OpenAI PMs are asked to lead the rollout of a new model into ChatGPT, the question is not just about launch planning. It is about whether you can reason through inference cost, latency budgets, and quality floors as a product decision. This lesson gives you that reasoning.

The latency-cost-quality triangle

Every inference decision lives inside a triangle. The reason to use this framing is that it makes something explicit: you cannot optimize all three simultaneously. Every lever you pull creates tension somewhere else. Candidates who understand this avoid two traps: proposing solutions that ignore cost, and proposing cost cuts that quietly destroy quality.

Moving toward quality usually means a larger model. Larger models are slower and more expensive. Moving toward lower latency usually means a smaller model or shorter context, which risks output quality. Moving toward lower cost means fewer tokens, more compression, or a smaller model, which circles back and affects the other two.

Your job is not to collapse this triangle. It is to define where your product should sit on it, and defend that decision with data.

The four concepts to own

1. Latency: know the two numbers

Interviewers at Nvidia and OpenAI push on this because it immediately separates candidates who have shipped AI products from candidates who have read about them.

Latency in LLMs is not one number. It is two distinct signals:

Time to First Token (TTFT) is the time from the user hitting send to the first token appearing on screen. This is the responsiveness signal. Users feel this viscerally.

Time Per Output Token (TPOT) is the time it takes to generate each token after the first. This controls how fast the response streams in once it starts.

They matter differently by use case:

Watch out for: Defaulting to "use a smaller model" as the answer to a latency problem. That is sometimes right, but the first question is where the latency is actually coming from. A bloated system prompt, slow retrieval from a vector database, or cold-start delays on serverless infrastructure all drive TTFT up. None of them is fixed by swapping models. Ask that question before recommending a solution.

2. Quality: define it before you optimize

Quality means: does the model produce accurate, relevant, useful output for this specific task? It is always relative to what you are asking the model to do.

Within a model family, larger models generally produce better outputs on complex reasoning tasks. But a smaller model fine-tuned on your domain can outperform a general-purpose flagship on a narrow task, and do so faster and more cheaply. The product judgment defines the quality floor before choosing the model, not after.

Quality degrades in predictable ways you should be able to name:

• Truncation. When the context window is exceeded, older content is dropped silently. The model loses memory of earlier turns or document context without alerting you.
• Hallucination under pressure. Aggressive context trimming strips the grounding that keeps responses accurate. The model fills gaps with plausible-sounding content.

3. Cost: Two layers PMs need to own

At companies like Nvidia and Apple, PM candidates are explicitly tested on their ability to connect infrastructure decisions to unit economics.

Cost per token is the direct compute or API cost for each token in and out. It scales with model size and usage volume. A chatbot that naively passes full conversation history on every call sees costs compound fast, because every token in the context window is charged on every single call.

Total cost of ownership (TCO) is the full picture: cost per token plus retrieval infrastructure, human review pipelines, engineering optimization time, and, for self-hosted deployments, fixed compute costs that need volume to justify.

Connect these costs to unit economics. The question to hold in your head: what is our cost per successful user interaction, and what is the revenue or retention value of that interaction? If those numbers are wildly out of balance, you have a problem to solve. You should be the one raising it.

4. The optimization levers

Interviewers weigh this section heavily because it shows whether you can translate diagnosis into action.

Model routing. Not every query needs your best model. Simple requests like rephrasing a sentence or answering an FAQ route to a smaller, faster model. Complex reasoning gets the flagship. The product judgment defines the quality floor for each interaction type. Get this right, and you can cut costs substantially without users noticing.

Batching. Group multiple user requests together and process them in a single inference pass. GPU utilization goes up, cost per request goes down. The trade-off is added queue latency before a request is processed.

Context management. Every token in your context window costs money on every call. Summarizing earlier conversation turns, truncating low-relevance chunks, and using a memory layer. These reduce costs without destroying quality if you have clearly defined what quality means. That definition is yours to own.

Caching. Responses to identical or near-identical queries can be returned without a new inference call. Works well for deterministic content, such as FAQ answers. Works poorly for open-ended conversation. Semantic caching extends this to similar but not identical queries.

Common pitfalls

Treating latency as a single number. TTFT and TPOT are distinct signals that matter differently depending on the use case. Conflating them signals surface-level reasoning.

Defaulting to "use a smaller model" for latency problems. Ask where the latency is actually coming from before recommending a solution.

Deferring cost decisions to engineering. "We would work with the team to optimize costs" is not a PM answer. Own the cost-value equation and name the thresholds that would trigger optimization work.

Skipping the eval question. Anytime you argue for a quality-cost tradeoff, you need a way to measure quality. "The output will be better" without a measurement mechanism does not clear the bar set by senior management at OpenAI, Nvidia, or Apple.