Debug API Errors in Production
You're the on-call engineer at a photo-sharing app with about a million daily active users. At 14:09 UTC you get paged:
[PAGE] api-service error rate 42% (threshold 2%) - 5xx elevated across all endpoints
Support tickets are flooding in. Photos won't post and the upload spinner runs forever. Some users say the whole app is broken, that feeds won't load and they can't even log in. As far as you know, nobody has touched anything.
You have logs, metrics, dashboards, and a shell. Nothing runs in this round. You say what you'd look at, and the interviewer tells you what you'd see.
What's in the download
README.md. The interviewer's full script: the opening prompt, the incident timeline, and a data bank of what to reveal for each question a candidate might ask.api-service/. The service that got paged. Login, feed, likes, comments, and the upload endpoint, plus its pool and client configuration.tagging-service/. The service uploads depend on. It stores images and calls an external AI API for content-safety labels before a photo publishes.clients/. The mobile and web upload code.gateway/. The load balancer configuration.
Your task
Diagnose the outage, then say what you'd do about it while it's still happening.
The best way to practice this is with a partner: hand them the README, have them play every tool you ask for, and give them nothing for free. Working solo, treat the service directories as your evidence and write down what you'd want to see at each step before you go look for it.
What to focus on
- Scope before hypothesis. What exactly is failing, since when, and for whom. A page that says "elevated across all endpoints" is a summary, not a measurement, and the per-endpoint breakdown usually changes the shape of the problem.
- Questions that rule things out. Every question you ask should eliminate a class of cause. "Show me recent deploys" is a good question because a clean change log kills a whole family of theories. Free-associating through dashboards is not an investigation.
- The symptom that doesn't fit. In most real incidents one detail refuses to line up with the obvious story. Sitting with that detail instead of explaining it away is usually what separates the right answer from a plausible one.
- Trigger versus root cause. What started an incident and what turned it into an outage are frequently different things, and only one of them is under your control. An answer that stops at the trigger has stopped early.
The mitigation conversation
Diagnosing it is half the round. The other half starts when the interviewer tells you the upstream fix is thirty minutes out and asks what you're doing right now.
Order matters more than completeness here. Stopping the bleeding, protecting the endpoints that have no business being down, and telling users what's happening all come before any architectural improvement. Then you can talk about what would have prevented it, and about what happens in the five minutes after the dependency recovers, which is its own problem.
Using AI on this problem
There's no code to hand an agent, so use it the way you'd use a colleague on the incident call. Describe the symptoms and ask what classes of cause would produce this specific combination, then rule them out one at a time with the evidence you have.
Where AI genuinely helps is the retrospective: once you have a diagnosis, ask it to argue against you and name the evidence that would falsify your theory. Incident reasoning is unusually vulnerable to a story that fits most of the facts.