Product Manager (Devices team) Interview Experience

I framed it as an offline eval problem around intent-to-tool routing. I said I'd build a representative prompt set, use LLM-as-a-judge with human spot checks, and score things like intent extraction accuracy, correct tool selection, and end-to-end task completion. I also wanted failure buckets for ambiguous intent, wrong tool choice, and bad action parameters so we weren't hiding everything inside one blended metric. The interviewer was clearly testing whether I actually understood AI evals, not whether I could recite generic PM metrics.

I didn't argue for waiting for a perfect model. I proposed a narrow MVP with intentional friction: preview-before-send, explicit confirmation, constrained action scopes, and clear user edit points before anything irreversible happens. My point was that one bad high-stakes action can permanently break trust, so the UX has to absorb today's model unreliability. I treated it as a competitive problem too: ship something useful now, but only where failure is sandboxed and the user stays in control.

I said thumbs up and thumbs down are way too blunt. I'd break it into tool-use success, hallucination rate, inappropriate response rate, and actual task completion in live sessions, then compare offline eval performance with online behavior to see where the model falls apart in reality. I also talked about richer logging, targeted human review on risky slices, and using LLM-as-a-judge carefully instead of treating it like ground truth. It felt like they wanted someone who understood both eval design and the limits of the eval itself.

I used a past consumer AI app and walked through how I pick the smallest wedge that solves a real problem fast. I said I prioritize by user value, technical readiness, and whether the thing will actually move growth or retention in the first few months. I also explained how I sequence features and cut anything that slows time to market without changing the core experience. They cared less about roadmap theater and more about whether I could tie product choices to GTM and real outcomes.

I told a story about a launch where a biometric fingerprint sensor had an unacceptably high false rejection rate right before deadline. I pulled the right cross-functional people together, dug into why it was failing, and realized we couldn't bet the whole user experience on shaky hardware. To unblock things, I shifted the primary authentication path toward a face-unlock style onboarding approach that was performing better, so we could still ship a strong biometric experience while reducing the fingerprint risk. They seemed to like that I treated it as both a technical and UX problem.

I set the goal as growing market share, with DAU as the headline outcome, and then worked backward from why someone would switch or keep coming back. My short-term view was to improve stickiness first with things like robust notifications, then invest in conversation quality through better memory and stronger tool calling for web search and document editing. I wasn't making a vague 'just be smarter' argument. I was pushing for a few high-impact, low-effort moves that make Gemini more useful in daily workflows faster.

I started by splitting the magical users from the disappointed majority instead of averaging the feedback. I wanted to know which learning jobs, inputs, and moments drove delight, and whether users even realized audio or video was available. My fixes were to tighten onboarding, make the multimodal affordances obvious, and improve personalization before jumping to a bigger-model answer. When he pushed on the technical path, I compared fine-tuning versus RAG directly: fine-tuning is slower, costlier, and needs labeled data, while RAG is faster and cheaper if users can plug in course material immediately.

I used a past wellness and nutrition AI app as my example and walked through the 0-to-1 thinking end to end. I focused on the core user pain point, what I learned once real people touched it, and how quickly 'cool AI' features can distract from the one job the product has to do well. I also talked about the metrics I used to judge whether it was actually working, not just whether people tried it once. The subtext of that round was very much, 'Can you make AI feel useful to normal people?'

I talked about using the interface to make today's model limitations feel manageable instead of pretending they don't exist. That means making multimodal entry points obvious, setting the right expectation when the system is thinking, and using the UX to reduce confusion around what the model can do well versus poorly. I also said a lot of these products fail because users never discover the behavior that would actually make the experience feel magical. So onboarding and affordances matter way more than PMs usually admit.