Product Manager, Toast IQ Evaluation

Toast creates technology to help restaurants and local businesses succeed in a digital world, helping business owners operate, increase sales, engage customers, and keep employees happy. At the center of this vision is Toast IQ: a single, unified assistant that orchestrates specialized agents across the Toast ecosystem. The Toast IQ Quality team makes sure Toast IQ delivers useful, reliable help to restaurant operators.

About the Role

As Product Manager, Toast IQ Evaluation, you’ll define what good help looks like for a restaurant operator and build the evidence that tells us whether Toast IQ delivers it. You’ll study real usage, work directly with operators, and turn what you learn into datasets, clear quality standards, and priorities for improvement. Your work will help teams across Toast understand where the assistant succeeds, where it falls short, and identify new opportunities for improvement. This is a hands-on product role. Much of your week will be spent reviewing sessions, labeling examples, investigating failures, and building better ways to do that work. Restaurant judgment, attention to detail, and the ability to turn evidence into action are central to the job. You’ll be the team’s first restaurant domain expert, working alongside engineers from one prioritized backlog and reporting through Product.

A Day in the life (Responsibilities):

• Understand the operator’s experience. Review real Toast IQ sessions every week. Describe what went wrong in plain operator language. Work directly with operators, including in live sessions, to check that our evaluations reflect what matters to them.
• Find and prioritize recurring failures. Group observations into failure modes, measure how often they occur, and assess their impact on operators. Use that evidence to identify the problems most worth solving.
• Build trustworthy evaluation datasets. Create and maintain labeled examples covering common workflows, edge cases, and known failures.
• Verify that improvements hold up. Validate fixes in real sessions and turn important production failures into regression cases that catch the same problems in future releases.
• Help teams across Toast evaluate their work. Lead reviewer calibration sessions and help product teams create and maintain evaluation cases for their capabilities.
• Turn findings into product decisions. Surface tools that aren’t delivering value, intents Toast IQ can’t handle, and workflows that need a product team’s attention. Make the operator’s problem and its impact clear enough to act on.
• Build tools that make the work better. Use AI coding tools to create faster review interfaces, automate recurring analysis, monitor common failures, and prototype fixes. Look for opportunities to improve every step from session review through validation.

What you'll need to thrive (Requirements)

• Restaurant knowledge: You’ve worked in a restaurant or closely with operators. You understand how decisions about menus, labor, sales, and inventory play out during a busy Friday night. This matters more than your years of product experience.
• Care for detailed work: You’re comfortable reading hundreds of transcripts, labeling examples by hand, and keeping datasets accurate and current.
• Comfort with structured data: You can investigate data using a database, queries, or an AI coding tool and check that the results are trustworthy.
• Precise thinking and writing: You can turn an ambiguous observation into a clear failure definition or scoring rule that someone else can apply consistently.
• Clear communication: You can explain what happened, why it matters to operators, and what should change in language that engineers, product teams, and leadership can use.
• Curiosity about AI systems: You want to understand how large language model systems fail and are willing to learn new evaluation methods on the job. 
• Familiarity with Toast's products: You’ll work closely with product teams across Toast, and knowing their products gives you a leg up in advising them.

Product management experience is highly valued, but it is not required. We care about the practice: identify a problem, understand it, explore solutions, build, measure, and iterate. If you’ve done that work under another title, we’d like to hear from you.

AI at Toast

At Toast, one of our company values is that we're hungry to build and learn. We believe learning new AI tools empowers us to build for our customers faster, more independently, and with higher quality. We provide these tools across all disciplines, from Engineering and Product to Sales and Support, and are inspired by how our Toasters are already driving real value with them. The people who thrive here are those who embrace changes that let us build more for our customers; it’s a core part of our culture.

Our Total Rewards Philosophy

We strive to provide competitive compensation and benefits programs that help to attract, retain, and motivate the best and brightest people in our industry. Our total rewards package goes beyond great earnings potential and provides the means to a healthy lifestyle with the flexibility to meet Toasters’ changing needs. Learn more about our benefits at https://careers.toasttab.com/toast-benefits.