AI Evals Technical Lead

About you:

• feel an unshakeable pull to work on agentic AI

• can usually break an AI or a piece of software in under a minute (if you want to)

• are a skilled developer yourself

• always develop an interest in the subject matter you’re building tests for, and are eager to do the same for the industrial products that run the world

• believe in manifesting the future of physical engineering

About P-1 AI:

We are building an engineering AGI. We founded P-1 AI with the conviction that the greatest impact of artificial intelligence will be on the built world—helping mankind conquer nature and bend it to our will. Our first product is Archie, an AI engineer capable of quantitative and spatial reasoning over physical product domains that performs at the level of an entry-level design engineer. We aim to put an Archie on every engineering team at every industrial company on earth.

Our founding team includes the top minds in model-based engineering, deep learning, and industries that are our customers. We just closed a $23 million seed round led by Radical Ventures that includes a number of other AI and industrial luminaries. We invite you to join our team of the world’s best engineers and AI researchers, building AI’s most impactful use case.

About the Role:

In this role, you’ll be responsible for the evals that we use to ensure that Archie is learning and retaining the skills needed to successfully perform its engineering work, and benchmark it against industry skill expectations. Working within a small, tightly-knit team of high-performers, you’ll be principally responsible for clearly defining, implementing, and validating these, including input from our engineering experts and industrial partners. You’ll also be responsible for translating these eval tests into multiple formats for use with different types of AI and non-AI systems and agents.

This role is remote and you can be based anywhere in the US or Canada, where you must have existing work authorization. You will be expected to travel to our San Francisco office for co-working sessions approximately one week out of every six. If you are already located in the SF Bay Area or are interested in relocation, you are of course welcome to work out of our SF office. Our AI team is based in the SF office, so there would be some benefit to you being in-office at least part of the time.

Responsibilities:

• Implement the system for organizing, transforming, running, grading, and reporting on eval benchmarks.

• Design and execute the process by which we develop and QA our evals, incorporating contributions from our own engineering team, industrial partners, and subject-matter experts.

• Ensure that evals run effectively within our CI/CD system, continuously benchmarking our evolving AI platform and the experiments we’re performing around it.

• Create methods for detecting and testing for common quality challenges of AI, including hallucinations, undesirable stochasticity, and regressions.

• Be a technical leader in the consistent implementation and organization of automated tests across other areas of our technology stacks.

Skills and Experience:

• Experience in constructing comprehensive test suites for software and/or AI systems, including coordinating the contributions of others.

• Experience designing metrics to evaluate systems and visualize their performance, including differences across successive generations.

• Experience in developing, managing, and running evals against LLM-based systems is a strong plus.

• Good communication skills with a variety of stakeholders (AI researchers, domain experts, application developers).

• Proficiency in Python programming, complex modules and modern software development tools and practices (Git, CI/CD, etc.).

• Ability to thrive in a fast-paced, dynamic startup environment.

Interview Process:

• Initial screening - with Head of Talent (30 mins)

• Hiring manager interview - with co-founder & Head of Engineering (45 mins)

• Programming interview - with member of technical staff & Head of Engineering (60 mins)

  • bring your own dev environment and tools

• Culture fit / Q&A - with co-founder & CEO (45 mins)