Summary
We are looking for an AI Engineer with a strong machine learning (ML) background and hands-on experience building modern AI systems. This role is best suited for someone who started in ML, applied modeling, or NLP, and later expanded into large language models (LLMs) and agentic AI systems. We are looking for someone with an evaluation-first mindset who believes AI systems should be designed with clear success criteria, testing methods, and monitoring plans from the start.
The ideal candidate brings solid ML foundations, experience working with third-party and open-source LLMs, and practical experience building multi-step AI workflows for real business problems. This background helps ensure these solutions are accurate, reliable, scalable, and grounded in sound evaluation practices. Humility, accountability, and a growth mindset are must-haves for this role. The right candidate is comfortable admitting mistakes, learning from feedback, and adjusting quickly when evidence shows a better path.
Why This Role Matters
This role matters because we need more than someone who can build AI features. We need someone who can build AI systems in a thoughtful and reliable way. That means starting with a clear plan for how quality, risk, and business impact will be measured, and carrying that through design, launch, and ongoing improvement. This role will also help strengthen how we run AI in practice, with solid MLOps and LLMOps across ML, LLM, and agentic AI systems.
What You’ll Do (Essential Responsibilities):
- Design, build, and deploy AI solutions powered by ML, LLMs, and agentic AI systems that address clear business problems.
- Define evaluation strategies upfront for each use case, including task success metrics, offline and online evaluation plans, error analysis, and production monitoring requirements.
- Build and improve LLM-based systems using prompt engineering, retrieval-augmented generation, and multi-step workflows.
- Apply MLOps and LLMOps practices, including experimentation, versioning, observability, alerting, model and prompt evaluation, and continuous improvement in production.
- Partner closely with product, engineering, and business stakeholders to prioritize AI use cases and align on success metrics, operational needs, and delivery timelines.
Who You Are (Soft Skills & Attributes):
- You bring an evaluation-first mindset and believe AI systems should not be designed or implemented without a clear plan to measure quality, risk, and business impact.
- You are thoughtful and practical, with sound judgment about when to experiment, when to simplify, when to stop, and when to productionize.
- You bring humility, own mistakes quickly, and use feedback and new evidence to improve your thinking, your systems, and your results.
- You work well with product and business stakeholders, helping turn ambiguous business problems into clear AI approaches, measurable success criteria, and realistic rollout plans.
What You'll Need (Required Knowledge, Skills & Abilities):
- Bachelors’ degree in Computer Science or related field, and knowledge, skills and abilities typically associated with 6+ years of total relevant experience across ML and modern AI systems including:
4+ years of hands-on experience in machine learning
2+ years building LLM-based applications, 1 of which consists of building agentic AI systems as part of that LLM application experience
- Expertise in ML, applied modeling, or NLP, including model development, evaluation, experimentation, and error analysis
- Hands-on experience building LLM-based applications, including context engineering, retrieval, evaluation frameworks, and model fine-tuning.
- Experience designing and implementing agentic AI systems, including multi-step workflows that use planning, memory, handoffs, tool orchestration, and human-in-the-loop review.
- Strong experience with MLOps for ML systems, including model lifecycle management, deployment, monitoring, retraining, and production success metrics.
- Strong experience with LLMOps for LLM-based applications, including prompt and workflow versioning, retrieval and response evaluation, observability, guardrails, and continuous improvement in production.
Advanced Python skills and experience taking AI solutions from prototype to production while balancing quality, latency, cost, reliability, and maintainability.
What Would be Nice (Preferred Skills & Experience):
- Experience with vector and graph databases, retrieval quality tuning, and domain-specific optimization for LLM-based systems.
- Experience with platform design, reusable components, and internal tooling that improves AI development speed and reuse.
- Experience with cloud-based AI deployment and scalable serving infrastructure for ML or LLM systems.
Similar Jobs
What you need to know about the Austin Tech Scene
Key Facts About Austin Tech
- Number of Tech Workers: 180,500; 13.7% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Dell, IBM, AMD, Apple, Alphabet
- Key Industries: Artificial intelligence, hardware, cloud computing, software, healthtech
- Funding Landscape: $4.5 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Live Oak Ventures, Austin Ventures, Hinge Capital, Gigafund, KdT Ventures, Next Coast Ventures, Silverton Partners
- Research Centers and Universities: University of Texas, Southwestern University, Texas State University, Center for Complex Quantum Systems, Oden Institute for Computational Engineering and Sciences, Texas Advanced Computing Center


.png)
