Lead QA automation for AI/LLM systems: define validation criteria for non-deterministic models, build automated and performance tests, validate RAG systems, perform API and integration testing, ensure release readiness and trace defects to data/models/requirements.
N-iX is looking for a Senior QA Automation Engineer to lead the validation and verification strategies for EDGE Group’s AI transformation. You will be responsible for defining "what good looks like" for non-deterministic AI systems, ensuring that Large Language Models (LLMs) and predictive engines meet the strict reliability standards of the defense sector.
Qualifications
- 5+ years in QA Automation
- Experience testing data-driven systems, AI, or ML models
- Strong analytical skills for defining test criteria in non-deterministic systems
- Ability to collaborate with engineers, data scientists, and product teams
- Experience in regulated industries (defense, aerospace) will be a plus
- English level - at least Upper-Intermediate
Technical Requirements
- Strong experience in Python and test automation frameworks (Pytest, Selenium/Playwright, API testing)
- Experience with performance testing tools (JMeter, Locust, K6)
- Familiarity with LLM evaluation tools (e.g., DeepEval, TruLens)
- Knowledge of SQL and data validation tools
- Experience with CI/CD pipelines (GitLab) and version control (Git)
- Exposure to test reporting tools (Allure, TestRail)
Key Responsibilities1. AI & LLM Testing
- Build automated tests to evaluate AI outputs for accuracy, consistency, and reliability
- Validate Retrieval-Augmented Generation (RAG) systems using metrics like relevance and correctness
- Create regression tests to detect changes in AI behavior (prompt drift)
- Test integrations between AI systems and enterprise platforms (e.g., SAP)
- Perform API testing, including security and access control validation
- Design performance tests for latency, scalability, and system throughput
- Align test cases with system requirements and user needs
- Support release readiness through structured testing reviews
- Track and manage defects, ensuring traceability to data, models, or requirements
We offer*:
- Flexible working format - remote, office-based or flexible
- A competitive salary and good compensation package
- Personalized career growth
- Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
- Active tech communities with regular knowledge sharing
- Education reimbursement
- Memorable anniversary presents
- Corporate events and team buildings
- Other location-specific benefits
*not applicable for freelancers
What you need to know about the Austin Tech Scene
Austin has a diverse and thriving tech ecosystem thanks to home-grown companies like Dell and major campuses for IBM, AMD and Apple. The state’s flagship university, the University of Texas at Austin, is known for its engineering school, and the city is known for its annual South by Southwest tech and media conference. Austin’s tech scene spans many verticals, but it’s particularly known for hardware, including semiconductors, as well as AI, biotechnology and cloud computing. And its food and music scene, low taxes and favorable climate has made the city a destination for tech workers from across the country.
Key Facts About Austin Tech
- Number of Tech Workers: 180,500; 13.7% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Dell, IBM, AMD, Apple, Alphabet
- Key Industries: Artificial intelligence, hardware, cloud computing, software, healthtech
- Funding Landscape: $4.5 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Live Oak Ventures, Austin Ventures, Hinge Capital, Gigafund, KdT Ventures, Next Coast Ventures, Silverton Partners
- Research Centers and Universities: University of Texas, Southwestern University, Texas State University, Center for Complex Quantum Systems, Oden Institute for Computational Engineering and Sciences, Texas Advanced Computing Center
