Build and validate challenging codebase-exploration tasks for AI agents using production Python repositories. Responsibilities include investigating runtime behavior and issue history, rewriting questions, refining rubrics and foils, running validation workflows, documenting solutions, and addressing expert review. The role is fully remote, asynchronous, contract-based, and task-paid, with work performed in a dedicated Studio environment.
This role is for one of our clients
Compensation: $130 per task
We are seeking experienced Python engineers to build the dataset that teaches AI agents to reason about real codebases.
Codebase Q&A is a reinforcement-learning environment for training AI agents to explore and reason about software repositories they have never seen before. You will build difficult codebase-exploration tasks: high-level engineering questions about production Python repositories that cannot be answered by reading a single file, and that require runtime evidence to resolve — the kind of question a senior engineer answers by actually investigating the system.
This track works on widely deployed Python codebases including Django, Twisted, Scrapy, SymPy, mypy, pip, coverage.py, mitmproxy, Certbot, Tornado, Werkzeug, sqlfluff and Meson.
Contract, task-based work paid at $130 per approved task, performed remotely in our Studio.
Requirements
Key Responsibilities
- Select tasks from a pre-validated pool of Python repositories, each pinned to a specific git commit and arriving with an engineering question, positive and negative rubrics, foils, and a golden solution
- Explore the repository until you genuinely understand the subsystem in question, including the relevant PR or issue history
- Rewrite the task's question so it defeats frontier coding agents while remaining well-defined and fairly answerable
- Adjust rubrics and foils so they reward correct reasoning and reject plausible-but-wrong answers
- Run the full validation loop in Studio — Check, Golden, Run, Analyze — and iterate until every stage passes
- Document your work in the task README and submit for expert review, addressing reviewer feedback on returned tasks
Ideal Qualifications
- 3+ years of professional software engineering experience, with substantial production Python
- Demonstrated ability to navigate a large, unfamiliar Python codebase and explain how it behaves at runtime — not just what the source says
- Fluency in Python internals that matter in real systems: import machinery, descriptors and metaclasses, async and event loops, packaging, and C-extension boundaries
- Experience with web frameworks, developer tooling, type checkers, or scientific/numeric libraries is highly relevant — the task mix is weighted toward architecture and system design (41%) and code onboarding (31%)
- Precise written English: the questions and rubrics you write are the product
- Comfortable with git at commit level, containerized environments, and command-line tooling
Working Details
- Fully remote and asynchronous, with expert team leads across US, India, and Nigeria time zones
- Work is performed in our Studio; time is tracked with Insightful, and platform access is provisioned through Okta
- Onboarding requires identity verification, a background check, signed Terms of Work and CIIAA, a tax form, and passing a short calibration quiz before task access is granted
- Payment is per approved task; every task is reviewed by a second engineer and a super reviewer before acceptance
Application Process
- Submit your resume or relevant technical background to get started
- Qualified applicants may be asked to complete a brief technical assessment or provide additional information
We consider all qualified applicants without regard to legally protected characteristics and provide reasonable accommodations upon request.
Similar Jobs
Artificial Intelligence • Fintech • Insurance • Marketing Tech • Software • Analytics
Handles inbound calls and warm leads, consults customers on insurance needs, recommends appropriate Property and Casualty coverages, and converts prospects into policyholders. The role includes paid training and licensing support, customer-focused sales, lead conversion, schedule flexibility, and remote work. Representatives must meet state licensing requirements, maintain a professional home workspace, and work four weekdays plus one weekend day.
Top Skills:
PcWired High-Speed Internet
Cloud • Software
Lead the architecture, operation, and reliability of Cisco ThousandEyes’ FedRAMP-compliant AWS GovCloud platform. Build resilient cloud-native services using infrastructure as code, maintain high availability and security for high-volume data streams, automate compliance controls and auditing, and optimize multi-region infrastructure. Collaborate with engineering and security teams, lead technical initiatives, address operational risks, and establish engineering standards for regulated environments.
Top Skills:
AWSKubernetesPythonTerraform
.png)
Artificial Intelligence • Big Data • Software • Analytics • Business Intelligence • Big Data Analytics
The Director of Revenue Operations will build and lead Hex’s RevOps center of excellence, owning compensation, territory design, capacity planning, quota setting, pipeline modeling, forecasting, and recurring operational programs. The role partners with Sales, Finance, Recruiting, Legal, Systems, and executive leadership to automate processes, improve forecast accuracy, support hiring plans, and report on go-to-market performance.
Top Skills:
AIAnaplanCaptivateiqCompensation ToolsCRMData ModelsFullcastPlanning ToolsSalesforceSpiff
What you need to know about the Austin Tech Scene
Austin has a diverse and thriving tech ecosystem thanks to home-grown companies like Dell and major campuses for IBM, AMD and Apple. The state’s flagship university, the University of Texas at Austin, is known for its engineering school, and the city is known for its annual South by Southwest tech and media conference. Austin’s tech scene spans many verticals, but it’s particularly known for hardware, including semiconductors, as well as AI, biotechnology and cloud computing. And its food and music scene, low taxes and favorable climate has made the city a destination for tech workers from across the country.
Key Facts About Austin Tech
- Number of Tech Workers: 180,500; 13.7% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Dell, IBM, AMD, Apple, Alphabet
- Key Industries: Artificial intelligence, hardware, cloud computing, software, healthtech
- Funding Landscape: $4.5 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Live Oak Ventures, Austin Ventures, Hinge Capital, Gigafund, KdT Ventures, Next Coast Ventures, Silverton Partners
- Research Centers and Universities: University of Texas, Southwestern University, Texas State University, Center for Complex Quantum Systems, Oden Institute for Computational Engineering and Sciences, Texas Advanced Computing Center
