AI Evaluation Engineer – Build & Test AI Coding Tasks

About the role

We are looking for an AI Evaluation Engineer to design, build, and assess realistic developer environments used to test AI coding agents. The work is project‑based, fully remote, and focuses on creating challenging tasks, defining success criteria, and writing robust tests that differentiate high‑performing models from weaker ones.

Key responsibilities

• Construct virtual company environments with codebases, infrastructure, tickets, documentation and conversation histories.
• Design intermediate‑state tasks, write clear prompts, and specify objective success conditions for AI agents.
• Develop functional and integration tests that accept all correct solutions while rejecting incorrect ones.
• Iterate on tasks and tests based on QA feedback, analysing agent failures and refining evaluation criteria.
• Collaborate with the project team to ensure evaluation datasets are fair, reproducible and representative of real‑world development work.

Required profile

• 5+ years of professional software development experience.
• Strong understanding of AI coding models and their typical failure modes.
• Proficiency in English (minimum B2 level).
• Experience writing comprehensive test suites for complex systems.

Required skills

• Python (FastAPI)
• JavaScript / TypeScript (React)
• Docker
• PostgreSQL
• Kafka
• Redis
• Functional and integration testing

What we offer

• Up to $40 USD per hour, based on experience and delivery speed.
• Flexible schedule – you set your own hours.
• Remote, project‑based engagement with leading tech companies.