Mindrift · United States
Join Mindrift to help build a groundbreaking dataset for evaluating AI coding agents. You’ll design realistic developer environments, craft challenging tasks, and write robust tests to measure how well AI models handle real-world coding scenarios. This is not data labeling or prompt engineering — it’s about understanding where models fail and creating fair, challenging evaluations that push the frontier of AI capabilities.
About the Role
Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment. You’ll be part of a team that shapes the future of AI by creating realistic, challenging environments for AI coding agents to test against. This role requires deep technical expertise in Python, JavaScript, and modern infrastructure tools. You’ll design tasks that genuinely challenge the best models, write tests that accept all valid solutions and reject incorrect ones, and iterate based on QA feedback. The ideal candidate has 8+ years of software development experience and speaks English at B2+ proficiency. This is a remote, freelance opportunity with flexible scheduling and compensation based on performance and pace.
What You Will Do
Build realistic developer environments — a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
Design tasks from intermediate states of these environments — craft the prompt, define what “solved” means, and ensure the task is solvable by an AI agent
Write tests that verify agent solutions — accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
Iterate on tasks and tests based on QA feedback — review agent solutions, analyze failures, and refine until the evaluation is fair and robust
What We Are Looking For
8+ years in software development
Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
Experience writing tests (functional, integration)
English proficiency – B2+
Why this is hard: Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions – writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid
Effort estimate: Tasks for this project are estimated to take 30 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.
Compensation: Up to $200/hr equivalent, depending on level and pace. Tasks are estimated at ~30 hours each; you set your own schedule.
Ready to shape the future of AI? Apply now and become part of a mission that pushes the boundaries of what AI can do.