Python Domain Expert for Coding-Agent Benchmarks
Half of all Coding & Software Evaluation roles come down within 18 days of being listed, and 44.8% are still up after four weeks. This one was posted today.
About the role
The work
- Design benchmark tasks for bug fixes, feature additions, continuous integration repairs, configuration migrations, integration updates, and runtime-state fixes.
- Define utility requirements that verify whether a coding agent completes engineering work correctly.
- Set safety requirements covering system constraints, developer intent, data integrity, privacy, permissions, and oversight.
- Build visible and hidden test suites for task completion and unsafe behavior.
- Create safe and unsafe reference solutions, evaluators, scoring rubrics, calibration notes, runnable repositories, and reference patches.
- Analyze coding-agent shortcuts such as disabling tests, weakening assertions, deleting protected data, leaking secrets, broadening permissions, and bypassing validation.
- Review coding-agent rollouts and work with engineering, quality, and security teams to improve benchmark quality and difficulty.
What it takes
- This is a fully remote contractor assignment lasting 4-8 weeks, with at least 20 hours per week and at least 4 hours of overlap with Pacific Time.
- The role is open worldwide and requires professional fluency in English.
- A bachelor's or master's degree in computer science or a related technical discipline, at least 8 years of hands-on software engineering experience, and professional Python experience plus Java, Go, or C++ experience are required.
- You should have built and maintained production software used by real customers and understand software architecture, debugging, performance optimization, deployment pipelines, monitoring, logging, observability, distributed tracing, high availability, fault tolerance, scalability, and disaster recovery.
- Practical secure development experience is required, including identifying vulnerabilities and validating fixes involving injection, authentication, authorization, memory safety, race conditions, deserialization, secrets management, and input validation.
- This is part-time contract work supporting advanced AI projects with technical collaborators.
How it works
Apply on OpenTrain with your resume, then complete the application on the hiring site.
About AI training work
AI training is the human work behind systems that can write, understand, and act on information. People with strong technical experience are paid to create examples, tests, and reviews that help improve how these systems perform and behave safely.
Where you can work
This role is 100% remote: you can work from any country except Andorra, Afghanistan, Barbados, Burkina Faso, Burundi, Brazil, Belarus, Belize, DR Congo, Central African Republic, Congo (Republic), Cameroon, China, Cuba, Cape Verde, Djibouti, Dominica, Eritrea, Fiji, Micronesia, Grenada, Georgia, Guinea, Equatorial Guinea, Guinea-Bissau, Honduras, Haiti, Iraq, Iran, Kyrgyzstan, Kiribati, Comoros, St. Kitts and Nevis, North Korea, Lebanon, Liberia, Lesotho, Libya, Montenegro, Marshall Islands, Mali, Myanmar, Mauritania, Maldives, Malawi, Nicaragua, Nepal, Nauru, Papua New Guinea, Palestine, Palau, Russia, Solomon Islands, Seychelles, Sudan, Sierra Leone, Somalia, Suriname, South Sudan, São Tomé and Príncipe, Syria, Eswatini, Chad, Togo, Tajikistan, Timor-Leste, Turkmenistan, Tonga, Tuvalu, Ukraine, Uganda, Vatican City, St. Vincent and Grenadines, Venezuela, Vanuatu, Samoa, Yemen, Zambia and Zimbabwe.
From OpenTrain AI's job posting, checked 18 Nov 2026.
About OpenTrain AI
A unified board of AI-training jobs with real per-role pay.
How it pays: Stripe payouts to your bank, in the 120 countries its payout docs list.
When it pays: When the employer approves each funded milestone. How often others pay
Interview: on some roles or projects (AI interview (switched on per job by the employer)). How the OpenTrain AI AI interview works
Similar open roles
Some links on AITraining.jobs are referral links. If you sign up through one, AITraining.jobs may earn a commission at no extra cost to you. It never changes what a platform pays you. A role at a platform that pays us a referral fee weighs more in this order.

Data-Video Generalist (US-based) (opens the job details)
micro1 · Robotics Data · Remote· todayDetails
$13–$15/hr
Audio and Video Data Annotator (opens the job details)
OpenTrain AI · Data Annotation · Remote· 3mo agoDetails
$6–$8/hr
Document Annotation Specialist (PDF) (opens the job details)
SuperAnnotate (SME) · Data Annotation · Remote· 1w agoDetails
up to $10/hr
Software Developer Expert (opens the job details)
AfterQuery · Coding · Remote· 3mo agoDetails
$75/hr
Prompt & Response Evaluator (opens the job details)
micro1 · AI Evaluation · Remote· 6d agoDetails
$40–$70/hr
An experienced engineer?
One profile for every AI training platform. Companies ask us for people with a specific language, country or credential: add yours once, and when a role fits, we introduce you, only with your yes. Free.
Get new OpenTrain AI roles by email
We check the OpenTrain AI job board every hour and email you its newest roles each week, plus similar paid roles elsewhere. Free, unsubscribe anytime.