SWE-Bench Task Auditor
The middle of its range, $80/hr, is above the $55/hr median of the Agentic Tasks & RL Environments roles open now, whose typical pay runs $50–$120/hr. How that pay moves week by week is in our research on Agentic Tasks & RL Environments work.
Half of all Agentic Tasks & RL Environments roles come down within 9.3 days of being listed, and 33.6% are still up after four weeks. This one was posted 77 days ago.
Some links on AITraining.jobs are referral links. If you sign up through one, AITraining.jobs may earn a commission at no extra cost to you. It never changes what a platform pays you.
About the role
Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity - and provide clear, rubric-based written feedback.
Basic Qualifications
- 3+ years professional software engineering
- Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles)
- Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking
- Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)
Preferred Qualifications
- Familiarity with SWE-Bench (Verified) or similar repository benchmarks
- Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.)
- Prior code-review or task-grading experience
Where you can work
This role is 100% remote, for applicants located in United States.
From Mercor's job posting, checked 14 Nov 2026.
About Mercor
Premium expert network: physicians, lawyers, engineers, PhDs.
How it pays: Weekly via Stripe (1099 contractors).
When it pays: Every Wednesday, for the week before. How often others pay
Interview: yes, for everyone it takes (AI Video Interview (plus role-specific assessments)). How the Mercor AI interview works
Similar open roles
Some links on AITraining.jobs are referral links. If you sign up through one, AITraining.jobs may earn a commission at no extra cost to you. It never changes what a platform pays you. A role at a platform that pays us a referral fee weighs more in this order.
AI Conversation Evaluator (opens the job details)
micro1 · AI Evaluation · Remote· 2d agoDetails
$40–$70/hr
Document Annotation Specialist (PDF) (opens the job details)
SuperAnnotate (SME) · Data Annotation · Remote· 4d agoDetails
up to $10/hr
Cloud Engineer Expert (opens the job details)
AfterQuery · Coding · Remote· 2mo agoDetails
$125/hr
Babel Audio · Transcription & Audio · Remote· 2w agoDetails
$17.50/hr
An experienced engineer?
One profile for every AI training platform. Companies ask us for people with a specific language, country or credential: add yours once, and when a role fits, we introduce you, only with your yes. Free.
Get new Mercor roles by email
We check the Mercor job board every hour and email you its newest roles each week, plus similar paid roles elsewhere. Free, unsubscribe anytime.