Model Evaluator – DevOps

About TuringTuring is one of the world’s fastest-growing AI companies accelerating the advancement and deployment of powerful AI systems. You’ll be at the forefront of evaluating how LLMs interact with real code, influencing the future of AI-assisted software development. This is a unique opportunity to blend practical software engineering with AI research.

About the Role

We are seeking experienced Software Engineers to support a frontier AI training and evaluation initiative focused on improving the coding capabilities of AI agents.

In this role, you will review and compare AI-generated coding solutions across real-world software engineering tasks. You will analyze complete agent execution workflows, evaluate code correctness, efficiency, safety, and overall quality, and provide structured ratings and evidence-based feedback to help improve AI-assisted software development.

What you'll do

• Review and compare three AI coding agents working on the same real-world developer request.
• Analyze each agent's complete execution, including shell commands, files modified, code changes, test results, and final responses.
• Evaluate each agent's performance across 10 quality dimensions , including instruction following, code correctness, efficiency, safety, honesty, and communication.
• Assign an overall quality score from 1 to 5 for each agent and rank all three solutions based on their performance.
• Assess code functionality, implementation quality, debugging approaches, testing practices, and adherence to software engineering best practices.
• Identify coding errors, incomplete implementations, inefficient approaches, security concerns, and incorrect assumptions.
• Write concise, structured, and evidence-based rationales explaining evaluation scores and comparative rankings.
• Ensure evaluations are technically accurate, objective, consistent, and aligned with established quality guidelines.

What we're looking for

• 3+ years of hands-on software development experience in at least one major programming language, such as Python, JavaScript/TypeScript, Java, Go, PHP , or similar.
• Strong understanding of software engineering fundamentals, including code review, debugging, testing, and development best practices.
• Practical experience working with shell commands, Git, Docker, build tools, and automated test outputs .
• Ability to independently review complex code changes and determine whether implementations correctly satisfy developer requirements.
• Strong understanding of code quality, efficiency, maintainability, security, and error handling.
• Experience interpreting terminal outputs, debugging logs, test failures, and version-control changes.
• Excellent analytical skills and attention to detail, with the ability to identify subtle implementation errors and incomplete solutions.
• Strong written communication skills for producing clear, objective, and evidence-based technical evaluations.

Nice to have

• Experience with code reviews, software quality assurance, technical assessments, or automated testing .
• Familiarity with AI coding agents and developer tools such as Claude Code, Cursor, GitHub Copilot , or similar.
• Experience evaluating LLM-generated code, AI coding agents, or automated programming solutions .
• Experience working across multiple programming languages, frameworks, and development environments.
• Familiarity with AI model evaluation, benchmarking, rubric-based scoring, or structured technical assessments.
• Prior experience in AI training, coding evaluation, or software engineering quality-control programs.

Assessment & Selection Process

• Candidates must complete three practice evaluation tasks as part of the selection process.
• Each assessment requires quality ratings, overall scores, comparative rankings, and written rationales.
• A reviewer will assess each task for technical accuracy, consistency, and quality of reasoning.
• Candidates will be onboarded only after all three practice tasks have been approved.

IMPORTANT - STRICT NO-AI POLICY

DO NOT USE CHATGPT OR ANY OTHER AI TOOL TO WRITE YOUR EVALUATION RATIONALES, EXPLANATIONS, OR REASONS.

ALL RATIONALES AND REASONING MUST BE WRITTEN INDEPENDENTLY BY THE CANDIDATE. COMPLIANCE WILL BE CHECKED. ANY USE OF AI-GENERATED RATIONALES OR REASONS WILL RESULT IN AUTOMATIC REJECTION.

Perks of Freelancing With Turing

• Work in a fully remote environment.
• Opportunity to work on cutting-edge AI projects with leading LLM companies.

Offer Details

• Commitments Required: At least 8 hours per day and 40 hours per week with overlap of 4 hours with PST.
• Employment type : Contractor assignment (no medical/paid leave)
• Duration of contract : 1 month; [expected start date is next week]
• Payment model: $25 per approved task
• Location : India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, Mexico, LATAM

Where you can work

This role is 100% remote, for applicants located in these 24 countries: Argentina, Bangladesh, Bolivia, Brazil, Chile, Colombia, Costa Rica, Dominican Republic, Ecuador, Egypt, Ghana, Guatemala, Honduras, India, Kenya, Mexico, Nigeria, Panama, Peru, Pakistan, Paraguay, El Salvador, Türkiye and Uruguay.