Evaluation Scenario Writer for AI Agents

The Work

You will create realistic, reusable evaluation scenarios for AI agents that perform real-world tasks. You will define the correct path, acceptable agent behavior, expected outputs, edge cases, and scoring rules.

You will also review agent outputs, improve scenarios for clarity and coverage, and work with developers and other contributors to test and refine evaluation frameworks.

• Design structured evaluation scenarios for LLM-based agents.
• Define gold-standard behavior and acceptable alternatives.
• Annotate task steps, expected outputs, edge cases, and scoring logic.
• Review agent responses and revise scenarios to improve coverage.
• Use structured formats such as JSON and YAML.

What It Pays and Takes

This is an intermediate-level, part-time contractor role for someone who combines careful writing with software testing or quality assurance thinking.

• Pay: $18 to $24 per hour.
• Time: 20 or more hours per week.
• Location: Open worldwide.
• Language: Strong written English.
• Required: Basic Python and JavaScript experience.
• Required: Analytical thinking and comfort designing structured evaluations.
• Preferred: Experience in software testing, QA, data analysis, or NLP annotation.
• Work type: Part-time contract.

How It Works

Apply on OpenTrain. The employer reviews applications there.

About AI Training Work

AI training work is the human work behind modern artificial intelligence, including writing tests, reviewing model responses, and rating whether systems behave correctly. People with technical and analytical skills are paid to provide the careful examples and judgments that help improve AI systems.

Where you can work

This role is 100% remote: you can work from any country except Andorra, Afghanistan, Barbados, Burkina Faso, Burundi, Brazil, Belarus, Belize, DR Congo, Central African Republic, Congo (Republic), Cameroon, China, Cuba, Cape Verde, Djibouti, Dominica, Eritrea, Fiji, Micronesia, Grenada, Georgia, Guinea, Equatorial Guinea, Guinea-Bissau, Honduras, Haiti, Iraq, Iran, Kyrgyzstan, Kiribati, Comoros, St. Kitts and Nevis, North Korea, Lebanon, Liberia, Lesotho, Libya, Montenegro, Marshall Islands, Mali, Myanmar, Mauritania, Maldives, Malawi, Nicaragua, Nepal, Nauru, Papua New Guinea, Palestine, Palau, Russia, Solomon Islands, Seychelles, Sudan, Sierra Leone, Somalia, Suriname, South Sudan, São Tomé and Príncipe, Syria, Eswatini, Chad, Togo, Tajikistan, Timor-Leste, Turkmenistan, Tonga, Tuvalu, Ukraine, Uganda, Vatican City, St. Vincent and Grenadines, Venezuela, Vanuatu, Samoa, Yemen, Zambia and Zimbabwe.