Benchmarks

Customer-service agents handle a simulated user under a written policy, with tools on a shared environment. Graded on the final database state; pass^k repeats each task to measure reliability.

3 subsets · 278 tasks

Live-speech turn-taking: pause handling, back-channelling, smooth turn-taking and user interruptions, scored on timing and behaviour.

4 subsets · 735 tasks

Spoken task-oriented dialogue across booking and service domains, with names, times and alphanumerics spoken aloud. Graded on joint goal accuracy.

1 subset · 1,000 tasks

Narrowband telephone speech between strangers and between family members. Scored as word accuracy (100 − WER).

2 subsets · 4,459 tasks

Spoken agent tasks with an adversarial safety subset: impersonation, pressure and instructions smuggled into the call.

1 subset · 412 tasks

Read speech in 102 languages; we run the 40 languages your agents support.

1 subset · 3,500 tasks

Prompt-injection attacks hidden in tool results, measured on task utility and attack success.

1 subset · 629 tasks

Realistic health conversations graded against physician-written rubrics. Official openai/simple-evals code and grader.

3 subsets · 9,671 tasks

Zapier's business-workflow benchmark: 600 tasks on 47 simulated SaaS apps, graded by deterministic end-state assertions.

3 subsets · 300 tasks

89 IT service-management tickets (access, offboarding, incidents, security response) against mock ServiceNow, Okta, Slack and Google Workspace APIs.

1 subset · 89 tasks

2,010 long-horizon legal tasks producing memos, redlines and spreadsheets, graded by two LLM judges.

1 subset · 2,010 tasks

100 end-to-end investment banking tasks (Excel models, pitch decks, memos) graded against expert rubrics.

1 subset · 100 tasks

2,500 expert-written closed-ended questions across dozens of subjects, graded by an LLM judge.

1 subset · 2,500 tasks

66 hard, realistic terminal tasks, each checked by a hidden verifier.

1 subset · 66 tasks

70 expert-curated scientific research workflows, each checked by an offline verifier.

1 subset · 70 tasks