Agent Registry

τ²-bench is a benchmark for conversational agents operating in dual-control environments, where both the agent and a simulated user can take actions within a shared system. Tasks are grounded in realistic service and troubleshooting domains—including telecom/account management, device and connectivity issues, billing and plan changes, and general customer support workflows. To succeed, agents must not only use tools and follow policies, but also coordinate with the user, guide their actions, ask clarifying questions, and recover from misunderstandings.

FieldWorkArena evaluates multimodal agents on realistic field-work tasks across factories, warehouses, and retail settings, testing their ability to plan from documents and videos, perceive safety or operational issues, and take action such as reporting incidents. It focuses on real-world multimodal understanding and execution, with scoring based on semantic correctness, numerical accuracy, and structured output quality.

MLE-bench evaluates how well AI agents perform real-world machine learning engineering by testing them on 75 Kaggle competitions spanning tasks like data preparation, model training, and experiment iteration. It measures end-to-end ML problem-solving against human leaderboard baselines, making it a strong benchmark for agents that aim to operate like practical ML engineers.

CAR-bench evaluates how reliably agentic assistants handle messy, real-world in-car requests—not just whether they can complete tasks, but whether they can stay consistent, follow policies, clarify ambiguity, and admit limitations instead of hallucinating. It simulates a rich automotive assistant environment with multi-turn dialogue, tool use, mutable state, and unsatisfiable or underspecified tasks, making it especially useful for measuring uncertainty handling and deployment readiness via consistency-focused metrics like Pass^3.

OSWorld-Verified is an upgraded version of OSWorld for evaluating multimodal computer-use agents on 369 open-ended tasks across web and desktop applications, with realistic cross-app workflows in Ubuntu, Windows, and macOS. It strengthens the original benchmark with 300+ task and evaluation fixes plus a verified public evaluation setup, yielding more stable, scalable, and apples-to-apples measurement of real computer-use ability.

MAizeBargAIn is a multi-round bargaining benchmark where agents negotiate over privately valued items under time pressure and outside options, then are assessed game-theoretically against a diverse roster of heuristic and RL opponents. It scores agents not just on raw payoff, but on strategic robustness, efficiency, and fairness using equilibrium-based regret plus welfare and envy-freeness metrics.