τ³-bench (tau3-bench)

τ³-bench (tau3-bench) is the third generation of Sierra's τ-Bench, a benchmark for tool-agent-user interaction in realistic service domains. It is maintained by Sierra Research and, despite the version name, still lives in the GitHub repository called tau2-bench. The τ³ release of March 2026 added a document-retrieval domain called τ-Banking, a voice evaluation mode and over 75 task-quality fixes to the existing airline, retail and telecom domains. The repository has since moved on to τ-Bench 2.0, which adds a multilingual track. You access it by cloning the repository and syncing dependencies with uv (Python 3.12 or 3.13) rather than by a dataset download, and the tasks, tools and policy documents ship as structured files inside the repo. What makes τ-Banking relevant here is that it puts retrieval inside the agent loop rather than testing it in isolation. The agent must navigate a knowledge base of 698 documents spanning 21 product categories (around 195K tokens of policy manuals, product catalogues and standard operating procedures) to decide what to do, and success is judged by whether the correct change is made to a simulated backend, for example opening a dispute or freezing a card. Retrieval is pluggable, defaulting to offline BM25 with embedding and shell-based options available, so it doubles as a testbed for RAG strategies under agentic pressure. For RAG and AI work it is an evaluation set, not training material, best used to measure whether a tool-using agent retrieves the right policy and acts on it correctly, which is closer to production RAG than a static question-answer benchmark. The licence is MIT (Copyright Sierra Research), so the code and data are free for commercial use with the usual notice retention. The practical caveat is cost, not licence: reproducing scores needs an external LLM to play the simulated user, and the voice mode needs real-time speech models from third parties, so each run incurs model API charges. It complements BFCL, which scores the call itself, and retrieval benchmarks such as BRIGHT and BEIR that we list separately.