GAIA

GAIA (General AI Assistants) is a benchmark for real-world assistant tasks, built by researchers from Meta, Hugging Face and AutoGPT (Mialon et al.) and presented at ICLR 2024. It contains 466 questions that are conceptually simple for people but hard for current systems, because answering them requires a chain of reasoning, live web browsing, handling of attached files and images, and the use of external tools. Each question has a single unambiguous answer, and the set is graded across three difficulty levels, from a few steps to long multi-step investigations.

You access GAIA through its gated dataset on Hugging Face: you sign in and accept the usage terms before the files are released. The public validation split ships with answers for local development, while the answers to the 300 held-out test questions are withheld, so scoring on the test set is done only by submitting to the public leaderboard. The data is now Parquet-backed (previously JSONL), roughly 110 MB in total, with attachments referenced by file path.

For RAG and agent work, GAIA is an evaluation set rather than training material. It is one of the most cited yardsticks for general assistants and agentic pipelines, measuring whether a system can plan, retrieve, read documents and combine tools to reach a correct answer, rather than testing a single skill in isolation. At launch, human respondents reached about 92 per cent while GPT-4 with plugins reached roughly 15 per cent, which captures the gap it was built to expose. It is best used as a hard end-to-end gate for assistants and tool-using agents before release.

On licensing, treat GAIA with care: the dataset card shows no formal licence identifier, access is gated, and the terms ask you not to reshare the data outside a gated or private Hugging Face repository, not to post it in a crawlable form, and not to train on the development set. Confirm the current terms on the dataset card before any redistribution or commercial use. It sits alongside our other agentic entries, such as BFCL, as a complementary, broader test of end-to-end assistant capability.