Aya Collection

The Aya Collection is a massively multilingual instruction dataset assembled by Cohere Labs (the research group formerly published on Hugging Face as Cohere For AI), released as part of the 2024 Aya project. It gathers roughly 513 million prompt and completion instances across 114 languages, which makes it one of the largest openly available resources for multilingual instruction tuning. It is built in three parts: templated data, where fluent speakers wrote instruction templates that were applied to existing curated datasets; translated data, where established English instruction sets were machine translated into many languages; and the human-curated Aya Dataset of around 202,000 annotator-written examples. Together these give broad coverage of lower-resource languages that most mainstream corpora barely reach.

You access it through the Hugging Face Hub as Parquet files, pulled straight into the datasets library, Dask or a Croissant loader, with a browsable viewer for inspection. It is freely downloadable with no access request or gate, and a companion language-split version lets you fetch a single language without downloading the whole collection. Each row carries the input, the target, a task type, a template identifier and the language and script, so the structure is consistent and easy to filter.

For AI work this is primarily a fine-tuning and instruction-tuning resource rather than a retrieval corpus, and it is best for teaching a model to follow instructions across many languages or for bootstrapping evaluation where you otherwise have no coverage. For RAG it is more useful as a source of question and answer style text and as training material for multilingual rerankers and query understanding than as a knowledge base in its own right.

The headline licence is Apache 2.0, which is permissive, but the collection is templated and translated from many upstream datasets that each carry their own terms, so a careful commercial deployment should check the provenance of the specific subsets it uses rather than relying on the top-level licence alone. It sits naturally alongside our Cohere multilingual Wikipedia entries and regional efforts such as Masakhane and CulturaX.