Masakhane

Masakhane is a research community before it is a dataset, and that shapes what you actually get: not one big download but a steady stream of openly released resources built by African researchers for African languages that mainstream corpora barely touch. The catalogue spans dozens of languages, Swahili, Yoruba, Hausa, Amharic, isiZulu, and many more, through named releases such as MasakhaNER for entity recognition, MasakhaNEWS for topic classification, AfriQA for question answering, and Lafand-MT for machine translation.

Nearly everything lives in the Masakhane GitHub organisation and on Hugging Face, so you can pull a release into a pipeline with the datasets library in a few lines. The data is human-checked rather than scraped, so it arrives cleaner than raw web text, but the shape differs from release to release: some are parallel sentence pairs for translation, others labelled documents or question-answer sets, and you will want to inspect each one before you standardise it.

For retrieval work, Masakhane is often the strongest, and sometimes the only, quality starting point when your users read and write an African language that general multilingual models handle poorly. It suits local-language question answering, translation-assisted retrieval, and honestly measuring whether your embeddings cope with a specific language rather than assuming they do.

Two things to watch. Scale is modest next to English resources, so a single release rarely covers a whole knowledge base and you will usually combine it with other sources. Licensing also varies from release to release, mostly CC BY (free to use with credit) or Apache 2.0, so check the terms on the exact dataset before any commercial use rather than trusting a single project-wide licence.

It pairs naturally with FLORES and MIRACL for multilingual evaluation, and with local-language Wikipedia editions when you need broader background text to retrieve against.