Cohere Wikipedia Multilingual Embeddings (2023-11)
This dataset does the most expensive part of a multilingual RAG pipeline for you. It takes the full Wikipedia snapshot from November 2023, spanning more than 300 languages, splits every article into paragraph-sized passages, and converts each passage into an embedding (a numeric vector capturing its meaning) using Cohere's Embed V3 multilingual model. In total there are close to 250 million passage embeddings, already aligned with the source text and metadata.
Because the vectors are precomputed, you can load them straight into a vector database and start running semantic search, where results are matched by meaning rather than exact keywords, without paying to embed the corpus yourself. It is distributed on Hugging Face, and given the scale you will usually stream it or pull only the languages you need rather than downloading everything. The model places every language in one shared vector space, so a query in any language can surface relevant passages written in others, which is the real draw for cross-lingual search.
One setup detail matters more than any other: to query this index you must embed your search text with the same Cohere Embed V3 model. Mix embedding models and you are comparing vectors from different spaces, and the results will be meaningless. That model is a paid API, so factor its per-query cost into your plans, or choose a dataset built with an open model if you need everything self-hosted.
For RAG, this is the fastest route to a broad, multilingual knowledge base, ideal when you need coverage across many languages and want cross-lingual answers. Two caveats sit beyond the model lock-in. The snapshot is frozen at November 2023, so it knows nothing since, and the licensing is split: the embeddings are Apache 2.0 while the underlying Wikipedia text stays under CC BY-SA, which requires attribution and is share-alike. Whether embeddings of share-alike text count as a derivative work is genuinely unsettled, so weigh that ambiguity into any commercial plans.
Related sources
Cohere Wikipedia 22-12 Embeddings (per-language)
Wikipedia encoded with the Cohere multilingual-22-12 embedding model, with embeddings (numeric vectors that capture meaning) computed on each article's title plus text. Published per language, covering many including Arabic, German, English, Spanish, French, Hindi, Italian, Japanese, Korean, Simple English, and Chinese. You can stream it rather than download it in full, which matters given the size.
Google Satellite Embedding (AlphaEarth Foundations)
Global annual satellite embeddings produced by Google and Google DeepMind's AlphaEarth Foundations model. Every 10 metre pixel holds a 64 dimensional vector summarising a year of multi-sensor Earth observation, available yearly from 2017.
LAION-400M with CLIP Embeddings
400 million image-text pairs filtered from Common Crawl using CLIP, a model that scores how well an image matches a caption, and released alongside their pre-computed CLIP embeddings and kNN indices for fast similarity search. The ready-made embeddings make it unusually easy to work with. Note that it ships image URLs, not the images themselves, so some links have decayed.
Major TOM
An open, globally dense collection of embeddings and ML-ready imagery derived from Copernicus Sentinel data, published by ESA Phi-lab. The embedding expansions run to well over 100 billion vectors covering most of the Earth.