Upstash Wikipedia 2024 BGE-M3 Embeddings
This dataset, published by Upstash, takes the June 2024 Wikipedia dump and turns it into a ready-to-use embedded corpus. Articles are split into paragraphs, each paragraph is prefixed with its article title, and the result is encoded with the multilingual BGE-M3 model from BAAI. Paragraphs shorter than 100 characters are dropped as low in information density, leaving roughly 144 million 1024-dimensional vectors spread across the 11 most popular Wikipedia languages (English, German, French, Russian, Spanish, Italian, Japanese, Portuguese, Farsi, Chinese and Turkish), with English alone contributing about 47 million.
The files are Parquet, one configuration per language, and you load them with the HuggingFace datasets library. Because some languages are large, you can stream a split rather than download it in full, then upsert the id, title, text and embedding fields straight into a vector database. The practical payoff is that the embedding step is already done, so you can stand up multilingual semantic search or a retrieval layer for RAG without running a GPU pipeline of your own.
One dependency matters. The vectors come from BGE-M3, so at query time you must embed the user's question with the same model (it is open and runs locally, an advantage over API-only models) or your similarity scores will be meaningless. The text is also a mid-2024 snapshot, so anything more recent is absent.
On licensing, Upstash releases the dataset under Apache 2.0, while the underlying Wikipedia text remains CC BY-SA, which carries attribution and share-alike obligations and leaves redistribution of embeddings over share-alike text in a grey area. Compared with our Cohere Wikipedia 22-12 listing, this release is fresher, uses a fully open embedding model you can run yourself rather than a paid API, and is a strong default when you want newer coverage without vendor lock-in.
Related sources
Cohere Wikipedia 22-12 Embeddings (per-language)
Wikipedia encoded with the Cohere multilingual-22-12 embedding model, with embeddings (numeric vectors that capture meaning) computed on each article's title plus text. Published per language, covering many including Arabic, German, English, Spanish, French, Hindi, Italian, Japanese, Korean, Simple English, and Chinese. You can stream it rather than download it in full, which matters given the size.
Cohere Wikipedia Multilingual Embeddings (2023-11)
The November 2023 Wikipedia dump across 300+ languages, split into passages and turned into embeddings (numeric vectors that capture meaning) with the Cohere Embed V3 multilingual model. Close to 250 million paragraph embeddings in total. Drop it into a vector database for semantic search over all of Wikipedia, or use it as a knowledge source for multilingual RAG, including cross-lingual search where a query in one language returns relevant results from another.
Google Satellite Embedding (AlphaEarth Foundations)
Global annual satellite embeddings produced by Google and Google DeepMind's AlphaEarth Foundations model. Every 10 metre pixel holds a 64 dimensional vector summarising a year of multi-sensor Earth observation, available yearly from 2017.
LAION-400M with CLIP Embeddings
400 million image-text pairs filtered from Common Crawl using CLIP, a model that scores how well an image matches a caption, and released alongside their pre-computed CLIP embeddings and kNN indices for fast similarity search. The ready-made embeddings make it unusually easy to work with. Note that it ships image URLs, not the images themselves, so some links have decayed.