LAION-400M with CLIP Embeddings
LAION-400M is a dataset of 400 million image-text pairs built by starting from Common Crawl and filtering the results with CLIP, a model that scores how well an image matches a piece of text. The result is a large, openly available pool of captioned images widely used for training and evaluating multimodal models.
What sets it apart is that it ships with the CLIP embeddings (numeric vectors that capture the meaning of each image and caption) and kNN indices already computed, so you can run efficient similarity search out of the box without encoding anything yourself. That pre-computed layer removes the most expensive step of working with a dataset this size.
The metadata is licensed under CC BY 4.0, but the dataset contains only URLs pointing to images rather than the images themselves, and each image remains under whatever terms its original host set. Some links have decayed over time, and you should review content-safety considerations before using it.
Related sources
Cohere Wikipedia 22-12 Embeddings (per-language)
Wikipedia encoded with the Cohere multilingual-22-12 embedding model, with embeddings (numeric vectors that capture meaning) computed on each article's title plus text. Published per language, covering many including Arabic, German, English, Spanish, French, Hindi, Italian, Japanese, Korean, Simple English, and Chinese. You can stream it rather than download it in full, which matters given the size.
Cohere Wikipedia Multilingual Embeddings (2023-11)
The November 2023 Wikipedia dump across 300+ languages, split into passages and turned into embeddings (numeric vectors that capture meaning) with the Cohere Embed V3 multilingual model. Close to 250 million paragraph embeddings in total. Drop it into a vector database for semantic search over all of Wikipedia, or use it as a knowledge source for multilingual RAG, including cross-lingual search where a query in one language returns relevant results from another.