Multimodal & image-text

Multimodal datasets pair images with the words that describe them, and they are what let a RAG system reach beyond plain text. If you want to retrieve a diagram from a manual, answer questions about a photo, or ground a model’s response in a picture rather than a paragraph, you need image-text pairs to train or evaluate the models that make that possible. The captions teach a model how pictures and language line up, so retrieval can cross from one to the other.

Choosing within this category comes down to how the pairs were made. Web-scraped sets like LAION-5B are enormous and cheap to obtain, but the captions are the original alt text from web pages, which is often noisy, promotional, or simply wrong. Curated sets are smaller and cleaner because a human or a careful pipeline checked the pairing. Bigger buys you coverage and variety, smaller buys you trustworthy captions, and most projects want a bit of both.

The thing to watch here is what you actually receive. Many of these datasets ship as lists of image URLs plus captions, not the images themselves, so you download the pictures yourself and some links will have rotted. Licensing is the other trap: the caption list may be openly licensed while the underlying photos each carry their own copyright, so “open dataset” rarely means “free to reuse every image”. Public-domain-only sets like PD12M exist precisely to sidestep that.

The sources below run from the giant web-scraped collections to the smaller, cleaner, and rights-cleared sets.