YFCC100M

YFCC100M (the Yahoo Flickr Creative Commons 100 Million dataset) collects 100M items uploaded to Flickr, roughly 99M photos and around 800,000 videos, each one released by its owner under a Creative Commons licence. The first thing you download is not the media but the catalogue: a large tab-separated file with one row per item, carrying tags, title, description, camera make and model, timestamps, and, for a sizeable share of the collection, geolocation. The photos and videos themselves sit behind URLs and are mirrored in the Multimedia Commons bucket on AWS.

The smoothest way in is the Multimedia Commons project, which hosts both the metadata and the media as an AWS Open Data set. Pull the metadata from the S3 bucket (working from inside AWS keeps transfer costs down), filter the rows to the licences and content you actually want, then fetch only the media you need rather than all 100M items. Precomputed extras are available too, including image features and machine-generated autotags, which spare you a first processing pass. For RAG the common move is to treat the tags, titles, and descriptions as searchable captions and pair them with the images to build multimodal embeddings using a model like CLIP.

It suits multimodal and image-text retrieval, vision-language embedding work, place recognition from geotags, and time-based analysis: any project that wants a large corpus of real, in-the-wild user photos with genuine metadata rather than scraped alt-text.

The caveats are real. Licences differ item by item, from permissive CC BY through to NonCommercial and NoDerivatives variants, so you must filter on each item's licence before you ship anything, and you cannot assume the set as a whole is safe for commercial reuse. Link rot is the other snag: the dataset dates from 2014 and many original Flickr URLs have since gone dead, so expect gaps if you rely on live fetching rather than the AWS mirror. The metadata is user-generated, so tags are noisy, multilingual, and uneven.

Set against newer web-scale sets like LAION or cleanly annotated ones like COCO and WIT, YFCC100M stands out for pairing real photographs with rich, human-written metadata and clear per-item Creative Commons terms.