AWS Open Data Registry

The AWS Open Data Registry is a catalogue, not a single dataset. It points to hundreds of high-value collections that organisations have chosen to publish openly on Amazon's cloud, from Landsat and Sentinel satellite imagery to reference human genomes, large climate model outputs, and the Common Crawl archive itself. Together they run to more than 300 PB. Each listing gives you the storage location (almost always an S3 bucket), a short description, a licence, and usually a few worked examples or tools maintained by the data provider.

Because the data already lives in AWS, the fastest and cheapest way to use it is from inside the same cloud, ideally in the region where the bucket sits, where there is no transfer cost. You can stream individual files straight into a processing job with the AWS CLI or an S3 client, or copy a slice into your own bucket. Formats vary by dataset and range from Parquet and CSV to domain-specific scientific formats like NetCDF, GeoTIFF, and FASTA.

For RAG, this is a strong source of large, cloud-ready scientific, geospatial, and climate corpora that would be painfully slow to download to a laptop but quick to process where they already live. It suits pipelines you are already running on AWS, where you can filter and extract the text or tabular fields you need close to the source.

The main thing to watch is that licences are set per dataset, not by the registry, so the terms differ from one entry to the next. Some are fully open, others carry attribution or non-commercial conditions, so read the listing before you build on it. Because most collections are raw scientific data rather than clean prose, expect to do real extraction and cleaning work.

Treat the registry as a discovery layer sitting above the individual datasets, much like Google Dataset Search or Hugging Face, but tuned to collections big enough that where they are hosted really matters.