What is a dataset?
What a dataset actually is, the forms it takes, and what separates a useful one from a liability.
A dataset is a structured collection of data gathered around a subject, kept together so it can be used as one thing. That is the whole idea: not a single number or a stray document, but a deliberate collection, organised consistently enough that a person or a program can work through all of it the same way. When this directory lists “datasets for RAG”, this is what it means: collections of text and data you can load, search, and build on.
The simplest picture: rows and columns
The most familiar kind of dataset is a table. Each row is one record (one patient, one transaction, one film) and each column is one attribute of it (age, amount, release year). A spreadsheet is a dataset. So is the data behind it. The power of the table is consistency: because every row has the same columns, you can sort, filter, count, and compute across thousands or millions of records without handling each one individually.
Structured and unstructured data
Tables are structured data: every value has a defined place and type. A great deal of the world’s information is not like that. The text of a news article, the transcript of a call, a scanned contract, an image: these are unstructured, meaning there is no neat grid of fields, just content.
This distinction matters enormously for AI. A table of sales figures and a corpus of ten thousand support emails are both datasets, but you use them very differently. Retrieval-Augmented Generation (RAG), the technique this site is built around, works mostly with unstructured text: you collect documents, split them up, and search them by meaning. If that idea is new, our What is RAG? guide starts from scratch.
The common formats
A dataset is distributed as files, and a handful of formats cover most of what you will meet:
- CSV (comma-separated values): plain-text tables, readable anywhere.
- JSON and JSONL: nested records, one object per line in the JSONL case, common for text data.
- Parquet: a compact columnar format built for large datasets, fast to scan and filter.
- XML and WARC: used for structured documents and web-crawl archives respectively.
None of these is “the dataset” in a deep sense; they are containers. The same collection can often be downloaded in several of them.
Where datasets come from
Datasets are created in many ways: measured (sensor readings, experiments), recorded (transactions, logs), written (articles, books, forum posts), crawled from the web, or generated by other models. Each origin carries its own quality and licensing story, which is why two datasets of the same size can be worlds apart in how usable they are.
What makes a dataset good (for AI and RAG)
Size is the least interesting property. The ones that matter for building an AI system are:
- Relevance. Does it actually contain the knowledge your users will ask about? Retrieval cannot surface a fact that is not in the collection.
- Cleanliness. Is it free of boilerplate, duplicates, and junk, or will you spend days tidying it before it is usable?
- A clear licence. Are you allowed to use it for what you have in mind, especially commercially? A dataset being free to download is not the same as being free to build a product on. Our data licensing guide covers the traps.
- Provenance. Do you know where the data came from and whether it is trustworthy?
Datasets, databases, and corpora
Three words that overlap. A database is a live system you query (it holds data and serves it); a dataset is a fixed collection you obtain and work with. A corpus is simply a dataset of text, the term used in language and AI work. On this site, “source” usually means a dataset or the place that publishes one.
How to find one
That is what the rest of this site is for. The directory groups curated datasets by subject, each with its licence, format, and how ready it is for retrieval. General hubs like Kaggle and Hugging Face Datasets are good for discovery and prototyping. The skill is not finding a dataset, it is finding the right one: relevant, clean, properly licensed, and suited to what you are building.