Glossary

The context window is the amount of text a language model can take in at one time, measured in tokens (the small chunks of text a model reads). It covers everything in a single request: your question, any instructions, and any documents you supply, plus the answer the model writes back. If the total goes over the limit, something has to be left out or trimmed.

This limit shapes how you build a Retrieval-Augmented Generation (RAG) system. You cannot pour an entire dataset into the model, so you retrieve only the handful of passages most relevant to the question and fit those inside the context window. Context windows have grown a lot over time, but they are still finite, so choosing what to include remains important.

See also: Token, Large language model (LLM), Prompt, Chunking

An embedding is a way of turning a piece of text into a list of numbers (a vector) that captures its meaning. Text with similar meaning ends up with similar numbers, so passages about the same topic sit close together in this numerical space, even when they use different words. A model trained for the job produces these vectors.

Embeddings are what make meaning-based search possible. Instead of matching exact keywords, you can compare embeddings to find the passages that are closest in meaning to a question. In a Retrieval-Augmented Generation (RAG) system, you typically embed your documents ahead of time, store those vectors in a vector database, and then embed each incoming question to find the best matches.

See also: Semantic search, Vector database, Similarity search, Retrieval

Fine-tuning is the process of taking a language model that has already been trained and training it a bit more on your own set of examples. This nudges the model towards a particular style, tone, or task, such as always replying in a certain format or handling the language of a specialist field. The model's underlying knowledge stays broadly the same; what changes is how it behaves.

Fine-tuning is often compared with Retrieval-Augmented Generation (RAG), and the two solve different problems. Fine-tuning is good for shaping behaviour and style, but it bakes information in at training time, so it is a poor fit for facts that change often. RAG is better when you need current or private information, because it supplies fresh documents at the moment of each question. Many systems use both together.

See also: Large language model (LLM), Retrieval-Augmented Generation (RAG), Grounding, Prompt

Grounding means tying a language model's answer to real source material rather than letting it rely on memory alone. When a response is grounded, the facts in it come from documents you provided, so you can trace each claim back to where it came from and check it. This is the opposite of a model inventing details, which is known as hallucination.

Grounding is the whole point of Retrieval-Augmented Generation (RAG). By retrieving relevant passages and asking the model to answer from that supplied text, you keep its responses anchored in sources you trust. Good grounding also lets you show citations, so a reader can see exactly which document an answer is based on.

See also: Retrieval-Augmented Generation (RAG), Hallucination, Retrieval, Knowledge base

A hallucination is when a language model produces text that sounds plausible but is not true. Because a model generates language by predicting what words tend to follow other words, it can fill gaps with invented facts, fake citations, or made-up details, all delivered in the same confident tone as its correct answers. This makes hallucinations easy to miss.

Reducing hallucinations is one of the main reasons people build Retrieval-Augmented Generation (RAG) systems. By retrieving real documents and asking the model to answer from that supplied text, you give it something factual to lean on rather than relying on memory alone. Good grounding does not remove the risk entirely, but it makes answers far easier to trust and to check against a source.

See also: Grounding, Retrieval-Augmented Generation (RAG), Large language model (LLM), Retrieval

A large language model (LLM) is an AI system trained on enormous amounts of text so it can understand and produce human language. At its core it predicts the next piece of text (a token) given everything that came before, and by doing this repeatedly it can write answers, summarise documents, translate, and hold a conversation. Well-known examples include the models behind ChatGPT, Claude, and Gemini.

An LLM knows a great deal, but only what it saw during training, so it can be out of date or unaware of your private data. It can also state things confidently that are not true. This is why techniques like Retrieval-Augmented Generation (RAG) exist: they supply the model with trusted, up-to-date documents at the moment you ask a question.

See also: Retrieval-Augmented Generation (RAG), Token, Context window, Hallucination

A prompt is the input you give a language model to get a response. It can be as simple as a question, or it can include detailed instructions, examples, and background information that guide how the model should answer. Crafting prompts carefully to get better results is often called prompt engineering.

In a Retrieval-Augmented Generation (RAG) system, the prompt is where everything comes together. You typically combine the user's question with the passages you retrieved and a short instruction telling the model to answer using that supplied context. A well-built prompt is a big part of what keeps the model's answers grounded in your data.

See also: Large language model (LLM), Context window, Grounding, Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation (RAG) is a way of making a language model answer questions using information you give it, rather than relying only on what it learned during training. When someone asks a question, the system first searches a collection of documents for the most relevant passages, then hands those passages to the model along with the question. The model writes its answer using that supplied context.

This matters because a language model on its own can only draw on its training data, which may be out of date, incomplete, or missing your private information. By retrieving fresh, relevant text at the moment of the query, RAG lets you keep the model current and pointed at sources you trust.

RAG is the reason a good data source matters so much. The quality of the answers depends heavily on the quality of the documents you retrieve from, which is what we help you find.

See also: Retrieval, Grounding, Embeddings, Knowledge base

Semantic search finds text by meaning rather than by matching exact words. Traditional keyword search looks for the same words you typed, so a search for "car" might miss a passage that only says "automobile". Semantic search understands that those mean the same thing and returns it anyway.

It works by turning both your query and the documents into embeddings, numerical representations of text where similar meanings sit close together. The system then looks for the document vectors nearest to your query vector. This is the retrieval engine behind most Retrieval-Augmented Generation (RAG) systems, and it is often combined with keyword search (an approach called hybrid search) to get the best of both.

See also: Embeddings, Similarity search, Retrieval, Hybrid search

A token is the basic unit of text that a language model works with. Rather than reading whole words or whole sentences at once, a model breaks text into tokens, which are often a word, part of a word, or a single punctuation mark. The word "unhappiness", for example, might be split into a few tokens. Breaking text up this way is called tokenisation.

Tokens matter for practical reasons. A model can only hold so many tokens at once (its context window), and most AI services charge by the number of tokens you send and receive. A rough rule of thumb for English is that 1,000 tokens is around 750 words, though this varies.

See also: Large language model (LLM), Context window, Chunking, Prompt

Chunking is the step where you break long documents into smaller passages, or chunks, before you store them. A whole report is too big to retrieve and feed to a language model in one go, so you cut it into pieces of a few sentences or paragraphs each.

The size of your chunks matters. Chunks that are too small lose the surrounding context that gives them meaning, while chunks that are too large dilute the useful part with filler and waste space in the model's context window (the limited amount of text it can read at once). Many systems overlap chunks slightly, repeating a sentence or two at the boundaries, so an idea that straddles two chunks is not lost.

See also: Retrieval, Embeddings, Context window, Vector database

Hybrid search blends two ways of finding information. The first is keyword search, which matches the exact words in your query and is great for names, codes, and rare terms. The second is vector search, which compares meaning through embeddings and can find relevant passages even when they use different wording.

Each approach has a blind spot. Keyword search misses paraphrases, while vector search can overlook an exact term like a product number or an acronym. Hybrid search runs both and merges their results, so a query for "GDPR fines in 2023" can match both the precise term "GDPR" and passages that describe data protection penalties in other words. In practice this often retrieves better results than either method alone.

See also: Semantic search, Similarity search, Retrieval, Reranking

A knowledge base is the body of information your RAG system draws on. It might be your product documentation, a set of policies, a research archive, or any collection of documents you want the language model to answer from. It is the source of truth the system retrieves from.

In a RAG pipeline the knowledge base is usually prepared ahead of time: documents are split into chunks, turned into embeddings, and loaded into a vector database so they can be searched by meaning. Keeping it accurate and up to date matters, because the model's answers are only as good and as current as the knowledge base behind them.

See also: Corpus, Retrieval, Grounding, Vector database

Reranking is a clean-up step that runs after your first search. The initial retrieval is fast but rough: it might return 50 candidate passages, roughly in the right ballpark but not perfectly ordered. A reranker looks at each candidate alongside the query more carefully and scores how well it actually answers the question.

Because this closer comparison is slower, you only run it on the shortlist the first pass produced, not your whole collection. The result is a better ordering, so the few passages you finally pass to the language model are the strongest ones. Rerankers are often a type of model called a cross-encoder, which reads the query and a passage together rather than comparing them as separate embeddings.

See also: Retrieval, Similarity search, Top-k, Embeddings

Retrieval is the step that puts the R in RAG (Retrieval-Augmented Generation). Given a question, it searches your collection of documents and pulls out the passages most likely to hold the answer.

The goal is to give the language model just the right context to work from, rather than hoping it already knows the answer. Good retrieval usually turns both the query and your documents into embeddings and compares them by meaning, though it can also use plain keyword matching or a mix of the two. Whatever the method, the passages it returns are what the model reads before it writes its reply, so weak retrieval leads to weak answers.

See also: Retrieval-Augmented Generation (RAG), Embeddings, Similarity search, Reranking

Similarity search is how a vector database finds relevant results. Every piece of text is stored as an embedding, a list of numbers that places it at a point in a large space where similar meanings sit near each other. To search, you turn the query into an embedding too and look for the stored points nearest to it.

Closeness is measured with simple maths, most often cosine similarity, which compares the direction two embeddings point in. Checking every stored item would be slow at scale, so most systems use approximate nearest neighbour (ANN) methods that trade a tiny bit of accuracy for a big gain in speed. This is what lets similarity search stay fast across millions of chunks.

See also: Embeddings, Vector database, Semantic search, Top-k

Top-k is a number you choose that tells a search how many results to hand back. The k is just a stand-in for that number: top-5 returns the five closest matches, top-20 returns the twenty closest. In retrieval, it controls how many chunks you pull from your knowledge base to answer a query.

Picking a good value is a balancing act. Too few chunks and you might miss the passage that holds the answer, too many and you crowd the model's context window (the limited amount of text it can read at once) with weak matches that distract it. A common pattern is to retrieve a generous top-k first, then use a reranker to trim that list down to the few strongest passages.

See also: Retrieval, Reranking, Similarity search, Context window

A vector database stores embeddings, the lists of numbers that capture the meaning of a piece of text, and lets you search them by closeness rather than by exact words. When you ask a question, it turns your question into an embedding too, then finds the stored embeddings that sit nearest to it.

In a RAG system this is where your knowledge lives. You split your documents into chunks, turn each chunk into an embedding, and load them all into the vector database. At query time it returns the handful of chunks most likely to answer the question, which you then pass to the language model. Popular options include Pinecone, Weaviate, Qdrant, and pgvector.

See also: Embeddings, Similarity search, Retrieval, Chunking

A corpus is a big collection of text assembled for a purpose. It might be millions of web pages, every article in an encyclopaedia, a pile of research papers, or a set of legal judgments. The plural is corpora. What makes it a corpus rather than a random heap of files is that it has been gathered deliberately and, usually, cleaned up and organised in a consistent way.

For Retrieval-Augmented Generation (RAG), the corpus is the body of documents you retrieve from when answering a question. The quality and coverage of your corpus shape the quality of your answers, so choosing the right one matters. A lot of the sources we catalogue are corpora you can download and build on.

See also: Knowledge base, Web crawl, Deduplication, Retrieval

Deduplication is the process of finding and removing copies of the same content from a dataset. On the web, the same article, boilerplate, or passage shows up again and again across many pages, so a raw crawl is full of repetition. Deduplication strips out those exact and near-duplicate copies, leaving a cleaner, smaller collection. Techniques like MinHash help spot text that is almost the same even when it is not word-for-word identical.

This matters for RAG in two ways. A deduplicated corpus is cheaper to store and faster to search, and at query time you are less likely to retrieve five copies of the same passage and crowd out other useful results. Many polished datasets advertise that they have been deduplicated, which is a good sign of quality.

See also: Corpus, Web crawl, Retrieval, Chunking

JSONL (JSON Lines, also called newline-delimited JSON or NDJSON) is a simple format where every line of the file is one complete JSON object. JSON is a common way of writing structured data as text, using labelled fields like "title" and "text". By putting one record per line, JSONL lets a program read a file one entry at a time instead of loading the whole thing into memory at once, which is handy when a dataset has millions of records.

This makes JSONL a popular choice for large text datasets used in RAG. Each line might hold a document with its content and metadata, so you can stream through the file, clean or filter records as you go, and split the text into passages for retrieval. It is easy to read, easy to write, and works with almost any programming language.

See also: Parquet, Corpus, WARC, Chunking

A knowledge graph stores information as a web of connected facts. Instead of paragraphs of text, it holds entities (things like people, places, companies, or concepts) and the relationships that join them. You can picture it as a map of dots and lines: each dot is an entity, and each line is a relationship, such as "was born in" or "is a subsidiary of". This structure lets software follow the connections and reason about how things relate.

Knowledge graphs are usually built from RDF triples and queried with SPARQL, a language made for searching them. Wikidata and DBpedia are well-known examples. For RAG, a knowledge graph gives you clean, precise facts and their relationships, which pairs well with the fuzzier, meaning-based search you get from text and embeddings.

See also: RDF, SPARQL, Knowledge base, Semantic search

Parquet is a file format designed for storing large tables of data. Unlike a CSV file, which stores data row by row, Parquet stores it column by column. That layout lets tools read just the columns they need, compress the data well, and skip over the rest, which makes queries faster and files smaller. Apache Parquet is an open standard, so it works across many data tools and languages.

You will see Parquet used a lot for machine learning and RAG datasets, including many collections on Hugging Face. If a dataset ships as Parquet, you can usually load it straight into tools like pandas, DuckDB, or Spark and start filtering the exact rows and columns you want without unpacking the whole thing first.

See also: JSONL, Corpus, WARC, Deduplication

RDF (Resource Description Framework) is a standard for writing down facts in a form that software can read. Each fact is a triple: a subject, a predicate, and an object, which you can read like a short sentence. For example, "Paris" (subject) "is the capital of" (predicate) "France" (object). Every part is given a unique identifier so that different datasets can refer to the same thing without confusion.

Stringing many of these triples together builds a knowledge graph, a network of connected facts. RDF is the format behind structured sources like Wikidata and DBpedia. You usually query RDF data with SPARQL, a language made for searching these connected facts, and it gives RAG systems clean, structured knowledge to draw on alongside plain text.

See also: SPARQL, Knowledge graph, Knowledge base, Corpus

SPARQL is a language for asking questions of structured data stored as RDF, the standard format that records facts as simple subject-predicate-object statements. If you know SQL, the language used to query database tables, SPARQL will feel familiar: you describe the pattern of facts you are looking for, and the query returns everything that matches. For example, you could ask for every city that is the capital of a country in Europe.

Many knowledge graphs, including Wikidata and DBpedia, publish a SPARQL endpoint, a web address you can send queries to and get answers back. For RAG, SPARQL lets you fetch precise, structured facts on demand rather than searching through paragraphs of text, which is useful when you need exact answers like dates, names, or relationships.

See also: RDF, Knowledge graph, Knowledge base, Retrieval

WARC (Web ARChive) is a standard file format for storing crawled web content. When a crawler visits a page, it can save not just the HTML but the full record of the exchange: the request it sent, the response it got back, and metadata like the time and the original address. Bundling all of that into one format means a WARC file is a faithful snapshot of the web at the moment it was captured.

You will meet WARC most often with large web datasets such as Common Crawl, which publishes its crawls in this format. Because a raw WARC file contains everything, including HTML markup and headers, most RAG pipelines extract the plain text first before splitting it into passages. Related formats like WET (extracted text) and WAT (metadata) exist for exactly that reason.

See also: Web crawl, Corpus, Parquet, JSONL

A web crawl is what happens when a program, often called a crawler or spider, works its way across the web by following links from page to page and saving what it finds. Run at scale, this produces enormous collections of web text. The results are often stored in the WARC format, which keeps a faithful record of each page as it was captured.

Web crawls are the raw material behind many of the biggest datasets used to train and feed language models. Common Crawl, for instance, publishes a fresh crawl of billions of pages every month. Because raw crawl data is messy and repetitive, it usually goes through cleaning and deduplication (removing repeated content) before it becomes a tidy corpus you would want to build a RAG system on.

See also: WARC, Corpus, Deduplication, Open data

Creative Commons (CC) is a set of ready-made, standardised licences that let creators share their work while keeping some rights. Instead of writing custom legal terms, a creator picks a CC licence and everyone knows exactly what they can do with the work.

The licences combine a few simple conditions. `BY` means you must credit the original author. `SA` (share-alike) means anything you build from it must carry the same licence. `NC` means non-commercial use only, and `ND` means no modified versions. For RAG, the key one to watch is `NC`: a non-commercial licence can rule out using that data in a paid product, so always read which letters a dataset's CC licence carries before you build on it.

See also: Share-alike (copyleft), Public domain, Open data

ODC-By, short for the Open Data Commons Attribution Licence, is an open licence designed specifically for databases and their contents. It lets you use, share, and adapt the data freely, including for commercial products, on one main condition: you credit the original source (that is what "attribution" means).

It exists because ordinary content licences do not always cover databases cleanly, since a database can be protected as a collection separately from the individual facts inside it. ODC-By closes that gap. For RAG, it is a friendly licence to work with: you can build ODC-By data into your knowledge base as long as you keep a clear note of where it came from.

See also: Open data, Creative Commons (CC), Share-alike (copyleft)

Open data is information that anyone can use, reuse, and share for any purpose, including commercial use. To count as truly open, it needs to be available in a machine-readable format (something software can read, like `CSV` or `JSON` rather than a scanned image) and released under an open licence that spells out what you are allowed to do with it.

For RAG, open data is the friendliest kind to build on. You can download it, process it, and feed it into your knowledge base (the collection of documents your system searches) without paying fees or asking permission. Just check the specific licence, because "open" still comes with conditions, such as crediting the original source.

See also: Public domain, Creative Commons (CC), Open Government Licence (OGL)

The Open Government Licence (OGL) is the standard licence the UK government uses to release public sector information, from statistics to maps to official records. It is deliberately permissive: you can copy, adapt, and reuse the data for any purpose, including commercial products, as long as you credit the source with a short attribution statement.

For anyone building a RAG system on British public data, the OGL is about as easy as licences get. Data from sources like data.gov.uk and the Office for National Statistics is often released under it, so you can pull that material into your knowledge base and only need to acknowledge where it came from.

See also: Open data, ODC-By, Public domain

Public domain means a work is not protected by copyright, so anyone can use it freely for any purpose without asking permission or paying a fee. Works enter the public domain when their copyright expires, or when the creator deliberately releases their rights (a tool called `CC0` is often used to do this).

Public domain material is the most permissive data you can build on. There are no attribution requirements and no share-alike strings attached, so you are free to chunk it, embed it, and mix it with other sources however you like. Classic books from Project Gutenberg and older reference works are common examples.

See also: Open data, Creative Commons (CC), Corpus

Share-alike, also known as copyleft, is a licence condition that says if you adapt or build on a work, whatever you create has to be released under the same licence. The idea is to keep the work open: it stays free for the next person, and so does anything derived from it.

You will see it as the `SA` in a Creative Commons licence, or in database licences like ODC-ODbL. For RAG, share-alike is worth watching closely. If you mix share-alike data into something you then share publicly, you may be required to release that combined work under the same open terms. That is fine for many projects, but it can clash with a closed, commercial product, so check before you build.

See also: Creative Commons (CC), Open data, ODC-By