Chunking your data
Why you split documents into chunks before retrieval, and how to do it well.
When you build a RAG (Retrieval-Augmented Generation) system, you feed it a pile of documents and ask it to answer questions using them. Before any of that can work, you have to break those documents into smaller pieces. Those pieces are called chunks, and deciding how to make them is one of the most important choices you will make. This guide explains why chunking matters and how to do it well.
Why whole documents are too big
Imagine you have a 40-page product manual and someone asks, “How do I reset the device?” The answer lives in a single paragraph on page 32. If you hand the whole manual to the model every time, two things go wrong.
First, models have a limit on how much text they can read at once, called the context window. A long document can be too big to fit, and even when it fits, it is expensive and slow to process.
Second, and more importantly, burying one useful paragraph inside 40 pages makes it harder for the system to find the right answer. RAG works by searching for the most relevant piece of text and passing only that to the model. If your smallest searchable piece is an entire manual, every search returns the whole thing, and the model has to hunt for the answer itself. That defeats the point.
So we split documents into chunks: smaller, self-contained passages that each cover one idea or a small cluster of ideas. When someone asks a question, the system retrieves the handful of chunks most likely to hold the answer, and passes just those to the model.
Chunk size and its trade-offs
The size of a chunk (usually measured in tokens, which are the word-pieces a model reads, roughly three-quarters of a word each) changes how well retrieval works.
Small chunks, say 100 to 200 tokens, are precise. Each one is about a single point, so when it matches a question, it matches well. The downside is that a small chunk can lose its context. A sentence that says “It supports up to 64 GB” is useless if the reader cannot tell what “it” refers to.
Large chunks, say 1,000 tokens or more, keep more surrounding context, so each passage makes sense on its own. The trade-off is precision: a big chunk often mixes several topics, so it can match a question for the wrong reason, and you spend more of your context window on text that is not relevant.
Most people find a middle ground works best. A common starting point is 300 to 500 tokens per chunk. There is no single correct number. It depends on your documents. Dense reference material (legal clauses, product specs) suits smaller chunks. Flowing prose (articles, guides) suits larger ones. Start in the middle, test with real questions, and adjust.
Overlap: giving chunks a shared edge
If you cut a document into strict, non-overlapping blocks, you will sometimes slice a sentence or an idea in half. The end of one chunk holds the question and the start of the next holds the answer, and neither chunk contains both.
Overlap fixes this. You let each chunk share a small stretch of text with the one before it, typically 10 to 20 per cent of the chunk size. If your chunks are 400 tokens, an overlap of 50 to 80 tokens repeats the last few sentences of one chunk at the start of the next. That way, an idea that straddles a boundary still appears whole in at least one chunk. The cost is a little duplicated text and slightly more storage, which is almost always worth it.
Semantic versus fixed-size chunking
There are two broad ways to decide where one chunk ends and the next begins.
Fixed-size chunking
You count off a set number of tokens (or characters) and cut, then repeat. It is fast, predictable, and easy to set up, which is why most people start here. The weakness is that it ignores meaning: it will happily cut in the middle of a sentence, a table, or a step-by-step list.
You can improve fixed-size chunking cheaply by cutting on natural boundaries instead of raw counts: split on paragraphs first, then sentences, and only break mid-sentence as a last resort. This is often called recursive splitting, and it gives you tidy chunks without much extra effort.
Semantic chunking
Here you split based on meaning. The text is grouped so that each chunk covers one coherent topic, and a new chunk starts when the subject shifts. This produces cleaner, more self-contained passages, which usually helps retrieval. The trade-off is that it is slower and more complex to set up, and for many projects the improvement over careful recursive splitting is modest. Reach for it when your documents have messy or uneven structure and fixed-size approaches keep letting you down.
Common mistakes to avoid
- Chunks that are too big. If retrieval feels vague and answers ramble, your chunks are probably too large and diluting the useful text.
- No overlap. Missing answers that clearly exist in your documents often means an idea was split across a boundary.
- Ignoring structure. Cutting through tables, code blocks, or headings scrambles their meaning. Keep structured elements intact where you can.
- Throwing away metadata. Keep track of where each chunk came from (the document title, section, or page). It helps you cite sources and debug bad answers.
- One size for everything. A setting that works for blog posts may fail for legal contracts. Match the chunking to the material.
Next steps
Start simple: recursive splitting at 300 to 500 tokens with 10 to 20 per cent overlap is a solid default. Then test it with real questions and adjust from there. To see the kinds of datasets people chunk in practice, browse the RAG Repo directory, and if any term here is unfamiliar, the glossary explains it in plain English.