Chunking strategies for RAG retrieval

Chunking is the step where you split a document into the smaller passages a retrieval system stores, searches, and hands to a language model. It rarely gets the attention that embedding models or vector databases attract, yet it shapes retrieval quality more than almost anything else. If a chunk is too large, its meaning gets diluted and the retriever struggles to match it to a query. If it is too small, it loses the context that made it useful. This post walks through the main strategies in order of increasing sophistication, with the trade-offs and the situations each one suits. It pairs closely with the chunk your documents lesson in our DIY RAG course, which builds a working pipeline step by step.

Fixed-size chunks with overlap

The simplest approach splits text into passages of a fixed length, measured in characters or tokens (the sub-word units a model actually reads), and stops when it hits the limit. To avoid cutting a sentence clean in half at every boundary, you add an overlap: each chunk repeats the last portion of the one before it, so a thought that straddles a boundary appears in full in at least one passage.

Fixed-size chunking is predictable and fast. You know exactly how many chunks a corpus will produce and roughly how much they will cost to embed. Its weakness is that it is blind to structure. It will happily split in the middle of a sentence, a table, or a code block, because it counts length and nothing else.

• Use it when documents are unstructured or inconsistent, and you want a reliable baseline.
• Reach for something smarter when the text has a clear shape you could exploit.

A common starting point is a few hundred tokens per chunk with an overlap of ten to twenty per cent, then adjust based on how retrieval performs. Treat those figures as a first guess, not a rule.

Recursive character splitting

Recursive splitting keeps the predictability of fixed-size chunks but respects natural boundaries where it can. It works down a priority list of separators: try to split on double newlines (paragraphs) first; if a resulting piece is still too long, split it on single newlines; then on sentence endings; and only as a last resort on individual characters. Each chunk grows until adding the next unit would exceed your size limit.

The result is passages that mostly break at paragraph or sentence boundaries, while still honouring a maximum size. This is the sensible default for mixed prose and the approach many people start with once fixed-size chunking shows its limits. It handles articles, reports, and documentation well without needing to understand any particular format.

Its limitation is that the separator list is generic. It does not know that a Markdown heading marks a new section, or that a numbered clause in a contract is a self-contained unit. For that, you need the document’s own structure.

Structure-aware chunking

Structure-aware chunking uses the document’s markup or layout to decide where passages begin and end. Instead of counting length, it splits on headings, sections, list items, table rows, or the tags in HTML and Markdown. Each structural unit becomes a chunk, or a group of adjacent units does if they are short.

Legal text is the clearest illustration. Statutes are written as a hierarchy of parts, sections, and subsections, and each section is already a coherent, self-contained unit of meaning. A source like Legislation.gov.uk publishes UK law with that structure intact, so a single section maps naturally onto a single chunk with almost no guessing. Pile of Law, a large collection of legal and administrative text assembled for training and retrieval, mixes many document types where the same principle applies: honour the section boundaries and each passage stays readable on its own. You can browse both, and more, in the legal category, and the same reasoning extends to structured reference works such as Wikipedia, whose articles carry clear section headings.

• Use it whenever the source format carries reliable structure: Markdown, HTML, well-formed legal or technical documents.
• Fall back to recursive splitting for the messy or unstructured remainder.

The trade-off is effort. You need a parser for each format, and structure varies in quality: a scanned PDF with no heading tags gives you nothing to work with. Where the structure exists, though, it is usually the best signal you have.

Sentence and semantic chunking

The strategies above all key off length or layout. Semantic chunking instead groups text by meaning. The usual method embeds each sentence, then walks through the document comparing neighbouring sentences: where the meaning shifts sharply, it starts a new chunk. Sentences that discuss the same idea stay together even if that makes chunks uneven in length.

This produces passages that are topically tight, which can improve retrieval because each chunk expresses one idea cleanly. The cost is real: you embed every sentence up front just to decide the boundaries, which is slower and more expensive than counting tokens, and the quality depends on the embedding model spotting the shifts. For many corpora the gain over good recursive or structure-aware splitting is modest. It is worth trying when passages vary widely in how densely they pack topics, and when retrieval quality matters more than build cost. The glossary defines embeddings and related terms if any of this is unfamiliar.

Contextual chunking

A retrieved chunk often loses the context that made it meaningful. A subsection that says “This exemption does not apply in those cases” is useless on its own, because “this exemption” and “those cases” were defined paragraphs earlier. Contextual chunking fixes this by prepending a short, generated summary of the parent document or section to each chunk before you embed and store it. The chunk then carries enough context to stand alone when it surfaces in isolation.

That added context helps in two ways: it improves the embedding, because the passage now describes what it is about, and it helps the language model, because the retrieved text is self-contained. The cost is an extra processing step, often a call to a language model per chunk to write the summary, plus slightly larger stored passages. For long, deeply nested documents where individual chunks are hard to interpret alone (legal codes and technical manuals are prime cases) the improvement in retrieval can be worth the expense.

Chunk size, overlap and metadata

Whichever strategy you choose, a few practical decisions recur.

• Size. A few hundred tokens per chunk is a common starting point. Smaller chunks are more precise but risk losing context; larger ones carry more context but dilute the match. Tune against your own queries rather than trusting a default.
• Overlap. A modest overlap, often ten to twenty per cent, reduces the chance that an answer falls between two chunks. Too much overlap wastes storage and returns near-duplicate passages.
• Metadata. Store useful fields alongside each chunk: the source name, the section or heading it came from, a publication or revision date, and a link back to the original. This lets you filter retrieval (for example, only current legislation), show citations, and debug why a passage was returned.

Carrying a date matters more than it first appears. Law is amended, documentation is revised, and news is superseded, so a date field lets you prefer current passages and drop stale ones. The legal and government categories are full of sources where version and date are part of the meaning.

Choosing a strategy

There is no single best method, only the one that fits your documents and your budget. A reasonable path is to start with recursive splitting as a baseline, switch to structure-aware chunking wherever the format gives you clean boundaries, and add contextual or semantic chunking only where retrieval quality justifies the extra cost. Measure each change against real queries rather than assuming.

When you are ready to try this on real data, browse the directory for a corpus that suits your use case, or read choosing your first RAG dataset for help picking one. The chunk your documents lesson then puts these ideas into a working pipeline.