Chunk your documents

You rarely want to retrieve a whole document. If someone asks a narrow question, you want the paragraph that answers it, not a 40-page manual. So before we embed anything, we split each document into small passages. This is chunking.

Why chunk at all

Two reasons:

• Precision. Retrieval works best when each stored item is about one thing. A whole document covers many topics, so its overall “meaning” is blurry. A short passage is sharp.
• Fit. You will paste the retrieved passages into a prompt, and the model’s context window is finite. Small chunks let you include several relevant passages instead of one giant one.

Size and overlap

Two dials matter:

• Chunk size — too big and retrieval gets vague and you can fit fewer passages; too small and each chunk loses the context that makes it meaningful. A sensible starting point is roughly 150 to 250 words.
• Overlap — chunks should share a little text at their edges, so a sentence that lands on a boundary is not split away from its context. An overlap of around 20% of the chunk size works well.

These are starting points, not laws. Different content wants different sizes, and the next-to-last lesson comes back to tuning them.

A simple chunker

We will split on words with a sliding window. It is not clever, but it is predictable and it works:

def chunk_text(text, size=200, overlap=40):
    words = text.split()
    step = size - overlap
    chunks = []
    for start in range(0, len(words), step):
        window = words[start : start + size]
        if window:
            chunks.append(" ".join(window))
    return chunks

Now turn every document into a flat list of chunks, keeping the source with each one so we can trace answers later:

chunks = []
for doc in docs:
    for text in chunk_text(doc["text"]):
        chunks.append({"text": text, "source": doc["source"]})

print(f"Split {len(docs)} documents into {len(chunks)} chunks")

Common mistakes

• No overlap. Boundaries slice sentences in half and answers fall through the gap. Always overlap.
• One size for everything. Dense reference text and chatty prose want different sizes. Start uniform, then adjust once you can measure retrieval quality.
• Chunking away structure. If your documents have headings, keeping a heading with its section (rather than splitting blindly) noticeably improves results. A nice upgrade for later.

Next steps

We have a list of passages. Next we turn each one into a vector that captures its meaning: embeddings.