Get your data

RAG is only as good as the documents it can retrieve from. So the first real step is to get a corpus: the set of documents your system will answer from.

Pick something small to start

Do not begin with a million files. A handful of documents, a few hundred kilobytes of text, is perfect for learning: it indexes in seconds and you can eyeball whether retrieval is working. You can scale up once the pipeline works.

Good starting options:

• Your own notes or docs — a folder of Markdown or text files you already know well, so you can judge the answers.
• An open dataset — if you want a ready-made corpus, browse the RAG Repo directory. A slice of Wikipedia is a classic first corpus: broad, clean, and openly licensed.

For this course we will assume a folder called docs/ containing .md or .txt files.

Load the documents

We want each document as plain text, with a note of where it came from so we can trace answers back to their source. This function walks a folder and returns a list of documents:

from pathlib import Path

def load_documents(folder):
    docs = []
    for path in Path(folder).rglob("*"):
        if path.suffix.lower() in {".md", ".txt"}:
            text = path.read_text(encoding="utf-8", errors="ignore")
            docs.append({"source": str(path), "text": text})
    return docs

docs = load_documents("docs")
print(f"Loaded {len(docs)} documents")

Each document is a small dictionary: the source (so we can show where an answer came from) and the text itself.

Tidy the text

Raw files are messy: repeated blank lines, stray whitespace, odd characters. A quick clean-up now saves noise later:

import re

def clean(text):
    text = re.sub(r"[ \t]+", " ", text)       # collapse runs of spaces
    text = re.sub(r"\n{3,}", "\n\n", text)     # collapse big gaps
    return text.strip()

for doc in docs:
    doc["text"] = clean(doc["text"])

A note on other formats

PDFs, Word files, and HTML need extracting into plain text first. That is a whole topic of its own (tools like Docling or a PDF text extractor handle it), so for now stick to Markdown and text files. Once your pipeline works, swapping in an extractor for other formats is a small change.

Next steps

We now have clean documents in memory. In the next lesson we will split them into small, retrievable passages: chunking.