Hardware for a local RAG system: what you actually need
A practical guide to sizing hardware for local RAG, from RAM and GPU for embeddings and local models to the storage your corpus and vector index will really consume.
Running a Retrieval-Augmented Generation (RAG) system locally, on your own machine or server rather than a cloud API, is a reasonable goal for privacy, cost control, and learning. The hardware question that follows is usually asked backwards. People ask which graphics card to buy before they know how much text they are indexing or whether they need a local generation model at all. This guide works through the four things that actually determine your requirements: storage, RAM, GPU and VRAM, and then how they combine into three practical tiers. It stays vendor-neutral and avoids specific products, because the right answer depends far more on the shape of your corpus than on any single component.
Storage: three very different numbers
The most common sizing mistake is to plan for one storage figure when there are really three, and they can differ by orders of magnitude.
- The raw source download. This is the data as you first obtain it, often compressed archives, and it is usually the largest number.
- The extracted and cleaned text. After you strip markup, boilerplate, and duplicates, and keep only the fields you need, this is typically much smaller than the raw download.
- The vector index. This holds the embeddings (numeric representations of your text chunks) plus some structural overhead. Its size depends on how many chunks you have and the embedding dimension, not on the byte size of the original text.
A concrete example helps. A single-language extract of Wikipedia is on the order of tens of gigabytes of clean text, which is comfortable on almost any modern disk. Common Crawl, by contrast, is petabyte-scale, so you never store all of it. You always subset: a language, a time window, a domain list, or a topic filter. If you find yourself planning storage for “all of Common Crawl”, that is a sign to narrow the corpus first. You can browse candidate corpora by size and licence in our directory and by theme in the web-corpora and encyclopaedic categories.
Estimating the vector index
The index size is the part people most often get wrong, and it is the easiest to estimate. A useful rule of thumb:
number_of_chunks multiplied by embedding_dimension multiplied by 4 bytes (for float32)
So one million chunks at 768 dimensions is roughly one million times 768 times 4 bytes, which is about 3 GB before overhead. Double the chunk count and you double the size. Move to a 1,536-dimension embedding model and you double it again. The graph structure used by fast search adds further overhead on top, so treat the raw figure as a floor rather than a ceiling.
That number is not fixed. Quantising the vectors, storing each dimension as an 8-bit integer rather than a 32-bit float, cuts the footprint to roughly a quarter. Product quantisation, which compresses groups of dimensions together, reduces it much further again, at some cost to retrieval accuracy. For many local systems the accuracy trade-off is acceptable, and it is worth measuring on your own queries rather than assuming.
The disk itself
For the index, prefer a fast NVMe SSD (a solid-state drive on the fast PCIe interface) over a spinning hard disk. Vector search touches many small, scattered reads, and the latency difference is felt directly in query time. Spinning disks are still fine for the raw archive and cold backups, where throughput matters more than latency. Whatever the size, leave generous headroom: intermediate files during extraction, index rebuilds, and a second copy while you migrate all want space, and a full disk is a slow disk.
RAM: where the index lives
The central RAM decision is whether the vector index sits in memory or on disk.
- In memory. Graph-based indexes such as HNSW (Hierarchical Navigable Small World) are fast precisely because they traverse the index in RAM. If your index is a few gigabytes and your machine has room to hold it alongside everything else, this is the responsive option.
- On disk. Disk-backed indexes keep most of the vectors on the SSD and page them in as needed. This is cheaper and lets a modest machine handle a large corpus, at the cost of higher and less predictable query latency.
Sizing memory is largely about holding the index plus working room. Take your estimated index size, add space for the operating system and any local generation model, and add headroom for embedding in batches: generating embeddings for many chunks at once is faster than one at a time, and each batch occupies memory while it is processed. If you are undecided, planning for the index to fit in RAM is the simpler path and removes a whole class of latency problems.
GPU and VRAM: it depends what runs locally
This is where budgets are won or lost, and the key distinction is between the two models in a RAG pipeline.
The embedding model converts text into vectors. At small and moderate scale it is light, and running it on the CPU is often perfectly workable, especially for a one-off indexing pass where you do not mind waiting. A GPU speeds up bulk embedding, but it is rarely the thing that forces a purchase.
The generation model, the local language model that writes the answer, is the demanding part. A quantised small model, in the range of roughly 7 to 8 billion parameters at 4-bit precision, needs on the order of 5 to 8 GB of VRAM (the memory on the graphics card), which many consumer GPUs provide. Larger models need considerably more VRAM, and the requirement grows quickly with parameter count. If a model does not fit in VRAM it can run on the CPU instead, but inference is much slower, sometimes to the point where interactive use is uncomfortable. Treat these figures as approximate: exact needs vary with quantisation method, context length, and the runtime you use.
A pragmatic route for local RAG is to keep generation modest, or to skip a local generation model entirely and send only the retrieved context to a hosted model, keeping your documents and the search step on your own hardware. The tooling that makes these choices concrete, vector stores, embedding runtimes, and serving frameworks, is catalogued under rag-infrastructure.
Three practical tiers
With those trade-offs in mind, most local setups fall into one of three shapes. The numbers below are approximate and meant as reasoning, not specifications.
Laptop, for prototyping. A recent laptop with a healthy amount of RAM and an SSD is enough to build and query an index of a focused corpus, think a documentation set, a subset of Wikipedia, or a modest domain collection. Embeddings run on the CPU. Generation either goes to a hosted API or uses a small quantised model, accepting slower responses. This tier is ideal for following along with a build such as our DIY RAG course before committing to more hardware.
Workstation, with a consumer GPU. Adding a single consumer graphics card with enough VRAM to hold a quantised small generation model makes local answers interactive and speeds up bulk embedding. RAM is sized to keep the index in memory, and an NVMe SSD holds the corpus and index with headroom. This tier comfortably handles indexes in the low single-digit gigabytes and larger with quantisation.
Small server, for a shared or always-on system. When several people query the system, or it runs continuously, the priorities shift to capacity and reliability: more RAM so a larger index stays resident, more and faster storage, and enough GPU memory for the generation model you have chosen. This is also where a disk-backed index earns its place, letting a large corpus live on SSD while RAM handles the hot working set.
Across all three, the discipline is the same: measure your chunk count and embedding dimension, estimate the index, decide where it lives, and only then decide what generation demands. Choosing a well-scoped dataset early makes every later decision smaller, which is the theme of choosing your first RAG dataset, and pre-embedded collections such as Cohere’s Wikipedia embeddings can remove the indexing step entirely for a first build. If you are still weighing options, the glossary explains the terms used here, and the wider blog covers the data side in more depth.
We plan to follow up with specific, current hardware recommendations, including concrete configurations at each tier, once we can keep them up to date. For now, size from your corpus outward and you will avoid buying for a problem you do not have.