GraphRAG: when a knowledge graph beats plain vector search

Plain vector Retrieval-Augmented Generation (RAG) works by embedding the query, embedding every chunk of your corpus, and returning the chunks whose vectors sit closest to the query vector. This is a similarity operation, and it is very good at one thing: finding passages that look like the question. For a lookup question (“what is the maximum grant under this scheme?”) the answer usually lives in a single passage that reads a lot like the query, so a good embedding model retrieves it and the generator does the rest.

The trouble starts when the answer is not sitting in one similar-looking passage. Two classes of question expose this, and both are common in real systems.

Where similarity search runs out

The first is the multi-hop question, where the answer requires connecting several facts that never appear together. “Which of the companies our client acquired in 2019 are headquartered in a country now under sanctions?” needs an acquisition list, a set of headquarters locations, and a sanctions list. No single chunk is similar to that query, and the chunks that matter are not especially similar to each other. Top-k similarity retrieval tends to return passages that echo the query’s surface words while missing the bridging facts entirely.

The second is the global, or corpus-level, question: “what are the main themes across these 800 incident reports?” There is no passage to retrieve because the answer is a property of the whole collection, not of any part of it. This is closer to query-focused summarisation than to retrieval, and stuffing the top-k most similar chunks into the prompt gives you a biased sample, not a synthesis.

You can paper over the first problem with iterative retrieval, and over the second with map-reduce summarisation, and both help. But a structural approach addresses the root cause: represent the corpus as entities and relationships, and retrieval becomes traversal rather than nearest-neighbour lookup.

What GraphRAG actually does

GraphRAG (Edge et al., 2024), from Microsoft Research, is the work that pushed this into the mainstream. Its pipeline has a clear shape. First, an LLM reads the source documents and extracts a knowledge graph: nodes are entities (people, organisations, places, concepts), edges are the relationships between them, and both carry short LLM-written descriptions. Because the same entity is mentioned across many documents, the graph naturally stitches the corpus together at the level of things rather than passages.

Second, a community-detection algorithm partitions the graph into a hierarchy of clusters of densely connected entities. Third, an LLM writes a summary of each community, bottom-up, so you end up with a tree of pre-computed summaries from fine-grained local clusters to broad corpus-level themes.

At query time the trick is to match the question to the right level. A global question is answered by fanning the query across the relevant community summaries, generating partial answers from each, and reducing them into one response. The paper reports that on global sensemaking questions over corpora around the million-token range, this produces more comprehensive and more diverse answers than a conventional vector-RAG baseline. Crucially, those are qualities that top-k retrieval structurally cannot deliver, because it never sees the whole corpus at once.

Note what GraphRAG is not: it is not primarily a multi-hop factoid retriever. Its headline strength is the global, query-focused summarisation case. That distinction matters when you decide whether it fits your workload.

Grounding retrieval in an existing graph

GraphRAG builds the graph from your unstructured documents. The other branch of the family starts from a graph that already exists. If your domain is covered by a public knowledge graph, you can traverse curated relationships directly instead of hoping an LLM extracts them correctly.

Wikidata is the obvious anchor here: tens of millions of entities with typed, machine-readable relationships, queryable with SPARQL, which lets you express a multi-hop question as an explicit graph path rather than a similarity guess. DBpedia offers structured facts extracted from Wikipedia with a mature ontology, and ConceptNet captures commonsense relations between everyday concepts that rarely appear as explicit statements in any document. The now-frozen Freebase still underpins many multi-hop QA benchmarks, so it is worth knowing even though it is no longer maintained. Our knowledge graphs category collects these and their neighbours, and the broader encyclopaedic category covers the Wikipedia-derived sources that most of them lean on.

The practical pattern is to link the entities mentioned in a query to nodes in the graph, walk the relevant edges to gather connected facts, and hand that structured evidence to the generator. Because the relationships are curated rather than inferred, the bridging facts a multi-hop question needs are already encoded, and you avoid the extraction errors that come with building a graph from scratch.

Hybrid traversal and vector search

The two approaches are not rivals. The strongest systems combine graph structure with vector similarity: use embeddings to find an entry point into the graph (which nodes does this query touch?), then traverse edges to gather connected context, then optionally re-rank the gathered evidence by similarity to the query.

HippoRAG (Gutierrez et al., 2024) is a clean illustration. It has an LLM build a knowledge graph over the corpus, then runs Personalised PageRank from the query’s entry nodes to do single-step multi-hop retrieval, pulling in associated passages that a one-shot similarity search would miss. The graph supplies the connectivity; the ranking algorithm supplies the relevance signal. This is the same instinct as the reflective and adaptive retrieval work such as Self-RAG (Asai et al., 2023): decide what to retrieve based on the structure of the problem, rather than always fetching a fixed top-k of look-alike chunks.

The costs are real

Structure is not free, and the bill arrives before you answer a single query. Building the graph with an LLM means running extraction over every document, which for a large corpus is a serious token cost and a slow, occasionally flaky, batch job. Entity resolution (deciding that “IBM”, “I.B.M.” and “International Business Machines” are one node) is a genuinely hard problem that graph quality lives or dies on. Community detection and summarisation add another LLM pass on top.

Then there is maintenance. When documents change, you cannot just upsert a vector; you have to decide which entities and edges are affected, re-extract, and re-summarise the touched communities. A vector index degrades gracefully as it drifts; a stale graph can quietly encode relationships that no longer hold.

For a large fraction of RAG systems, none of this is worth it. If your questions are lookups answered by a single passage, plain vector search over well-chosen chunks is faster to build, cheaper to run, and easier to keep fresh. Reaching for a graph there is over-engineering.

When to reach for it

A short decision rule holds up well in practice. Prefer graph or structured retrieval when at least one of these is true: your questions are genuinely multi-hop and need bridging facts that never co-occur; your questions are global and ask about themes, patterns or summaries across the whole corpus; or you already have a trustworthy structured source (an internal knowledge graph, or a public one like Wikidata) you can lean on instead of building from scratch. Otherwise, start with vector RAG, measure where it fails, and add structure only against the failure modes you can actually observe.

If you are assembling the raw material for either approach, the directory catalogues data sources by domain and licence, and the rest of the research section covers the retrieval and evaluation questions that sit alongside this one.