Graph RAG

Graph RAG is a variant of retrieval-augmented generation that uses a knowledge graph as the retrieval substrate — or as a complement to vector search — rather than relying only on similarity search over embedded documents. By representing information as entities and relationships, Graph RAG captures structural context that flat document chunks lose: which people work at which company, which product depends on which service, which policy supersedes which other policy.

The technique gained prominence in 2024 through Microsoft Research's GraphRAG project, which demonstrated that graph-structured retrieval could answer certain kinds of questions substantially better than standard vector-search RAG. Since then it has become a mainstream RAG variant supported by major LLM tooling frameworks and vector database platforms.

The problem Graph RAG addresses

Standard vector-based RAG works well when the answer to a query lives in one or two contiguous passages of text. The system embeds documents, indexes them, and returns the top-k passages most similar to the query embedding. The LLM synthesizes those passages into an answer.

This breaks down for two categories of questions. First, questions that require synthesizing information across many disconnected sources — "What are all the ways our product integrates with Salesforce?" — because vector search returns a handful of passages when the answer requires aggregating dozens. Second, questions that depend on structural relationships — "Which of our customers is most similar to Acme Corp?" — because similarity in text is not the same as similarity in structural attributes.

Graph RAG addresses both by capturing entities and their relationships explicitly. Instead of searching a flat corpus of documents, the system traverses a graph where nodes are entities (companies, products, policies, people) and edges are relationships (owns, uses, supersedes, reports to).

How Graph RAG works

A Graph RAG pipeline typically has four stages.

Extract entities and relationships. An LLM processes source documents and extracts structured triples: (Entity A, relationship, Entity B). This is done once as an indexing step. For a knowledge base of 10,000 pages, this extraction might yield 50,000 to 500,000 triples depending on document density.

Build the graph. The extracted triples are loaded into a graph database (Neo4j, TigerGraph, or a graph layer over a vector DB). Entities become nodes; relationships become edges. Entities that appear in multiple documents are consolidated into a single node — Acme Corp mentioned in a sales document, a support ticket, and a case study becomes one node with three attached document references.

Cluster and summarize. The graph is analyzed for community structure — clusters of densely connected nodes that represent coherent topical areas. Each community gets an LLM-generated summary. These summaries become searchable retrieval units alongside the raw entities and relationships.

Query the graph. At query time, the system identifies relevant starting entities (typically through a hybrid of keyword and vector search), traverses the graph to gather related information, and passes the traversed subgraph to the LLM for synthesis. For broad questions, community summaries are retrieved. For specific questions, the traversal is deeper but narrower.

Global vs local queries

Graph RAG is especially useful for "global" queries that require reasoning across a corpus. Consider the question "What are the main themes in our customer feedback?" A vector RAG system would retrieve a few semantically similar passages and produce a partial answer. A Graph RAG system with pre-computed community summaries can produce a genuinely global answer because it has summaries of every thematic cluster in the corpus.

For "local" queries with a specific answer in the corpus — "What's Acme Corp's contract renewal date?" — standard vector RAG is often faster and equally effective. Well-designed Graph RAG systems combine both: they use vector search for local queries and graph traversal or community summaries for global ones, selecting the appropriate strategy based on the query type.

What Graph RAG costs

Graph RAG's benefits come with real costs.

Indexing cost. Extracting entities and relationships from a large corpus requires many LLM calls. For a 10,000-page knowledge base, initial indexing can cost thousands of dollars in LLM inference and take hours to days. Incremental updates as documents change require rebuilding affected subgraphs.

Storage and complexity. Maintaining a graph database alongside (or instead of) a vector database adds an operational layer to the RAG stack. Graph databases have their own query languages, tuning knobs, and failure modes.

Query latency. Graph traversal can be slower than vector search, especially for deep traversals. Well-designed systems cap traversal depth or use precomputed community summaries to avoid runtime traversal for the most common queries.

These costs are worth paying when the corpus has strong relational structure and users regularly ask questions that span multiple documents. They are not worth paying for smaller corpora or use cases dominated by simple lookup queries.

Where Graph RAG fits in an AI stack

Graph RAG is one of several patterns that extend standard RAG. Others include agentic RAG (using an agent to iteratively refine the retrieval), hybrid search (combining vector and keyword retrieval), and reranking (re-scoring an initial retrieval with a more expensive model).

The right approach for a given application depends on the queries users actually ask, the structural richness of the corpus, and the cost budget for indexing and inference. For customer-support AI agents working over a large, richly-connected knowledge base — with entities like products, features, error codes, and policies — Graph RAG often measurably outperforms flat vector RAG on the hardest questions.