Measuring RAG faithfulness: is the answer actually supported?

A retrieval-augmented generation (RAG) system can pull the exactly right passage into context and still produce an answer that passage never supports. It can also be correct by luck, stating a true fact that happens not to appear in anything it retrieved. Both cases pass a naive “is the answer right?” check and both are failures of grounding. The property that separates a trustworthy RAG answer from a plausible one is faithfulness: every claim in the generated answer is entailed by the retrieved context. This piece looks at how to measure it, the resources built for the job, and how far to trust the automatic numbers.

Faithfulness is not correctness

It helps to keep three things apart. Correctness asks whether the answer matches ground truth. Relevance asks whether the answer addresses the question. Faithfulness asks whether the answer’s claims are supported by the passages the system actually retrieved. These come apart constantly in practice.

A model can retrieve a strong passage and then embellish it, adding a date or a figure that is not in the text. That is unfaithful even if the added detail turns out to be true. A model can also ignore a correct retrieval and answer from parametric memory, which is unfaithful even when the answer is right, because the stated support does not hold. For anything where a user might act on the answer, a legal summary, a clinical note, a financial figure, faithfulness is the property you can defend. Correctness alone tells you the answer was right this time; faithfulness tells you the system did the thing it claimed to do.

This is the research-depth companion to our beginner-level walk-through in evaluating your RAG system. Here the focus is narrower: grounding, and how to measure it without fooling yourself.

Why retrieval quality does not settle it

Most evaluation effort still lands on the retriever, and for good reason: nothing downstream can be faithful to context that was never fetched. Retrieval benchmarks such as BEIR and MTEB tell you whether the right documents surface, and the wider retrieval-benchmarks category exists precisely to pressure-test that stage. But a perfect retriever bounds faithfulness from above without guaranteeing it. The generator still has to use what it was given and nothing else.

That generation step is where faithfulness evaluation earns its keep, and it needs its own corpora and its own metrics. The rag-benchmarks category collects resources aimed at the whole pipeline, not just the lookup.

RAGTruth: annotated hallucination at the span level

The most useful single resource here is RAGTruth (Niu et al., 2024, arXiv:2401.00396). It is a corpus of nearly 18,000 responses generated by a range of large language models in genuine RAG settings, spanning question answering, data-to-text writing, and news summarisation. Crucially, the annotation is not a single pass or fail label per answer. Human annotators marked hallucinations at the word or span level, and sorted each into a four-part taxonomy: evident conflict, subtle conflict, evident baseless information, and subtle baseless information. The dual and triple annotation protocol reported response-level agreement above 91 per cent, which is high for a task this subjective.

Span-level labels change what you can build. Instead of only scoring whether a whole answer is faithful, you can train a detector that flags the exact tokens a model invented. The RAGTruth authors showed that a relatively small model fine-tuned on the corpus reaches detection performance competitive with prompting a much larger frontier model. Follow-up work has pushed this further. LettuceDetect (Kovács et al., 2025, arXiv:2502.17125) trains a ModernBERT-based token classifier on RAGTruth that handles long context and runs at tens of examples per second on one GPU, while Lynx (Ravi et al., 2024, arXiv:2407.08488) fine-tunes a larger open model on a mix that includes RAGTruth and frames the task as natural language inference: does the context entail the answer or contradict it? The natural language inference lens is a recurring theme, and it makes the underlying question explicit. Faithfulness is entailment.

The RAGAS decomposition

The other widely used approach does not train a detector at all. RAGAS (Es et al., 2023, arXiv:2309.15217) proposes a reference-free suite that breaks evaluation into components you can inspect separately. Faithfulness is computed by decomposing the answer into individual claims and checking what fraction of them can be inferred from the retrieved context. Answer relevance measures how well the answer addresses the question. Context precision and context recall score the retrieval. Reporting these as separate numbers rather than one blended score is the point: a low faithfulness score with high context precision tells you the generator is the problem, while the reverse points at retrieval.

RAGAS runs these checks by prompting a language model, and the original paper reported agreement with human annotators of around 95 per cent on faithfulness. Treat that figure as evidence the metric is useful, not as proof it is calibrated for your domain. The decomposition is doing real work, but every sub-metric ultimately leans on a model judging text, which brings us to the caveats.

LLM-as-judge, and why to distrust it a little

Most faithfulness metrics in production today, including the RAGAS style, reduce to an LLM-as-judge call: show a model the context and the answer, ask whether each claim is supported. It is cheap, fast, and correlates with human judgement well enough to be worth running. It is also biased in ways that matter.

The foundational study here, Zheng et al. (2023, arXiv:2306.05685), introduced MT-Bench and Chatbot Arena and catalogued the failure modes directly: position bias (the judge favours whichever answer it sees first), verbosity bias (longer answers score higher), self-enhancement or self-preference bias (a model rates its own outputs more generously), and plain limits on reasoning. For faithfulness scoring specifically, self-preference is the sharpest risk: if the same model family generates and judges, unfaithful-but-fluent answers can be waved through. Prompt sensitivity compounds this. Small changes to the judging prompt move scores enough to reorder systems.

None of this makes LLM-as-judge useless. It makes it a measurement instrument that needs calibrating rather than a ground truth. Practical mitigations are well established: swap the order of options and average to cancel position bias, use a judge from a different model family than the generator, pin and version the exact judging prompt so scores stay comparable across runs, and prefer a claim-by-claim entailment check over a single holistic verdict, since narrower questions are more stable. A span-level detector trained on RAGTruth makes a useful second opinion precisely because it was fitted to human labels rather than prompted cold.

Anchoring on human-labelled data

Whatever automatic metric you run, anchor it to a small set you have checked by hand. Benchmarks help you build that anchor. Open-RAGBench offers an open, reproducible set of queries with retrieval corpora and answers for end-to-end evaluation, and classic question-answering sets such as Natural Questions give you real user queries with human-annotated answers to test grounding against. The datasets directory and the rag-benchmarks category list more. The point of the hand-checked set is not scale; a hundred carefully adjudicated examples will tell you whether your automatic faithfulness score tracks reality on your data, which no leaderboard number can.

Putting it into a pipeline

A workable faithfulness setup combines three layers rather than betting on one metric. Run an automatic decomposed metric, RAGAS-style faithfulness plus a trained span detector, on every change so regressions surface fast. Keep a small human-labelled evaluation set and periodically confirm the automatic numbers still correlate with it, re-checking whenever you change the judge model or its prompt. And log the retrieved context alongside every generated answer, because a faithfulness failure you cannot trace to specific passages is a failure you cannot fix.

The last layer is behavioural, not just diagnostic. When support is weak, the safe response is to abstain or hedge rather than answer confidently. A calibrated span detector gives you the signal to gate on: if a large fraction of the answer’s claims are unsupported, suppress the answer, ask for clarification, or route to a human. That is often more valuable than any single point on a benchmark, because in the settings where faithfulness matters most, a confident wrong answer costs far more than an honest “I could not find support for this.”

Faithfulness will not fall out of a better retriever or a bigger generator on its own. It is a distinct property, measured with distinct tools, and worth measuring directly. For the ground-up version of the wider evaluation problem, start with evaluating your RAG system; for the retrieval side that bounds all of this, see our note on reranking and two-stage retrieval.