Evaluate citation accuracy as its own RAG layer

An eval suite can report strong faithfulness and strong contextual recall while the footnote under a sentence points at the wrong passage, because neither metric reads the sentence-to-passage mapping your users click. DeepEval's RAG guide frames five core metrics — contextual precision, contextual recall and contextual relevancy on the retriever side, answer relevancy and faithfulness on the generator side — as component-level coverage, concedes that the metrics are extremely generic, and routes any criterion outside that set to GEval; no attribution metric appears in it.^[2] Confident AI's breakdown is explicit that faithfulness checks the output for contradiction against the retrieval context rather than checking whether the output cites that context.^[3]

Braintrust's survey of RAG evaluation tooling describes the same industry split, retrieval quality on one side and generation accuracy on the other, with the nearest attribution concept in its roundup being Galileo's Chunk Attribution scorer, which asks whether retrieved chunks grounded the answer at all.^[7] That is a whole-answer question, not a per-sentence one. Faithfulness judged over the context as a single blob can pass on an answer whose every claim is supported somewhere while each inline marker points at the wrong chunk, which is precisely the failure a reader reports as a broken citation.

So define three scores and keep them apart. Retrieval quality asks whether the supporting passage was retrieved at all. Faithfulness asks whether the claim is supported somewhere in the retrieved context. Citation correctness asks whether the cited passage supports the sentence it is attached to. The survey 'Evaluation of Retrieval-Augmented Generation' frames retrieval and generation metrics jointly as relevance, accuracy and faithfulness, and identifies shortcomings in current benchmarks; passage-level attribution is one of the gaps that framing leaves open, so expect to build this layer rather than adopt it.^[8]

Citation correctness is not computable after the fact unless the trace records which chunk each sentence cited. No framework recovers that mapping from a question, a context list, and a finished answer, so the first change is to the trace schema rather than to the metric set: every generated sentence or claim carries the chunk ID it cites, and every cited chunk ID is resolvable back to the retrieval result that produced it. Teams that skip this end up re-deriving citations with a second model call, which measures that model's guess rather than the product's behavior.

Once the spans exist, the metric is a custom judge criterion rather than a built-in. DeepEval points at GEval for anything outside its built-in set rather than shipping an attribution metric, so treat citation correctness as a criterion you author, version, and validate yourself.^[2] Braintrust's argument for splitting retrieval from generation applies directly to that authoring decision: one answer-level score cannot tell you which component failed, so a mis-citation needs its own scored dimension instead of a deduction from a blended quality figure.^[7]

Keep the five component metrics as documented, because they still localise failures to the reranker, the embedding model, chunk size and top-K, the prompt template, and the generator.^[2] The change is narrower than replacing them: stop treating faithfulness as a proxy for attribution, and report citation correctness beside it.

A meaningful slice of this work needs no annotated set and can start on live traffic. Evidently's guide lists link presence — whether the answer includes a link to the source and whether that link is valid — as a programmatic check that needs no labels, alongside reference-free faithfulness implemented as an LLM judge over question, context and answer returning a faithful or not-faithful verdict with an explanation, and it names faithfulness specifically as a production-monitoring metric because it requires no ground truth.^[1] DeepEval designates answer relevancy, faithfulness and contextual relevancy a referenceless trio that permits end-to-end evaluation without a labeled expected output.^[2]

LangSmith's RAG tutorial makes the division concrete at the evaluator level: of its four evaluators, only correctness consumes reference outputs, while relevance, groundedness and retrieval_relevance derive everything from the run's own inputs and outputs.^[4] That is the line to draw in your own suite. Evidently also documents LLM-judged retrieval relevance scored per chunk or over the whole context with a valid or invalid verdict, and cites a Microsoft result of GPT-4 reaching near-human performance on chunk-level relevance in Bing's RAG system, which is the published basis for trusting a per-chunk judge at all.^[1]

The citation-specific addition is a judge scoped to a single pair: one sentence, and only the passage that sentence cites, returning supports, partially supports, or does not support. Narrowing the input is what separates this from faithfulness over the whole context, because a judge that can see every retrieved chunk will find support somewhere and pass a mis-citation. Add deliberate hallucination probes next to it — Evidently recommends asking questions the system should not be able to answer, including missing, outdated, and false-premise questions, and verifying that it admits the gap instead of guessing and citing something.^[1]

Then state the ceiling out loud, because it decides what the reviewed set is for. Without labels you cannot compute recall, as Evidently puts plainly, so these checks find mis-citations and unsupported claims but stay silent when the correct passage was never retrieved and no citation could have been right.^[1]

The documentation is consistent about where labels become mandatory. DeepEval states that contextual precision and contextual recall require an expected output, which acts as the ground truth for what information the retrieval context should contain, and Confident AI's breakdown names those same two as the only metrics that take an expected output in its code samples.^[2][3] Confident AI also supplies the reason to keep recall paired with relevancy: without contextual recall, anyone can hit perfect contextual relevancy by retrieving as little information as possible.^[3]

Patronus requires gold labels for context sufficiency and answer correctness, and defines sufficiency through attributable statements — statements in the gold reference answer that can be directly matched with a statement in the retrieved context — which is the closest published analogue to passage-level attribution scoring.^[6] Evidently is blunt that recall is the payoff for doing the labeling work, that without a ground-truth set you cannot compute recall, and it documents the labeled retrieval metrics worth computing: precision@k, recall@k, hit rate, and NDCG@k, which rewards correct items appearing higher in the ranking.^[1]

For citation work the label has to be a passage set, not a sentence of prose. Evidently specifies labeling the relevant documents per query as document IDs, chunk IDs, or links, and that form is what citation correctness is scored against, since the question is which passage should have been under the sentence.^[1] Budget for re-labeling too: Evidently warns that changing chunk boundaries reshuffles everything already labeled, and citation-level labels are hit hardest because they are anchored to chunks.^[1] Version the set and tie every reported accuracy number to a specific version, following Patronus's practice guidance.^[6]

The Hugging Face cookbook documents a reusable generate-then-critique recipe that inverts the pipeline. It samples a chunk from the knowledge base, prompts a model for a factoid question answerable from that chunk, explicitly forbids the question from referencing the passage — no 'according to the passage' or 'context' phrasing — and stores the context, question, answer and source document per record.^[5] For citation work, the sampled chunk ID is the gold supporting passage, so record it as a label rather than as provenance metadata.

The filtering step is what keeps the set usable. Three independent critic agents each score candidates from 1 to 5: groundedness, meaning whether the question can be answered unambiguously from the given context, from unanswerable at the bottom to clearly answerable at the top; relevance to the target audience; and stand-alone context independence, where any reference to 'the context' forces the lowest score. Rows scoring 4 or above on all three survive, and the cookbook's cost guidance is concrete: aim for more than 100 final samples and therefore generate over 200, because roughly half get filtered out.^[5]

Evidently inverts the flow the same way and adds the review discipline around it: auto-generated examples must be reviewed before use, domain experts are worth pulling in for edge cases and fine-print scenarios, a few dozen to a few hundred examples suffices to compare versions, and regenerating the set between runs introduces noise that makes results unreliable.^[1] Freeze the set for the duration of an experiment series and version it, so a metric movement reflects a system change rather than a dataset change.^[6]

Deliberately include the cases where citations go wrong. Multi-source queries, where only part of the supporting evidence is retrieved, are where markers most often attach to a neighbouring chunk that looks topically right, and they rarely appear in questions generated one chunk at a time. Grow the set from production as well: fold in real user queries and the mis-citation failures your unlabeled checks already surfaced.

Two independent sources converge on the same prompt mechanics for judges like this one. LangSmith's tutorial defines every grading schema with the explanation field declared before the boolean verdict, and states that the ordering matters because the order in which fields are defined is the order the model generates them, which forces reasoning before the verdict; it runs at temperature 0 with strict structured output, and tunes strictness per evaluator, with its retrieval-relevance grader lenient by design so that it flags only facts completely unrelated to the question.^[4] The Hugging Face cookbook demands an evaluation rationale before the total rating for the same stated reason, that asking for reasoning first gives the model more tokens to think before committing to a score token, and argues for detailed per-level rubric descriptions because a vague scale produces outputs that will not be consistent enough between different examples; it normalises its 1 to 5 scores to a 0 to 1 range with (x-1)/4.^[5]

Apply that to citation support with two deliberate choices. Prefer a small discrete verdict set — supports, partially supports, does not support — over a continuous score, because the discrete labels are what reviewers can agree on. And calibrate strictness per check rather than globally: a citation-support judge should be strict about the cited passage specifically, while a retrieval-relevance judge can stay lenient about peripheral chunks, which is exactly the asymmetry LangSmith's per-evaluator tuning describes.^[4]

Validate the judge before you publish its number. Measure judge-versus-human agreement on the reviewed set and treat the judge as an instrument that itself needs calibration, not as the measurement of record. Patronus reports purpose-trained evaluators, Lynx for hallucination detection and Glider, returning pass or fail plus reasoning and claiming advantages in long-context cases, which makes them a reasonable second opinion on faithfulness while your own criterion handles attribution.^[6]

Then watch the two failure modes that quietly corrupt averages. Patronus warns about framework-level problems including RAGAS statement-extraction failures and cost from unoptimised recursive LLM calls.^[6] A claim- or statement-extraction step that fails on some inputs drops those examples from the denominator without saying so, and recursive judge calls over long contexts invite throttling, so log extraction failures as their own counted outcome rather than letting them disappear.

Publish the number separately. Citation correctness belongs in dashboards and release gates as its own figure rather than folded into an aggregate quality score, because the whole point of separating the layers is that each one routes to different work: a retrieval miss goes to embeddings, chunking and top-K, a faithfulness failure goes to grounding instructions and model configuration, and a mis-citation goes to how the generator attaches markers to sentences. Braintrust's case for splitting retrieval from generation is the same argument one level down — a single score cannot tell you which component failed.^[7]

Keep the unlabeled and labeled halves visibly distinct in the report, since they answer different questions and have different cadences. Marker resolution, link validity, the scoped citation-support judge, and the unanswerable probes run continuously on production traffic with no ground truth.^[1] The labeled half — recall@k, hit rate, NDCG@k and attributable-statement coverage — runs against the versioned reviewed set on release candidates, and it is the only place a never-retrieved supporting passage shows up at all.^[1][6]

Finally, read the three numbers together when you debug. Evidently frames debugging as explicitly asking whether the system failed to retrieve the right documents or whether the model hallucinated with correct context in hand, and the survey's joint treatment of retrieval and generation metrics is the same instinct at benchmark scale.^[1][8] Citation correctness is the third question in that sequence: the right passage was retrieved, the claim was supported, and the marker still pointed somewhere else.