Why it matters
RAG evaluation tests two systems at once: did retrieval find the right evidence, and did generation use it correctly? Optimizing only end-to-end scores hides retrieval failures the model patched over with parametric guesses.
How it works
Build golden question sets with expected source IDs and answer assertions. Score retrieval recall and precision separately from answer correctness, citation presence, and policy compliance.
Run evaluations on schedule and on every knowledge release.
Example
Nintendo maintains 200 Help Center questions with expected Refund Policy section IDs. Weekly eval reports show retrieval missed the pro-rata section on 8 percent of billing questions, prompting chunk boundary fixes before answer quality degrades.
Common mistakes
- 1Eval sets that only check final text, not retrieved sources
- 2Questions scraped once and never updated after policy changes
- 3Ignoring citation-required failures as cosmetic
Definitions are useful. Governed knowledge is better.
Wiki helps you turn these concepts into a real system for the knowledge behind your AI.
See Wiki in action