RAG evaluation: retrieval and generation must gate separately
A RAG demo can look strong while retrieval is weak. The generator papers over misses until traffic or a new index rebuild exposes them. Gate retrieval and generation on their own thresholds.
What to measure
- Retrieval: hit rate or recall on a frozen set of queries with known sources
- Generation: groundedness or faithfulness against retrieved context
- Refusal when context is missing, conflicting, or out of policy
- Cost and latency budgets per successful task
Agree pass marks before the run. Re-run after prompt, model, chunking, embedding, or index changes.
How Arviser uses this
LLM Quality Audit (from €8,000) scores groundedness, golden set, and a suite you can keep. AI Readiness Check (from €2,900) maps whether those gates exist at all.
Next: Shared metrics · Golden set · CI gates · Request scope