Production sampling for LLM quality
Offline golden sets catch regressions before release. Production sampling catches drift after release: new intents, stale retrieval, model swaps the vendor pushed without telling you.
A workable pattern
- Serve the user first. Score async.
- Sample a slice (often 5–10%), stratified by topic or risk when you can.
- Oversample high-risk surfaces (payments, policy, health-adjacent copy).
- Turn recurring live failures into frozen regression cases.
Track groundedness, refusal, cost, and latency by model, prompt, and retrieval version. An average that looks fine can hide one broken route.
Where Arviser fits
LLM Quality Audit includes production sampling as one of six scored areas and leaves a suite you can keep running. Readiness maps whether any sampling path exists before you spend on scale.
Next: CI gates · Shared metrics · How we score · Request scope