Why a working demo is not a production gate
Many teams have a polished LLM demo and still cannot ship. The demo uses curated prompts, a quiet corpus, and a friendly audience. Production adds messy queries, edge cases, cost spikes, and people who will try to break the system.
What usually breaks
- Groundedness. Answers invent fees, policies, or product facts when retrieval is thin.
- Refusal. Account-takeover and policy-bypass scripts succeed more often than the demo suggested.
- No shared score. Product says “good enough.” Risk has no number to accept or reject.
- No regression gate. Prompt and model changes ship without a suite that fails the build.
What to measure first
Agree pass marks before the next model change: task success, groundedness or faithfulness, refusal, latency, and unit cost. Put the suite in CI or on a fixed schedule. Keep patient and Art. 9 health data out of website forms and public evals.
How Arviser helps
- AI Readiness Check (from €2,900, one week): gates, risk register, kill or continue, 90-day roadmap.
- LLM Quality Audit (from €8,000, two weeks): six scored areas, eng report, one-page exec, repeatable suite.
100% of the Readiness fee credits toward the next signed Arviser engagement within 90 days. LLM Quality Audit is billed on its own and is not credited forward.
Next: Request audit scope · Sample audit · Readiness vs Audit · Checklist