AI production-readiness checklist
This checklist helps you measure and document readiness. It does not certify safety, CE/MDR, or EU AI Act conformity.
1. Purpose and ownership
- Named product owner and engineering owner for the LLM feature
- Written intended use (what the system must and must not do)
- Kill criteria agreed (when we stop the pilot)
2. Data and retrieval
- Sources of truth listed (docs, APIs, tickets)
- Access control on retrieval matches source systems
- PII and secrets policy for prompts and logs
- Health-tech only: no patient data in public evals; synthetic or client-vaulted sets
3. Quality gates
- Golden and eval set exists (≥50 cases for a thin product; more for customer-facing)
- Metrics defined: grounding, task success, refusal, latency, cost per task
- Pass/fail thresholds written before the next model or prompt change
- Regression suite runs in CI or on a fixed schedule
4. Safety and abuse
- Prompt-injection and jailbreak cases in the suite
- Tool and agent permissions inventoried (least privilege)
- Human escalation path when the model abstains or fails
5. Operations
- Logging of inputs/outputs with retention limits
- Cost and latency dashboards
- On-call or owner for quality regressions
- Rollback plan for prompt/model/config
6. Documentation for stakeholders
- One-page risk summary for leadership
- Evidence folder counsel can reuse (tests, scores, gaps)
- EU AI Act: inventory row and transparency notes if customer-facing — with your lawyer, not as “certified”
Score yourself
| Checks done | Signal |
|---|---|
| Under 10 | Pilot theatre — Readiness Check maps blockers in one week |
| 10–18 | Partial — LLM Quality Audit measures the live system |
| 19+ | Ready to harden — still re-test after every material change |
Next step: Request audit scope on arviser.com. Optional: sample LLM Quality Audit (work email for the full file).