Arviser
LLM Quality Audit
Two weeks. Fixed scope in writing. Scores, failure modes, and a suite you can re-run before go-live. Overrun on us.
From €8,000 · 50% on signature · 50% on delivery · no forward fee credit
What a scorecard looks like
Synthetic example. Same component as the home page and sample. Live audits use your thresholds.
What it is
Independent measurement of a customer-facing or internal LLM feature on a documented test set: task success, groundedness, refusal, regression coverage, production sampling, cost and latency. You leave with ranked failure modes and a suite engineers can re-run. Open tools where they fit (Promptfoo, DeepEval, Ragas, LangSmith). Decision and evidence pack, not a platform.
Examples of the founder's open benchmarks and engineering figures are in the Open proof section on the home page.
Technical LLM audit vs Big4 vs certification bodies
Arviser measures a live LLM or RAG surface. It is not a certification body.
| Axis | Arviser Technical LLM Quality Audit | Big4 Governance Audit | Certification Bodies (SGS, BSI, TÜV) |
|---|---|---|---|
| Primary Focus | Live LLM or RAG surface: scores, failure modes, repeatable suite | Policy, controls, programme design, board reporting | Conformity assessment against a published standard |
| Who Executes | Senior engineers (10+ yrs) under founder oversight | Advisory and risk teams; large programme staffing | Accredited auditors and assessors |
| Deliverables | Eng report, 1-page exec, CI harness when access allows, 60-minute review | Governance findings, control maps, remediation roadmap | Certificate or conformity statement when the scheme allows |
| Timeline | 2 weeks fixed scope | Often multi-month programmes | Scheme-dependent assessment cycles |
| Published Price | From €8,000 fixed scope (€6,800 Startup Co-Branded) | €40,000–€150,000+ | Conformity certification (scheme fees, not a fixed LLM audit SKU) |
Published Pricing: from €8,000 (€6,800 Co-Branded) vs European market benchmarks above. Not legal advice. Not EU AI Act conformity assessment.
Six areas we score
Golden set
Labeled cases for the live surface. Breadth, label quality, freshness. Guide.
Groundedness and faithfulness
Answers tied to allowed sources. Inventing vs refusing when evidence is missing. RAG evaluation.
Refusal and safety
Jailbreaks, policy breaches, unsafe tool use where agents act. Refusal both ways.
CI and regression
Evals on change with pass or fail gates, or only on a laptop. CI harness when access allows. CI gates.
Production sampling
Live traffic feeding the suite, or a frozen demo set. Production sampling.
Cost and latency
Unit cost and delay under realistic load, including retries. Cost and latency.
Scoring rubric (1–5)
| Score | Meaning |
|---|---|
| 5 | Meets threshold with headroom; regressions gated. |
| 4 | Meets threshold; minor gaps documented. |
| 3 | Borderline; go-live only with explicit risk acceptance. |
| 2 | Fails on material cases; remediate before scale. |
| 1 | Unsafe or unmeasured; no-go until evidence rebuilt. |
Pass marks are agreed before final scoring. Full rubric.
Week 1 and week 2
- Week 1. Kickoff, NDA, access, golden set. Agree pass marks in writing.
- Week 2. Measure six areas, package evidence, 60-minute readout.
One LLM or RAG surface in the base price. Multi-agent or multi-locale scoped separately. Detailed timeline.
What you get
- Executive summary (1 page). Leadership reads this: verdict, top risks, next step.
- System map. Surfaces, owners, environments in scope.
- Method and limits. What we ran, how often, what we did not claim.
- Scores and failure modes. Six areas with examples, not only averages.
- Ranked fixes. Severity and engineer-day sizing.
- Reusable suite. Cases and rubrics; CI harness when access allows.
Typical eng report is about 20–30 pages plus the one-page exec summary and raw suite appendix. Plus a 60-minute review with the person who ran the work.
Deliverables are software quality metrics and suites. Not clinical evidence under MDR/IVDR.
EU AI Act dates we can evidence (add-on)
Orientation for counsel. Not legal advice. Not certification.
| From | What applies | What we check |
|---|---|---|
| 2 Feb 2025 | Art. 5 prohibitions, Art. 4 literacy | In-scope behavioural patterns; literacy docs as process evidence |
| 2 Aug 2026 | Art. 50 transparency | AI self-identification; content labelling under your config |
| 2 Dec 2026 | Machine-readable marking | Technical marks for media outputs where in scope |
| 2 Dec 2027 | Annex III high-risk | Logging, oversight, data-quality notes for counsel |
| 2 Aug 2028 | Annex I product AI | Boundary notes when LLM sits in a regulated product |
Ideal for
- Customer-facing LLM or RAG in pilot or early production.
- Evidence before scale, procurement, or counsel review.
- Independent score of what a build vendor shipped.
Not included
- AI Act conformity assessment or “certified” claims.
- Clinical evidence under MDR/IVDR.
- Patient data via website forms.
- Full product build (Pilot-to-Production is separate).
Price
From €8,000 excl. VAT. Scope fixed in writing before kickoff. If work runs long inside that scope, that is on us.
Payment: 50% on signature, 50% on report delivery.
Working baseline: no prioritized roadmap and working eval baseline for the agreed surface means you do not pay the Audit fee.
Credit: LLM Quality Audit is billed on its own and is not credited toward later packages. Readiness Check and Founder Consultation keep their own credit windows.
FAQ
What access do you need?
Read-only where possible: prompts, retrieval config, sample traces, CI logs, eval configs. NDA before sensitive material. Prefer your environment. No patient data via website forms.
We have no golden set yet.
Common. Absence is often a top finding. We sample production or staging traces, cluster by intent, and bootstrap a candidate set as part of the audit.
How is the €8,000 split?
Half on signature, half on delivery of the report and suite. No day-rate clock inside the written scope.
Must we use OpenAI?
No. Method is provider-agnostic: Anthropic, Google, open-weight, hosted, or self-hosted.
Does this certify EU AI Act compliance?
No. Optional add-on is technical evidence for your counsel. Not legal advice. Not conformity assessment.
Request written scope or ask a question
Free written scoping and technical Q&A reply within 48 hours on workdays (Warsaw), or select Direct Founder Consultation (€250/h, 100% credited in 30 days) below. Controller: Arviser Sp. z o.o. Privacy. No patient data.