Arviser

Arviser

LLM Quality Audit

Two weeks. Fixed scope in writing. Scores, failure modes, and a suite you can re-run before go-live. Overrun on us.

From €8,000 · 50% on signature · 50% on delivery · no forward fee credit

What a scorecard looks like

Synthetic example. Same component as the home page and sample. Live audits use your thresholds.

What it is

Independent measurement of a customer-facing or internal LLM feature on a documented test set: task success, groundedness, refusal, regression coverage, production sampling, cost and latency. You leave with ranked failure modes and a suite engineers can re-run. Open tools where they fit (Promptfoo, DeepEval, Ragas, LangSmith). Decision and evidence pack, not a platform.

Examples of the founder's open benchmarks and engineering figures are in the Open proof section on the home page.

Technical LLM audit vs Big4 vs certification bodies

Arviser measures a live LLM or RAG surface. It is not a certification body.

Arviser Technical LLM Quality Audit vs Big4 Governance Audit vs Certification Bodies (SGS, BSI, TÜV)
Axis Arviser Technical LLM Quality Audit Big4 Governance Audit Certification Bodies (SGS, BSI, TÜV)
Primary Focus Live LLM or RAG surface: scores, failure modes, repeatable suite Policy, controls, programme design, board reporting Conformity assessment against a published standard
Who Executes Senior engineers (10+ yrs) under founder oversight Advisory and risk teams; large programme staffing Accredited auditors and assessors
Deliverables Eng report, 1-page exec, CI harness when access allows, 60-minute review Governance findings, control maps, remediation roadmap Certificate or conformity statement when the scheme allows
Timeline 2 weeks fixed scope Often multi-month programmes Scheme-dependent assessment cycles
Published Price From €8,000 fixed scope (€6,800 Startup Co-Branded) €40,000–€150,000+ Conformity certification (scheme fees, not a fixed LLM audit SKU)

Published Pricing: from €8,000 (€6,800 Co-Branded) vs European market benchmarks above. Not legal advice. Not EU AI Act conformity assessment.

Six areas we score

Golden set

Labeled cases for the live surface. Breadth, label quality, freshness. Guide.

Groundedness and faithfulness

Answers tied to allowed sources. Inventing vs refusing when evidence is missing. RAG evaluation.

Refusal and safety

Jailbreaks, policy breaches, unsafe tool use where agents act. Refusal both ways.

CI and regression

Evals on change with pass or fail gates, or only on a laptop. CI harness when access allows. CI gates.

Production sampling

Live traffic feeding the suite, or a frozen demo set. Production sampling.

Cost and latency

Unit cost and delay under realistic load, including retries. Cost and latency.

Scoring rubric (1–5)

ScoreMeaning
5Meets threshold with headroom; regressions gated.
4Meets threshold; minor gaps documented.
3Borderline; go-live only with explicit risk acceptance.
2Fails on material cases; remediate before scale.
1Unsafe or unmeasured; no-go until evidence rebuilt.

Pass marks are agreed before final scoring. Full rubric.

Week 1 and week 2

  1. Week 1. Kickoff, NDA, access, golden set. Agree pass marks in writing.
  2. Week 2. Measure six areas, package evidence, 60-minute readout.

One LLM or RAG surface in the base price. Multi-agent or multi-locale scoped separately. Detailed timeline.

What you get

  1. Executive summary (1 page). Leadership reads this: verdict, top risks, next step.
  2. System map. Surfaces, owners, environments in scope.
  3. Method and limits. What we ran, how often, what we did not claim.
  4. Scores and failure modes. Six areas with examples, not only averages.
  5. Ranked fixes. Severity and engineer-day sizing.
  6. Reusable suite. Cases and rubrics; CI harness when access allows.

Typical eng report is about 20–30 pages plus the one-page exec summary and raw suite appendix. Plus a 60-minute review with the person who ran the work.

Deliverables are software quality metrics and suites. Not clinical evidence under MDR/IVDR.

EU AI Act dates we can evidence (add-on)

Orientation for counsel. Not legal advice. Not certification.

FromWhat appliesWhat we check
2 Feb 2025Art. 5 prohibitions, Art. 4 literacyIn-scope behavioural patterns; literacy docs as process evidence
2 Aug 2026Art. 50 transparencyAI self-identification; content labelling under your config
2 Dec 2026Machine-readable markingTechnical marks for media outputs where in scope
2 Dec 2027Annex III high-riskLogging, oversight, data-quality notes for counsel
2 Aug 2028Annex I product AIBoundary notes when LLM sits in a regulated product

Full dates guide.

Ideal for

  • Customer-facing LLM or RAG in pilot or early production.
  • Evidence before scale, procurement, or counsel review.
  • Independent score of what a build vendor shipped.

Not included

  • AI Act conformity assessment or “certified” claims.
  • Clinical evidence under MDR/IVDR.
  • Patient data via website forms.
  • Full product build (Pilot-to-Production is separate).

Price

From €8,000 excl. VAT. Scope fixed in writing before kickoff. If work runs long inside that scope, that is on us.

Payment: 50% on signature, 50% on report delivery.

Working baseline: no prioritized roadmap and working eval baseline for the agreed surface means you do not pay the Audit fee.

Credit: LLM Quality Audit is billed on its own and is not credited toward later packages. Readiness Check and Founder Consultation keep their own credit windows.

FAQ

What access do you need?

Read-only where possible: prompts, retrieval config, sample traces, CI logs, eval configs. NDA before sensitive material. Prefer your environment. No patient data via website forms.

We have no golden set yet.

Common. Absence is often a top finding. We sample production or staging traces, cluster by intent, and bootstrap a candidate set as part of the audit.

How is the €8,000 split?

Half on signature, half on delivery of the report and suite. No day-rate clock inside the written scope.

Must we use OpenAI?

No. Method is provider-agnostic: Anthropic, Google, open-weight, hosted, or self-hosted.

Does this certify EU AI Act compliance?

No. Optional add-on is technical evidence for your counsel. Not legal advice. Not conformity assessment.

Request written scope or ask a question

Free written scoping and technical Q&A reply within 48 hours on workdays (Warsaw), or select Direct Founder Consultation (€250/h, 100% credited in 30 days) below. Controller: Arviser Sp. z o.o. Privacy. No patient data.

1-click prompt templates:
Optional details (role, industry, timezone)

Discuss your project