Arviser

AI security risks — what can go wrong in plain language

An AI assistant that reads your documents or takes actions can leak data, show the wrong person’s information, or do something nobody approved — even when it looks internal. This catalog helps you pick what matters before a review. Not certification.

Product: AI Security Review. Markdown source for tools: /guides/ai-security-risks.md. Open catalog on GitHub: nareyko/ai-security-risks (MIT; links WardBench open proof).

If the assistant can read untrusted text, see sensitive data, and send something outward (link, email, API, ticket), treat that as urgent — lethal trifecta.

1. Untrusted content steering the assistant

ID In plain words Example What it costs the business Typical systems Ask your team
R-01 Instructions hide inside a document, ticket, or log A support ticket quietly tells the bot to ignore policy and list customer emails Policy bypass, silent data leak, wrong automated action Doc search; support bots; log copilots Can text we did not type change what the assistant does?
R-02 Uploaded files hide instructions the UI never shows A PDF in the knowledge base carries white-on-white instructions the uploader never saw Same as above via PDF/HTML tricks Upload, OCR, email attachments Do we strip or quarantine odd file formats before indexing?
R-03 Someone plants content that ranks for many questions One planted page answers dozens of common FAQ queries ahead of real docs One bad page reaches lots of users Search over a large knowledge base Can one low-trust document dominate answers for common queries?

2. Data ending up with the wrong people

ID In plain words Example What it costs the business Typical systems Ask your team
R-04 Secrets or personal data appear in answers or logs Chat pastes an API key that lived in an indexed config file Credential theft; privacy incidents; contract breach Anything indexed from configs, logs, CRM Would a fake secret we plant in the index show up in chat?
R-05 User A sees customer/team B’s information Tenant A’s user gets chunks that belong only to tenant B Cross-tenant incident; lost trust Multi-tenant SaaS; shared company search Are permissions checked before text goes into the model?
R-06 A copy of the search index is as sensitive as the files Someone with vector-store access recovers readable text from embeddings and backups Breach of “just embeddings” still exposes content Vector databases and their backups Who can download the index, and is it encrypted like source data?
R-07 Internal system instructions or tool list leak A user coaxing the chat reveals the system prompt and tool names Easier attacks; exposed architecture Any chat product Can a user extract the system prompt or tool names?
R-08 Indexes, backups, or debug traces are too widely shared Broad Datadog access lets staff read full production prompt traces Same leak without using the chat at all Ops access to stores and traces Who can read production indexes and prompt logs?

3. The assistant does things, not only answers

ID In plain words Example What it costs the business Typical systems Ask your team
R-09 It changes systems or spends money without a real approval The agent refunds a payment or changes ACL with no human click Outages; fraud; irreversible actions Agents with tools Which actions need a human click every time?
R-10 It runs with stronger rights than the employee using it A normal user’s chat runs tools under an admin service account One user effectively becomes admin Shared service account agents Does the tool use the employee’s identity, or a god-token?
R-11 It can run code or shell outside a tight box A coding agent executes shell commands on a host that can reach production Host compromise; lateral movement Coding agents; run-this tools Where does generated code execute, and what can it reach?
R-12 A plugged-in tool’s description steers unsafe behavior An MCP connector description nudges the model toward a dangerous tool call Supply-chain style abuse via connectors MCP / plugin ecosystems Do we inventory and review every connector description?

4. Memory, long-running agents, and chains of agents

ID In plain words Example What it costs the business Typical systems Ask your team
R-13 Saved memory or a multi-step plan is hijacked Poisoned memory makes the next sessions chase an attacker’s goal Lasting wrong behavior across sessions Stateful assistants Who can write to agent memory, and how do we wipe it?
R-14 One agent persuades another without strong checks Agent A tells agent B to export data; B trusts A with no mutual auth Blast radius across the agent mesh Multi-agent setups How do agents authenticate each other?
R-15 One bad step cascades through a pipeline A bad intermediate chunk corrupts every downstream transform in the job Wider outage or data corruption Chained workflows; stream transforms What stops a bad intermediate output from flowing downstream?
R-16 An autopilot agent keeps going after policy change A cron agent keeps writing after its policy was revoked Undead automation Scheduled / always-on agents Where is the kill switch, and who owns it?
R-17 The assistant’s own answers get re-learned as truth Yesterday’s wrong bot answer is re-indexed and cited as official next week Infection that is hard to find later Log→index loops Do we ever index model output without review?

5. Poisoned knowledge people will trust

ID In plain words Example What it costs the business Typical systems Ask your team
R-18 A low-privilege editor plants bad official content A junior wiki edit becomes the “official” answer everyone retrieves Wrong ops decisions at scale Wiki / CMS feeding search Who can publish into the index, and is there review?
R-19 Training or eval data is poisoned A planted row in the fine-tune or golden set skews later judgments Durable model bias or backdoors Fine-tunes; golden sets Where does train/eval data come from, and who signs changes?

6. Answers that leak, and people who over-trust

ID In plain words Example What it costs the business Typical systems Ask your team
R-20 The answer itself sends data out (link, image, HTML) The reply includes an external image URL that loads with a secret in the path Silent exfiltration via the browser or ticket UI Chat UIs; ticket bots Can answers include external links or images that load automatically?
R-21 Staff trust the confident tone and skip checks An operator hits Approve because the bot sounded sure Human-approved mistakes Ops copilots with a human in the loop Which actions stay hard even when the bot sounds sure?
R-22 Confident but wrong operational advice The bot invents a runbook step that takes production down Incidents from following the bot Support / runbook assistants Do risky answers require sources — or a Quality Audit?

7. Vendors and the stack you buy

ID In plain words Example What it costs the business Typical systems Ask your team
R-23 A vendor or connector in the stack is compromised A third-party embeddings or connector vendor is breached upstream Wide blast radius outside your app code Model host; embeddings; vector DB Do we have an inventory of AI components and owners?
R-24 The model provider keeps or trains on your prompts Hosted API retains chat content with no zero-retention option Data leaves your control with unclear retention Hosted LLM APIs What does the DPA say about retention, region, and training?

8. Cost and availability

ID In plain words Example What it costs the business Typical systems Ask your team
R-25 Usage or loops explode the bill or take the service down A tool loop runs overnight and burns the token budget Denial of wallet / outage Public bots; agents with loops What are the hard caps on tokens, steps, and spend?

9. Privacy and deletion

ID In plain words Example What it costs the business Typical systems Ask your team
R-26 You cannot fully delete someone’s data from AI stores CRM row deleted; vectors, caches, and traces still hold the person Failed deletion requests; regulatory exposure Any personal data in indexes/traces Can we delete chunks, vectors, caches, and traces end-to-end? (counsel owns legal framing)

10. Ordinary app security still matters

ID In plain words Example What it costs the business Typical systems Ask your team
R-27 Classic login/API holes sit next to the model An IDOR on the retrieval API skips the model’s ACL entirely AI controls become irrelevant The web/API wrapping the assistant Did we test authorization on tools and retrieval like any other API? See vs pentest.

AI Security Review · GitHub catalog · Security vs Quality Audit · OWASP mapping · All guides