# AI security risks — what can go wrong in plain language

An AI assistant that reads your documents or takes actions can leak data, show the wrong person’s information, or do something nobody approved — even when it looks internal. This catalog helps you pick what matters before a review. Not certification.

Product: [AI Security Review](/ai-security). Markdown source for tools: [/guides/ai-security-risks.md](/guides/ai-security-risks.md). Open catalog on GitHub: [nareyko/ai-security-risks](https://github.com/nareyko/ai-security-risks) (MIT; links [WardBench](https://github.com/WardBench/wardbench) open proof).

If the assistant can **read untrusted text**, **see sensitive data**, and **send something outward** (link, email, API, ticket), treat that as urgent — [lethal trifecta](/guides/lethal-trifecta).

## 1. Untrusted content steering the assistant

| ID | In plain words | Example | What it costs the business | Typical systems | Ask your team |
| --- | --- | --- | --- | --- | --- |
| R-01 | Instructions hide inside a document, ticket, or log | A support ticket quietly tells the bot to ignore policy and list customer emails | Policy bypass, silent data leak, wrong automated action | Doc search; support bots; log copilots | Can text we did not type change what the assistant does? |
| R-02 | Uploaded files hide instructions the UI never shows | A PDF in the knowledge base carries white-on-white instructions the uploader never saw | Same as above via PDF/HTML tricks | Upload, OCR, email attachments | Do we strip or quarantine odd file formats before indexing? |
| R-03 | Someone plants content that ranks for many questions | One planted page answers dozens of common FAQ queries ahead of real docs | One bad page reaches lots of users | Search over a large knowledge base | Can one low-trust document dominate answers for common queries? |

## 2. Data ending up with the wrong people

| ID | In plain words | Example | What it costs the business | Typical systems | Ask your team |
| --- | --- | --- | --- | --- | --- |
| R-04 | Secrets or personal data appear in answers or logs | Chat pastes an API key that lived in an indexed config file | Credential theft; privacy incidents; contract breach | Anything indexed from configs, logs, CRM | Would a fake secret we plant in the index show up in chat? |
| R-05 | User A sees customer/team B’s information | Tenant A’s user gets chunks that belong only to tenant B | Cross-tenant incident; lost trust | Multi-tenant SaaS; shared company search | Are permissions checked before text goes into the model? |
| R-06 | A copy of the search index is as sensitive as the files | Someone with vector-store access recovers readable text from embeddings and backups | Breach of “just embeddings” still exposes content | Vector databases and their backups | Who can download the index, and is it encrypted like source data? |
| R-07 | Internal system instructions or tool list leak | A user coaxing the chat reveals the system prompt and tool names | Easier attacks; exposed architecture | Any chat product | Can a user extract the system prompt or tool names? |
| R-08 | Indexes, backups, or debug traces are too widely shared | Broad Datadog access lets staff read full production prompt traces | Same leak without using the chat at all | Ops access to stores and traces | Who can read production indexes and prompt logs? |

## 3. The assistant does things, not only answers

| ID | In plain words | Example | What it costs the business | Typical systems | Ask your team |
| --- | --- | --- | --- | --- | --- |
| R-09 | It changes systems or spends money without a real approval | The agent refunds a payment or changes ACL with no human click | Outages; fraud; irreversible actions | Agents with tools | Which actions need a human click every time? |
| R-10 | It runs with stronger rights than the employee using it | A normal user’s chat runs tools under an admin service account | One user effectively becomes admin | Shared service account agents | Does the tool use the employee’s identity, or a god-token? |
| R-11 | It can run code or shell outside a tight box | A coding agent executes shell commands on a host that can reach production | Host compromise; lateral movement | Coding agents; run-this tools | Where does generated code execute, and what can it reach? |
| R-12 | A plugged-in tool’s description steers unsafe behavior | An MCP connector description nudges the model toward a dangerous tool call | Supply-chain style abuse via connectors | MCP / plugin ecosystems | Do we inventory and review every connector description? |

## 4. Memory, long-running agents, and chains of agents

| ID | In plain words | Example | What it costs the business | Typical systems | Ask your team |
| --- | --- | --- | --- | --- | --- |
| R-13 | Saved memory or a multi-step plan is hijacked | Poisoned memory makes the next sessions chase an attacker’s goal | Lasting wrong behavior across sessions | Stateful assistants | Who can write to agent memory, and how do we wipe it? |
| R-14 | One agent persuades another without strong checks | Agent A tells agent B to export data; B trusts A with no mutual auth | Blast radius across the agent mesh | Multi-agent setups | How do agents authenticate each other? |
| R-15 | One bad step cascades through a pipeline | A bad intermediate chunk corrupts every downstream transform in the job | Wider outage or data corruption | Chained workflows; stream transforms | What stops a bad intermediate output from flowing downstream? |
| R-16 | An autopilot agent keeps going after policy change | A cron agent keeps writing after its policy was revoked | Undead automation | Scheduled / always-on agents | Where is the kill switch, and who owns it? |
| R-17 | The assistant’s own answers get re-learned as truth | Yesterday’s wrong bot answer is re-indexed and cited as official next week | Infection that is hard to find later | Log→index loops | Do we ever index model output without review? |

## 5. Poisoned knowledge people will trust

| ID | In plain words | Example | What it costs the business | Typical systems | Ask your team |
| --- | --- | --- | --- | --- | --- |
| R-18 | A low-privilege editor plants bad official content | A junior wiki edit becomes the “official” answer everyone retrieves | Wrong ops decisions at scale | Wiki / CMS feeding search | Who can publish into the index, and is there review? |
| R-19 | Training or eval data is poisoned | A planted row in the fine-tune or golden set skews later judgments | Durable model bias or backdoors | Fine-tunes; golden sets | Where does train/eval data come from, and who signs changes? |

## 6. Answers that leak, and people who over-trust

| ID | In plain words | Example | What it costs the business | Typical systems | Ask your team |
| --- | --- | --- | --- | --- | --- |
| R-20 | The answer itself sends data out (link, image, HTML) | The reply includes an external image URL that loads with a secret in the path | Silent exfiltration via the browser or ticket UI | Chat UIs; ticket bots | Can answers include external links or images that load automatically? |
| R-21 | Staff trust the confident tone and skip checks | An operator hits Approve because the bot sounded sure | Human-approved mistakes | Ops copilots with a human in the loop | Which actions stay hard even when the bot sounds sure? |
| R-22 | Confident but wrong operational advice | The bot invents a runbook step that takes production down | Incidents from following the bot | Support / runbook assistants | Do risky answers require sources — or a [Quality Audit](/llm-quality-audit)? |

## 7. Vendors and the stack you buy

| ID | In plain words | Example | What it costs the business | Typical systems | Ask your team |
| --- | --- | --- | --- | --- | --- |
| R-23 | A vendor or connector in the stack is compromised | A third-party embeddings or connector vendor is breached upstream | Wide blast radius outside your app code | Model host; embeddings; vector DB | Do we have an inventory of AI components and owners? |
| R-24 | The model provider keeps or trains on your prompts | Hosted API retains chat content with no zero-retention option | Data leaves your control with unclear retention | Hosted LLM APIs | What does the DPA say about retention, region, and training? |

## 8. Cost and availability

| ID | In plain words | Example | What it costs the business | Typical systems | Ask your team |
| --- | --- | --- | --- | --- | --- |
| R-25 | Usage or loops explode the bill or take the service down | A tool loop runs overnight and burns the token budget | Denial of wallet / outage | Public bots; agents with loops | What are the hard caps on tokens, steps, and spend? |

## 9. Privacy and deletion

| ID | In plain words | Example | What it costs the business | Typical systems | Ask your team |
| --- | --- | --- | --- | --- | --- |
| R-26 | You cannot fully delete someone’s data from AI stores | CRM row deleted; vectors, caches, and traces still hold the person | Failed deletion requests; regulatory exposure | Any personal data in indexes/traces | Can we delete chunks, vectors, caches, and traces end-to-end? (counsel owns legal framing) |

## 10. Ordinary app security still matters

| ID | In plain words | Example | What it costs the business | Typical systems | Ask your team |
| --- | --- | --- | --- | --- | --- |
| R-27 | Classic login/API holes sit next to the model | An IDOR on the retrieval API skips the model’s ACL entirely | AI controls become irrelevant | The web/API wrapping the assistant | Did we test authorization on tools and retrieval like any other API? See [vs pentest](/ai-security/vs-pentest). |

[AI Security Review](/ai-security#scope) · [GitHub catalog](https://github.com/nareyko/ai-security-risks) · [Security vs Quality Audit](/ai-security/vs-quality-audit) · [OWASP mapping](/guides/owasp-llm-top-10-2026) · [All guides](/guides)
