AI security risks — what can go wrong in plain language
An AI assistant that reads your documents or takes actions can leak data, show the wrong person’s information, or do something nobody approved — even when it looks internal. This catalog helps you pick what matters before a review. Not certification.
Product: AI Security Review . Markdown source for tools: /guides/ai-security-risks.md . Open catalog on GitHub: nareyko/ai-security-risks (MIT; links WardBench open proof).
If the assistant can read untrusted text , see sensitive data , and send something outward (link, email, API, ticket), treat that as urgent — lethal trifecta .
1. Untrusted content steering the assistant
ID
In plain words
Example
What it costs the business
Typical systems
Ask your team
R-01
Instructions hide inside a document, ticket, or log
A support ticket quietly tells the bot to ignore policy and list customer emails
Policy bypass, silent data leak, wrong automated action
Doc search; support bots; log copilots
Can text we did not type change what the assistant does?
R-02
Uploaded files hide instructions the UI never shows
A PDF in the knowledge base carries white-on-white instructions the uploader never saw
Same as above via PDF/HTML tricks
Upload, OCR, email attachments
Do we strip or quarantine odd file formats before indexing?
R-03
Someone plants content that ranks for many questions
One planted page answers dozens of common FAQ queries ahead of real docs
One bad page reaches lots of users
Search over a large knowledge base
Can one low-trust document dominate answers for common queries?
2. Data ending up with the wrong people
ID
In plain words
Example
What it costs the business
Typical systems
Ask your team
R-04
Secrets or personal data appear in answers or logs
Chat pastes an API key that lived in an indexed config file
Credential theft; privacy incidents; contract breach
Anything indexed from configs, logs, CRM
Would a fake secret we plant in the index show up in chat?
R-05
User A sees customer/team B’s information
Tenant A’s user gets chunks that belong only to tenant B
Cross-tenant incident; lost trust
Multi-tenant SaaS; shared company search
Are permissions checked before text goes into the model?
R-06
A copy of the search index is as sensitive as the files
Someone with vector-store access recovers readable text from embeddings and backups
Breach of “just embeddings” still exposes content
Vector databases and their backups
Who can download the index, and is it encrypted like source data?
R-07
Internal system instructions or tool list leak
A user coaxing the chat reveals the system prompt and tool names
Easier attacks; exposed architecture
Any chat product
Can a user extract the system prompt or tool names?
R-08
Indexes, backups, or debug traces are too widely shared
Broad Datadog access lets staff read full production prompt traces
Same leak without using the chat at all
Ops access to stores and traces
Who can read production indexes and prompt logs?
3. The assistant does things, not only answers
ID
In plain words
Example
What it costs the business
Typical systems
Ask your team
R-09
It changes systems or spends money without a real approval
The agent refunds a payment or changes ACL with no human click
Outages; fraud; irreversible actions
Agents with tools
Which actions need a human click every time?
R-10
It runs with stronger rights than the employee using it
A normal user’s chat runs tools under an admin service account
One user effectively becomes admin
Shared service account agents
Does the tool use the employee’s identity, or a god-token?
R-11
It can run code or shell outside a tight box
A coding agent executes shell commands on a host that can reach production
Host compromise; lateral movement
Coding agents; run-this tools
Where does generated code execute, and what can it reach?
R-12
A plugged-in tool’s description steers unsafe behavior
An MCP connector description nudges the model toward a dangerous tool call
Supply-chain style abuse via connectors
MCP / plugin ecosystems
Do we inventory and review every connector description?
4. Memory, long-running agents, and chains of agents
ID
In plain words
Example
What it costs the business
Typical systems
Ask your team
R-13
Saved memory or a multi-step plan is hijacked
Poisoned memory makes the next sessions chase an attacker’s goal
Lasting wrong behavior across sessions
Stateful assistants
Who can write to agent memory, and how do we wipe it?
R-14
One agent persuades another without strong checks
Agent A tells agent B to export data; B trusts A with no mutual auth
Blast radius across the agent mesh
Multi-agent setups
How do agents authenticate each other?
R-15
One bad step cascades through a pipeline
A bad intermediate chunk corrupts every downstream transform in the job
Wider outage or data corruption
Chained workflows; stream transforms
What stops a bad intermediate output from flowing downstream?
R-16
An autopilot agent keeps going after policy change
A cron agent keeps writing after its policy was revoked
Undead automation
Scheduled / always-on agents
Where is the kill switch, and who owns it?
R-17
The assistant’s own answers get re-learned as truth
Yesterday’s wrong bot answer is re-indexed and cited as official next week
Infection that is hard to find later
Log→index loops
Do we ever index model output without review?
5. Poisoned knowledge people will trust
ID
In plain words
Example
What it costs the business
Typical systems
Ask your team
R-18
A low-privilege editor plants bad official content
A junior wiki edit becomes the “official” answer everyone retrieves
Wrong ops decisions at scale
Wiki / CMS feeding search
Who can publish into the index, and is there review?
R-19
Training or eval data is poisoned
A planted row in the fine-tune or golden set skews later judgments
Durable model bias or backdoors
Fine-tunes; golden sets
Where does train/eval data come from, and who signs changes?
6. Answers that leak, and people who over-trust
ID
In plain words
Example
What it costs the business
Typical systems
Ask your team
R-20
The answer itself sends data out (link, image, HTML)
The reply includes an external image URL that loads with a secret in the path
Silent exfiltration via the browser or ticket UI
Chat UIs; ticket bots
Can answers include external links or images that load automatically?
R-21
Staff trust the confident tone and skip checks
An operator hits Approve because the bot sounded sure
Human-approved mistakes
Ops copilots with a human in the loop
Which actions stay hard even when the bot sounds sure?
R-22
Confident but wrong operational advice
The bot invents a runbook step that takes production down
Incidents from following the bot
Support / runbook assistants
Do risky answers require sources — or a Quality Audit ?
7. Vendors and the stack you buy
ID
In plain words
Example
What it costs the business
Typical systems
Ask your team
R-23
A vendor or connector in the stack is compromised
A third-party embeddings or connector vendor is breached upstream
Wide blast radius outside your app code
Model host; embeddings; vector DB
Do we have an inventory of AI components and owners?
R-24
The model provider keeps or trains on your prompts
Hosted API retains chat content with no zero-retention option
Data leaves your control with unclear retention
Hosted LLM APIs
What does the DPA say about retention, region, and training?
8. Cost and availability
ID
In plain words
Example
What it costs the business
Typical systems
Ask your team
R-25
Usage or loops explode the bill or take the service down
A tool loop runs overnight and burns the token budget
Denial of wallet / outage
Public bots; agents with loops
What are the hard caps on tokens, steps, and spend?
9. Privacy and deletion
ID
In plain words
Example
What it costs the business
Typical systems
Ask your team
R-26
You cannot fully delete someone’s data from AI stores
CRM row deleted; vectors, caches, and traces still hold the person
Failed deletion requests; regulatory exposure
Any personal data in indexes/traces
Can we delete chunks, vectors, caches, and traces end-to-end? (counsel owns legal framing)
10. Ordinary app security still matters
ID
In plain words
Example
What it costs the business
Typical systems
Ask your team
R-27
Classic login/API holes sit next to the model
An IDOR on the retrieval API skips the model’s ACL entirely
AI controls become irrelevant
The web/API wrapping the assistant
Did we test authorization on tools and retrieval like any other API? See vs pentest .
AI Security Review · GitHub catalog · Security vs Quality Audit · OWASP mapping · All guides
Need a written scope, fixed-hour sprint, or founder review?
Send your system context or technical question via our form for a free written reply within 48 hours , book a 24h PoC Spike (from €2,400 · 50% Risk-Shared) , a 40h Prototype Sprint (from €3,900) , or a Direct 1-on-1 Founder Consultation (€250/h, 100% credited in 30 days) .
We use essential cookies to run the site. Analytics (Google and Bing) load only if you accept.
Reject analytics
Accept analytics