Skip to content

Runbook 01 AI Threat Modeling Methodology

This runbook provides a repeatable methodology for performing an AI Threat Modeling Assessment.

Use it when reviewing:

  • Generative AI applications
  • LLM applications
  • RAG systems
  • AI agents
  • AI copilots
  • Enterprise AI platforms
  • AI-enabled cloud workflows
  • AI integrations with business systems
  • Model and data pipelines

The purpose is to help an AI Security Engineer move systematically from:

Business Use Case
Architecture
Assets
Trust Boundaries
Threats
Attack Paths
Risk
Security Controls

This runbook is designed to answer four questions:

What are we protecting?

How could the system be abused?

What would the business impact be?

Which controls should prevent, detect or contain the attack?

At the end of the assessment, you should have:

  • Defined scope

  • Business context

  • Validated architecture

  • Component inventory

  • Asset register

  • Data-flow register

  • Trust-boundary register

  • Threat-actor profiles

  • Attack-surface register

  • Threat scenarios

  • STRIDE analysis

  • OWASP GenAI mapping

  • MITRE ATLAS mapping where relevant

  • Abuse cases

  • Attack paths

  • Risk register

  • Threat-to-control matrix

  • Security requirements

  • Detection requirements

  • Residual-risk assessment

  • Executive summary

Use this runbook:

When a new AI system is moving toward production.

Before high-risk architecture decisions become difficult to change.

Especially when enterprise or sensitive data will be indexed.

Whenever AI moves from generating information to performing actions.

For example:

Email
Cloud APIs
Databases
Code Execution
Ticketing
Financial Systems

Example:

Read-Only Agent
Production Write Agent

Example:

Public Documentation
Confidential Enterprise Data

Use lessons from the incident to update the threat model.

An enterprise AI threat modeling session may involve:

  • AI Security Engineer

  • Security Architect

  • AI / ML Engineer

  • Application Engineer

  • Cloud / Platform Engineer

  • Application Security

  • Data Owner

  • SOC representative

  • Product Owner

  • Risk / Governance representative

Not every assessment requires every role.

The important principle is:

Include the people who understand the architecture, data, business impact and security controls.

Collect the following before beginning.

  • Business purpose

  • Primary users

  • Critical business functions

  • Availability requirements

  • Regulatory or contractual requirements

  • Architecture diagram

  • Component inventory

  • Network relationships

  • External providers

  • Model providers

  • Tool integrations

  • Human identities

  • Workload identities

  • Agent identities

  • Service accounts

  • API credentials

  • Data sources

  • Data classification

  • RAG sources

  • Model inputs

  • Model outputs

  • Retention

  • IAM policies

  • Agent permissions

  • Tool permissions

  • Database permissions

  • RAG permissions

  • Logging

  • Monitoring

  • Detection

  • Incident response

  • Kill-switch capabilities

Record:

## System Name
## Business Owner
## Technical Owner
## Security Owner
## Architecture Version
## Threat Model Version
## Assessment Date

Ask:

Why Does This AI System Exist?
Who Uses It?
What Business Process Does It Support?
What Happens If It Fails?
What Happens If It Produces Incorrect Results?

Document the answers clearly.

Create:

## In Scope
## Out of Scope

Include all components that materially influence AI risk.

Typical in-scope components:

  • Application

  • LLM

  • RAG

  • Vector database

  • Agent

  • Tools

  • Data sources

  • IAM

  • Cloud infrastructure

Example:

## Assumptions
- Enterprise SSO is enabled.
- MFA is enabled.
- Vector database is private.
- Agent uses dedicated workload identity.

Do not silently treat assumptions as confirmed facts.

Example:

## Unknowns
- Does RAG preserve source ACLs?
- Does the LLM provider retain prompts?
- Are agent tool calls centrally logged?

Unknowns become validation tasks.

Step 6 — Review the Architecture Diagram

Section titled “Step 6 — Review the Architecture Diagram”

Start with a high-level view.

Example:

User
AI Application
AI Orchestrator
├── LLM
├── RAG
│ └── Vector Database
└── AI Agent
└── Enterprise Tools

Do not begin threat analysis until the architecture is sufficiently understood.

Ask engineering:

Is This Diagram Accurate?
Are Any Components Missing?
Are Any External Services Missing?
Are All Identities Shown?
Are Privileged Tools Shown?
Are Production Connections Shown?

Incorrect architecture produces incorrect threat modeling.

Use:

ID Component Purpose Owner Identity Exposure
C-01 AI Application User interface App Team User Session Internal
C-02 LLM Model inference AI Team API Identity External/Internal
C-03 RAG Retrieval AI Team Workload ID Private
C-04 Agent Tool execution AI Team Agent ID Privileged

Look beyond the model.

Typical AI assets include:

User Identities
Agent Identities
Enterprise Data
RAG Documents
Vector Database
Embeddings
Models
System Configuration
Prompts
API Credentials
Agent Memory
Logs
Production Resources

Use:

ID Asset Owner Classification Criticality
A-01 Enterprise Documents Data Team Confidential High
A-02 Agent Identity AI Team Restricted Critical
A-03 Vector DB AI Team Confidential High

Ask:

Which assets would create the greatest business impact if compromised?

Examples:

Restricted Customer Data
Production Cloud
Administrative Agent Identity
Model Registry
Security Credentials

Document them separately.

Document every significant flow.

Example:

DF-01 User → AI Application
DF-02 Application → LLM
DF-03 RAG → Vector Database
DF-04 Agent → Cloud API

Use:

ID Source Destination Data Identity Classification

Pay special attention to:

Enterprise → External LLM
Enterprise → Embedding Provider
Agent → External API
AI → SaaS Tool

Ask:

  • What data crosses?

  • Where is it processed?

  • Is it retained?

  • Is the provider approved?

For sensitive information ask:

Where Is It Created?
Where Is It Processed?
Where Is It Stored?
Where Is It Logged?
Where Is It Retained?
Where Is It Deleted?

Typical AI trust boundaries include:

User → Application
Application → External LLM
Document Source → RAG
RAG → Vector Database
Retrieved Content → LLM
LLM → Agent
Agent → Tool
Tool → Production System

Step 15 — Build the Trust Boundary Register

Section titled “Step 15 — Build the Trust Boundary Register”

Use:

ID Source Destination Trust Change Primary Concern
TB-01 User App User → Enterprise Malicious input
TB-02 App External LLM Enterprise → Third Party Data exposure
TB-03 Agent Cloud API AI → Privileged System Unauthorized action

Highlight whenever authority increases.

Example:

Normal Employee
AI Agent
Privileged Cloud Identity

Ask:

Can a lower-privileged identity indirectly use higher agent privilege?

This may reveal a confused deputy risk.

Consider:

External Attacker
Compromised Employee
Malicious Insider
Compromised Administrator
Malicious Document Contributor
Supply-Chain Attacker
Compromised Workload
External Content Author

For each actor document:

## Existing Access
## Capability
## Objectives
## Reachable Attack Surfaces
## Target Assets

AI systems may process attacker-controlled:

  • Documents

  • Emails

  • Websites

  • Tickets

  • Code

  • API responses

An attacker may influence AI behavior without directly accessing the AI application.

Step 19 — Identify Direct Attack Surfaces

Section titled “Step 19 — Identify Direct Attack Surfaces”

Examples:

Chat Interface
API
Authentication
File Upload
Agent Interface
Administration Console

Step 20 — Identify Indirect Attack Surfaces

Section titled “Step 20 — Identify Indirect Attack Surfaces”

Examples:

RAG Documents
Email
External Websites
Tool Responses
Support Tickets
Training Data

Step 21 — Identify Infrastructure Attack Surfaces

Section titled “Step 21 — Identify Infrastructure Attack Surfaces”

Include:

  • IAM

  • CI/CD

  • Model registry

  • Container registry

  • Cloud APIs

  • Vector database

  • Kubernetes

  • Secrets

Step 22 — Build the Attack Surface Register

Section titled “Step 22 — Build the Attack Surface Register”

Use:

ID Surface Exposure Threat Actors Assets at Risk

Step 23 — Apply STRIDE to Important Components

Section titled “Step 23 — Apply STRIDE to Important Components”

Ask:

Can an attacker impersonate:

  • User

  • Agent

  • Service

  • Tool

Can they modify:

  • Prompts

  • RAG content

  • Metadata

  • Models

  • Configuration

Can actions occur without attribution?

Can confidential information be exposed?

Can AI resources, APIs or tools be exhausted?

Can lower-privileged users gain higher capability?

Step 24 — Apply STRIDE at Trust Boundaries

Section titled “Step 24 — Apply STRIDE at Trust Boundaries”

Example:

User
TB-01
AI Application

Ask all six STRIDE questions where relevant.

Do not force every category if it does not make sense.

Phase 9 — Apply AI-Specific Security Guidance

Section titled “Phase 9 — Apply AI-Specific Security Guidance”

Validate relevant current OWASP guidance during the assessment.

Look for threat patterns involving:

  • Prompt injection

  • Sensitive data disclosure

  • Data / model poisoning

  • Supply chain

  • Improper output handling

  • Excessive agency

  • Vector / embedding security

  • Resource consumption

Do not simply copy category names.

Map them to the actual architecture.

Avoid:

Prompt Injection — High

Prefer:

A malicious document contributor may insert adversarial instructions into an indexed knowledge source. If the content is retrieved and influences a privileged AI agent without independent authorization, the agent may perform an unauthorized enterprise action.

Ask:

What Is the Attacker Trying to Achieve?
How Would They Discover the AI System?
What Do They Need to Access?
What Could They Manipulate?
What Would They Do Next?

Use current MITRE ATLAS mappings where they improve understanding.

Typical areas may include:

  • Reconnaissance

  • Discovery

  • AI resource manipulation

  • Data manipulation

  • Credential abuse

  • Collection

  • Impact

Do not force ATLAS mappings into every finding.

A strong threat scenario contains:

Threat Actor
+
Attack Surface
+
Security Weakness
+
Target Asset
+
Impact

Use:

# T-XX — Threat Name
## Threat Actor
## Attack Surface
## Trust Boundary
## Target Asset
## Preconditions
## Threat Scenario
## Existing Controls
## Security Gap
## Business Impact

Remove threats that:

  • Do not apply

  • Require unrealistic assumptions

  • Have no meaningful impact

Threat modeling should prioritize quality over quantity.

If the architecture uses RAG, review the complete lifecycle.

Ask:

Where Does Knowledge Come From?
Who Can Modify It?
What Is Its Classification?
Can External Content Enter?

Verify:

  • Source identity

  • Provenance

  • ACL preservation

  • Logging

  • Integrity

Step 33 — Review Retrieval Authorization

Section titled “Step 33 — Review Retrieval Authorization”

The key question:

Can users only retrieve documents they are authorized to access?

Validate:

User Identity
Authorization
Allowed Data Scope
Vector Search

Consider:

  • Unauthorized retrieval

  • Cross-tenant access

  • Knowledge poisoning

  • Indirect prompt injection

  • Metadata tampering

  • Vector DB compromise

  • Stale data

  • Deletion failure

If agents are present, treat them as action systems.

Record:

Tool Capability Privilege Risk
Search Read Low Low
Ticket Write Medium Medium
Cloud Change Write High Critical

Ask:

Which Identity Does the Agent Use?
What Permissions Does It Have?
How Is It Authenticated?
How Is It Revoked?

Step 37 — Review User-Context Authorization

Section titled “Step 37 — Review User-Context Authorization”

Do not ask only:

Can Agent Perform Action?

Ask:

Can User Perform Action?
+
Can Agent Perform Action?
+
Is Target Allowed?

Validate:

  • Operation

  • Target

  • Environment

  • Scope

  • Destination

Ask:

  • Is memory persistent?

  • Is it isolated?

  • Can it be poisoned?

  • Does it contain confidential information?

Classify high-impact actions.

Examples:

Read Data → No Approval
Create Standard Ticket → Usually No Approval
Restart Production → Approval
Modify Network → Approval
Modify IAM → Consider Removing Capability

Ask:

How do we stop the agent immediately?

Required options may include:

  • Disable workload identity

  • Revoke tokens

  • Block tools

  • Stop runtime

  • Emergency deny policy

Examples:

Unauthorized RAG Retrieval
Knowledge Poisoning
Privileged Agent Abuse
Agent Credential Compromise
Sensitive Data Exfiltration

Use:

# AC-XX — Abuse Case
## Threat Actor
## Objective
## Starting Position
## Preconditions
## Abuse Scenario
## Target Assets
## Business Impact

Example:

Compromised Employee
AI Agent
Missing User Authorization
Privileged Tool
Production Resource

Record what must be true.

Example:

Employee identity compromised
Agent reachable
Agent has production permissions
User authorization missing

Example:

Compromised User
[MFA]
AI Agent
[User Authorization]
Tool
[Least Privilege]
Production

Use attack trees when several routes lead to the same asset.

Example:

Production Impact
OR
┌──────────────┼──────────────┐
▼ ▼ ▼
Agent Abuse Credential Theft RAG Attack

Consider:

  • Exposure

  • Required access

  • Attack complexity

  • Threat actor capability

  • Existing controls

Consider:

  • Asset criticality

  • Data sensitivity

  • Agent privilege

  • Business disruption

  • Regulatory impact

  • Safety where applicable

Include:

Autonomy
Agent Privilege
Blast Radius
Persistence
Number of Users
External Tools
Detectability

Use the organization’s risk methodology.

For simple assessments:

Likelihood × Impact = Risk

Classify control effectiveness:

  • None

  • Weak

  • Partial

  • Strong

Reassess the scenario after controls.

Record reasoning.

Examples:

  • Authentication

  • Authorization

  • Least privilege

  • Source governance

  • Data segmentation

  • Tool restrictions

  • Parameter validation

  • Approval

  • Rate limits

Examples:

  • Identity alerts

  • Retrieval monitoring

  • RAG source-change monitoring

  • Agent tool monitoring

  • Model registry alerts

  • Cloud audit alerts

Examples:

  • Disable user

  • Disable agent

  • Revoke token

  • Remove malicious RAG source

  • Rebuild index

  • Roll back model

  • Restore cloud configuration

Step 56 — Build the Threat-to-Control Matrix

Section titled “Step 56 — Build the Threat-to-Control Matrix”

Use:

Threat Prevent Detect Respond

Step 57 — Convert Controls Into Testable Requirements

Section titled “Step 57 — Convert Controls Into Testable Requirements”

Avoid:

Secure RAG.

Prefer:

RAG-SEC-001
The RAG retrieval service must enforce source-document
authorization using the initiating user's identity.

Avoid:

Secure the agent.

Prefer:

AGENT-SEC-001
Production AI agents must use dedicated least-privilege
workload identities.

Each requirement should have:

  • Owner

  • Priority

  • Target date

  • Validation method

Phase 19 — Define Detection Requirements

Section titled “Phase 19 — Define Detection Requirements”

Step 59 — Convert Threats Into Detection Use Cases

Section titled “Step 59 — Convert Threats Into Detection Use Cases”

Example:

Threat:

Compromised user collects sensitive RAG data.

Detection:

Alert when a user retrieves an abnormal volume of confidential content or accesses multiple restricted knowledge domains.

Another:

Threat:

Agent performs unexpected production operation.

Detection:

Alert on privileged AI-agent tool activity inconsistent with the initiating user’s role.

Step 60 — Define Containment for Critical Threats

Section titled “Step 60 — Define Containment for Critical Threats”

For each critical threat ask:

How Do We Stop It?
How Do We Revoke Access?
How Do We Preserve Evidence?
How Do We Restore the System?

Document required runbooks where needed.

Turn assumptions into checks.

Example:

Assumption:
Agent is read-only.
Validation:
Review actual IAM policy.

Example:

Requirement:

General employees cannot retrieve HR data.

Test:

General Employee → HR Document → DENIED

Expected:

Allowed Read → ALLOWED
Unauthorized Write → DENIED

In an authorized non-production environment:

Agent Enabled → Tool Works
Agent Disabled → Tool Denied

Use:

ID Evidence Supports Date
EV-01 Agent IAM Policy AGENT-SEC-001 YYYY-MM-DD
EV-02 RAG ACL Test RAG-SEC-001 YYYY-MM-DD

Examples of evidence:

  • IAM policies

  • Architecture diagrams

  • Access-test results

  • Logging screenshots

  • Configuration extracts

Include:

System Reviewed
Scope
Overall Risk
Critical Risks
High Risks
Positive Controls
Priority Recommendations
Residual Risk

Keep it concise and decision-focused.

Step 67 — Prepare the Technical Findings

Section titled “Step 67 — Prepare the Technical Findings”

Each high-risk finding should contain:

Threat
Attack Path
Asset
Evidence
Impact
Existing Controls
Control Gap
Recommendation
Risk

Prefer:

Priority 1 — Critical Architecture Issues
Priority 2 — High-Risk Controls
Priority 3 — Monitoring and Hardening
Priority 4 — Improvements

Avoid presenting 50 recommendations with equal priority.

Engineering confirms:

  • Architecture

  • Identities

  • Data flows

  • Permissions

  • Integrations

Security validates:

  • Threats

  • Attack paths

  • Risk

  • Controls

  • Requirements

Business owners validate:

  • Business impact

  • Remediation priority

  • Residual risk

  • Risk acceptance where required

Reassess when:

New LLM Added
New RAG Source Added
Sensitive Data Added
New Agent Tool Added
Agent Permissions Increased
Production Write Access Added
External Provider Changed
Architecture Changed
Security Incident Occurred

Record:

Threat Model Version:
Architecture Version:
Last Review:
Next Review:
Status:

Keep historical versions where required for:

  • Audit

  • Risk decisions

  • Architecture history

  • Incident investigation

  • Business purpose understood

  • Users identified

  • Criticality defined

  • In-scope documented

  • Out-of-scope documented

  • Assumptions documented

  • Unknowns documented

  • Architecture validated

  • Components inventoried

  • External services identified

  • Identities identified

  • Assets identified

  • Data classified

  • Crown jewels identified

  • Data flows mapped

  • External flows identified

  • Sensitive-data paths identified

  • Trust boundaries identified

  • Privilege boundaries identified

  • Third-party boundaries identified

  • External attackers considered

  • Compromised users considered

  • Insiders considered

  • Supply-chain attackers considered

  • Indirect content attackers considered

  • Public interfaces reviewed

  • Internal interfaces reviewed

  • RAG reviewed

  • Agents reviewed

  • Infrastructure reviewed

  • STRIDE applied

  • OWASP GenAI considered

  • MITRE ATLAS considered

  • Threat scenarios documented

  • Source permissions reviewed

  • Ingestion reviewed

  • Retrieval authorization reviewed

  • Poisoning reviewed

  • Indirect injection reviewed

  • Vector security reviewed

  • Agent identities reviewed

  • Tools inventoried

  • Permissions reviewed

  • User authorization reviewed

  • Parameters reviewed

  • Memory reviewed

  • Approval reviewed

  • Kill switch reviewed

  • Abuse cases documented

  • Attack paths documented

  • Preconditions documented

  • Control breakpoints identified

  • Likelihood evaluated

  • Impact evaluated

  • Blast radius evaluated

  • Inherent risk documented

  • Residual risk documented

  • Preventive controls mapped

  • Detective controls mapped

  • Response controls mapped

  • Security requirements created

  • Owners assigned

  • Validation methods defined

  • Logging requirements defined

  • Detection requirements defined

  • Containment procedures defined

  • Executive summary completed

  • Top risks prioritized

  • Recommendations prioritized

  • Review triggers defined

Decision Questions for AI Security Engineers

Section titled “Decision Questions for AI Security Engineers”

Whenever you feel stuck during an AI threat modeling engagement, return to these questions:

What Is the Business Trying to Do?
What Can the AI Access?
What Can the AI Change?
Who Can Influence the AI?
What Content Can Influence the AI?
Which Identities Does It Use?
Where Does Privilege Increase?
Which Assets Are Behind That Privilege?
How Could an Attacker Reach Them?
Where Can We Break the Path?
How Would We Detect It?
How Would We Stop It?

Do not start by listing every OWASP or ATLAS category.

Start with the architecture.

The critical risk may exist in:

  • IAM

  • RAG

  • Tools

  • Agent identity

  • Cloud permissions

  • CI/CD

Internal documents can be malicious, compromised or incorrect.

Model output should not automatically authorize actions.

Agent permissions should not replace user authorization.

Broad agent permissions dramatically increase blast radius.

AI systems still require:

  • Identity security

  • Cloud security

  • Network security

  • Application security

  • Supply-chain security

Threat modeling should produce operational requirements.

A documented control may not be an effective control.

Use this structure for a professional assessment:

AI-Threat-Model/
├── 01 Executive Summary.md
├── 02 Business Context and Scope.md
├── 03 Architecture.md
├── 04 Component Inventory.md
├── 05 Asset Register.md
├── 06 Data Flows.md
├── 07 Trust Boundaries.md
├── 08 Threat Actors.md
├── 09 Attack Surface.md
├── 10 STRIDE Analysis.md
├── 11 OWASP GenAI Mapping.md
├── 12 MITRE ATLAS Mapping.md
├── 13 Threat Register.md
├── 14 Abuse Cases.md
├── 15 Attack Paths.md
├── 16 Risk Register.md
├── 17 Threat-to-Control Matrix.md
├── 18 Security Requirements.md
├── 19 Detection Requirements.md
├── 20 Validation Register.md
├── 21 Evidence Register.md
├── 22 Residual Risk.md
└── 23 Review History.md

The assessment is complete when you can clearly demonstrate:

Business Context
Architecture
Asset
Threat
Attack Path
Risk
Security Control
Requirement
Validation

For every major risk, you should be able to answer:

What asset is at risk?

Who could attack it?

How could they reach it?

Which controls prevent or limit the attack?

How would we detect the activity?

How would we contain it?

What residual risk remains?

Asset
Production Cloud
Threat
Privileged AI Agent Abuse
Attack Path
Compromised User
→ AI Agent
→ Missing User Authorization
→ Cloud Tool
→ Production
Risk
Critical
Controls
Least Privilege
User Authorization
Parameter Validation
Human Approval
Requirement
AGENT-SEC-001
Validation
Unauthorized User Production Action → DENIED

This traceability is one of the strongest indicators of a mature threat model.

The complete methodology is:

1. Define Scope
2. Understand Business Context
3. Validate Architecture
4. Identify Assets
5. Map Data Flows
6. Identify Trust Boundaries
7. Identify Threat Actors
8. Map Attack Surfaces
9. Apply STRIDE
10. Apply OWASP GenAI Guidance
11. Use MITRE ATLAS
12. Threat Model RAG
13. Threat Model Agents
14. Build Abuse Cases
15. Build Attack Paths
16. Assess Risk
17. Map Security Controls
18. Define Requirements
19. Define Detection and Response
20. Validate Controls
21. Document Evidence
22. Report and Review
23. Maintain the Threat Model

The core principle is:

Threat modeling is not about predicting every possible AI attack. It is about understanding the system well enough to identify realistic attack paths to important assets and designing controls that make those paths difficult, detectable and containable.

➡️ Runbook 02 — AI Risk Assessment Methodology

You now have a repeatable process for identifying and analyzing AI threats.

The next runbook focuses specifically on how to assess and prioritize the resulting risks.

You will create a repeatable methodology for evaluating:

  • Threat likelihood

  • Business impact

  • Asset criticality

  • Exposure

  • Attack complexity

  • Required attacker access

  • AI agent privilege

  • Autonomy

  • Blast radius

  • Persistence

  • Detectability

  • Existing control effectiveness

  • Inherent risk

  • Residual risk

  • Risk treatment

  • Risk acceptance

The goal is to move from:

“I can identify and document AI threats.”

to:

“I can consistently determine which AI risks require immediate remediation, which can be reduced through controls and which may be accepted by the appropriate risk owner.”

➡️ Next: Runbook 02 — AI Risk Assessment Methodology