Runbook 01 AI Threat Modeling Methodology
Runbook Purpose
Section titled “Runbook Purpose”This runbook provides a repeatable methodology for performing an AI Threat Modeling Assessment.
Use it when reviewing:
- Generative AI applications
- LLM applications
- RAG systems
- AI agents
- AI copilots
- Enterprise AI platforms
- AI-enabled cloud workflows
- AI integrations with business systems
- Model and data pipelines
The purpose is to help an AI Security Engineer move systematically from:
Business Use Case ↓Architecture ↓Assets ↓Trust Boundaries ↓Threats ↓Attack Paths ↓Risk ↓Security ControlsThis runbook is designed to answer four questions:
What are we protecting?
How could the system be abused?
What would the business impact be?
Which controls should prevent, detect or contain the attack?
Runbook Outcome
Section titled “Runbook Outcome”At the end of the assessment, you should have:
-
Defined scope
-
Business context
-
Validated architecture
-
Component inventory
-
Asset register
-
Data-flow register
-
Trust-boundary register
-
Threat-actor profiles
-
Attack-surface register
-
Threat scenarios
-
STRIDE analysis
-
OWASP GenAI mapping
-
MITRE ATLAS mapping where relevant
-
Abuse cases
-
Attack paths
-
Risk register
-
Threat-to-control matrix
-
Security requirements
-
Detection requirements
-
Residual-risk assessment
-
Executive summary
When to Use This Runbook
Section titled “When to Use This Runbook”Use this runbook:
Before Production Deployment
Section titled “Before Production Deployment”When a new AI system is moving toward production.
During Architecture Design
Section titled “During Architecture Design”Before high-risk architecture decisions become difficult to change.
When Adding RAG
Section titled “When Adding RAG”Especially when enterprise or sensitive data will be indexed.
When Adding AI Agents
Section titled “When Adding AI Agents”Whenever AI moves from generating information to performing actions.
When Adding New Tools
Section titled “When Adding New Tools”For example:
EmailCloud APIsDatabasesCode ExecutionTicketingFinancial SystemsWhen AI Permissions Change
Section titled “When AI Permissions Change”Example:
Read-Only Agent ↓Production Write AgentWhen Data Sensitivity Changes
Section titled “When Data Sensitivity Changes”Example:
Public Documentation ↓Confidential Enterprise DataAfter Significant Security Incidents
Section titled “After Significant Security Incidents”Use lessons from the incident to update the threat model.
Recommended Participants
Section titled “Recommended Participants”An enterprise AI threat modeling session may involve:
-
AI Security Engineer
-
Security Architect
-
AI / ML Engineer
-
Application Engineer
-
Cloud / Platform Engineer
-
Application Security
-
Data Owner
-
SOC representative
-
Product Owner
-
Risk / Governance representative
Not every assessment requires every role.
The important principle is:
Include the people who understand the architecture, data, business impact and security controls.
Inputs Required
Section titled “Inputs Required”Collect the following before beginning.
Business Information
Section titled “Business Information”-
Business purpose
-
Primary users
-
Critical business functions
-
Availability requirements
-
Regulatory or contractual requirements
Architecture Information
Section titled “Architecture Information”-
Architecture diagram
-
Component inventory
-
Network relationships
-
External providers
-
Model providers
-
Tool integrations
Identity Information
Section titled “Identity Information”-
Human identities
-
Workload identities
-
Agent identities
-
Service accounts
-
API credentials
Data Information
Section titled “Data Information”-
Data sources
-
Data classification
-
RAG sources
-
Model inputs
-
Model outputs
-
Retention
Permission Information
Section titled “Permission Information”-
IAM policies
-
Agent permissions
-
Tool permissions
-
Database permissions
-
RAG permissions
Operational Information
Section titled “Operational Information”-
Logging
-
Monitoring
-
Detection
-
Incident response
-
Kill-switch capabilities
Phase 1 — Define the Assessment
Section titled “Phase 1 — Define the Assessment”Step 1 — Identify the System
Section titled “Step 1 — Identify the System”Record:
## System Name
## Business Owner
## Technical Owner
## Security Owner
## Architecture Version
## Threat Model Version
## Assessment DateStep 2 — Define the Business Purpose
Section titled “Step 2 — Define the Business Purpose”Ask:
Why Does This AI System Exist?
Who Uses It?
What Business Process Does It Support?
What Happens If It Fails?
What Happens If It Produces Incorrect Results?Document the answers clearly.
Step 3 — Define Scope
Section titled “Step 3 — Define Scope”Create:
## In Scope
## Out of ScopeInclude all components that materially influence AI risk.
Typical in-scope components:
-
Application
-
LLM
-
RAG
-
Vector database
-
Agent
-
Tools
-
Data sources
-
IAM
-
Cloud infrastructure
Step 4 — Document Assumptions
Section titled “Step 4 — Document Assumptions”Example:
## Assumptions
- Enterprise SSO is enabled.- MFA is enabled.- Vector database is private.- Agent uses dedicated workload identity.Do not silently treat assumptions as confirmed facts.
Step 5 — Document Unknowns
Section titled “Step 5 — Document Unknowns”Example:
## Unknowns
- Does RAG preserve source ACLs?- Does the LLM provider retain prompts?- Are agent tool calls centrally logged?Unknowns become validation tasks.
Phase 2 — Understand the Architecture
Section titled “Phase 2 — Understand the Architecture”Step 6 — Review the Architecture Diagram
Section titled “Step 6 — Review the Architecture Diagram”Start with a high-level view.
Example:
User ↓AI Application ↓AI Orchestrator ├── LLM ├── RAG │ └── Vector Database └── AI Agent └── Enterprise ToolsDo not begin threat analysis until the architecture is sufficiently understood.
Step 7 — Validate the Architecture
Section titled “Step 7 — Validate the Architecture”Ask engineering:
Is This Diagram Accurate?
Are Any Components Missing?
Are Any External Services Missing?
Are All Identities Shown?
Are Privileged Tools Shown?
Are Production Connections Shown?Incorrect architecture produces incorrect threat modeling.
Step 8 — Build the Component Inventory
Section titled “Step 8 — Build the Component Inventory”Use:
| ID | Component | Purpose | Owner | Identity | Exposure |
|---|---|---|---|---|---|
| C-01 | AI Application | User interface | App Team | User Session | Internal |
| C-02 | LLM | Model inference | AI Team | API Identity | External/Internal |
| C-03 | RAG | Retrieval | AI Team | Workload ID | Private |
| C-04 | Agent | Tool execution | AI Team | Agent ID | Privileged |
Phase 3 — Identify Assets
Section titled “Phase 3 — Identify Assets”Step 9 — Build an Asset Register
Section titled “Step 9 — Build an Asset Register”Look beyond the model.
Typical AI assets include:
User IdentitiesAgent IdentitiesEnterprise DataRAG DocumentsVector DatabaseEmbeddingsModelsSystem ConfigurationPromptsAPI CredentialsAgent MemoryLogsProduction ResourcesUse:
| ID | Asset | Owner | Classification | Criticality |
|---|---|---|---|---|
| A-01 | Enterprise Documents | Data Team | Confidential | High |
| A-02 | Agent Identity | AI Team | Restricted | Critical |
| A-03 | Vector DB | AI Team | Confidential | High |
Step 10 — Identify Crown Jewels
Section titled “Step 10 — Identify Crown Jewels”Ask:
Which assets would create the greatest business impact if compromised?
Examples:
Restricted Customer DataProduction CloudAdministrative Agent IdentityModel RegistrySecurity CredentialsDocument them separately.
Phase 4 — Map Data
Section titled “Phase 4 — Map Data”Step 11 — Map Data Flows
Section titled “Step 11 — Map Data Flows”Document every significant flow.
Example:
DF-01 User → AI Application
DF-02 Application → LLM
DF-03 RAG → Vector Database
DF-04 Agent → Cloud APIUse:
| ID | Source | Destination | Data | Identity | Classification |
|---|
Step 12 — Identify External Data Flows
Section titled “Step 12 — Identify External Data Flows”Pay special attention to:
Enterprise → External LLM
Enterprise → Embedding Provider
Agent → External API
AI → SaaS ToolAsk:
-
What data crosses?
-
Where is it processed?
-
Is it retained?
-
Is the provider approved?
Step 13 — Map the Data Lifecycle
Section titled “Step 13 — Map the Data Lifecycle”For sensitive information ask:
Where Is It Created?
Where Is It Processed?
Where Is It Stored?
Where Is It Logged?
Where Is It Retained?
Where Is It Deleted?Phase 5 — Identify Trust Boundaries
Section titled “Phase 5 — Identify Trust Boundaries”Step 14 — Find Where Trust Changes
Section titled “Step 14 — Find Where Trust Changes”Typical AI trust boundaries include:
User → Application
Application → External LLM
Document Source → RAG
RAG → Vector Database
Retrieved Content → LLM
LLM → Agent
Agent → Tool
Tool → Production SystemStep 15 — Build the Trust Boundary Register
Section titled “Step 15 — Build the Trust Boundary Register”Use:
| ID | Source | Destination | Trust Change | Primary Concern |
|---|---|---|---|---|
| TB-01 | User | App | User → Enterprise | Malicious input |
| TB-02 | App | External LLM | Enterprise → Third Party | Data exposure |
| TB-03 | Agent | Cloud API | AI → Privileged System | Unauthorized action |
Step 16 — Identify Privilege Boundaries
Section titled “Step 16 — Identify Privilege Boundaries”Highlight whenever authority increases.
Example:
Normal Employee ↓AI Agent ↓Privileged Cloud IdentityAsk:
Can a lower-privileged identity indirectly use higher agent privilege?
This may reveal a confused deputy risk.
Phase 6 — Identify Threat Actors
Section titled “Phase 6 — Identify Threat Actors”Step 17 — Build Threat Actor Profiles
Section titled “Step 17 — Build Threat Actor Profiles”Consider:
External Attacker
Compromised Employee
Malicious Insider
Compromised Administrator
Malicious Document Contributor
Supply-Chain Attacker
Compromised Workload
External Content AuthorFor each actor document:
## Existing Access
## Capability
## Objectives
## Reachable Attack Surfaces
## Target AssetsStep 18 — Include Indirect Attackers
Section titled “Step 18 — Include Indirect Attackers”AI systems may process attacker-controlled:
-
Documents
-
Emails
-
Websites
-
Tickets
-
Code
-
API responses
An attacker may influence AI behavior without directly accessing the AI application.
Phase 7 — Map the Attack Surface
Section titled “Phase 7 — Map the Attack Surface”Step 19 — Identify Direct Attack Surfaces
Section titled “Step 19 — Identify Direct Attack Surfaces”Examples:
Chat InterfaceAPIAuthenticationFile UploadAgent InterfaceAdministration ConsoleStep 20 — Identify Indirect Attack Surfaces
Section titled “Step 20 — Identify Indirect Attack Surfaces”Examples:
RAG DocumentsEmailExternal WebsitesTool ResponsesSupport TicketsTraining DataStep 21 — Identify Infrastructure Attack Surfaces
Section titled “Step 21 — Identify Infrastructure Attack Surfaces”Include:
-
IAM
-
CI/CD
-
Model registry
-
Container registry
-
Cloud APIs
-
Vector database
-
Kubernetes
-
Secrets
Step 22 — Build the Attack Surface Register
Section titled “Step 22 — Build the Attack Surface Register”Use:
| ID | Surface | Exposure | Threat Actors | Assets at Risk |
|---|
Phase 8 — Apply STRIDE
Section titled “Phase 8 — Apply STRIDE”Step 23 — Apply STRIDE to Important Components
Section titled “Step 23 — Apply STRIDE to Important Components”Ask:
Spoofing
Section titled “Spoofing”Can an attacker impersonate:
-
User
-
Agent
-
Service
-
Tool
Tampering
Section titled “Tampering”Can they modify:
-
Prompts
-
RAG content
-
Metadata
-
Models
-
Configuration
Repudiation
Section titled “Repudiation”Can actions occur without attribution?
Information Disclosure
Section titled “Information Disclosure”Can confidential information be exposed?
Denial of Service
Section titled “Denial of Service”Can AI resources, APIs or tools be exhausted?
Elevation of Privilege
Section titled “Elevation of Privilege”Can lower-privileged users gain higher capability?
Step 24 — Apply STRIDE at Trust Boundaries
Section titled “Step 24 — Apply STRIDE at Trust Boundaries”Example:
User ↓TB-01 ↓AI ApplicationAsk all six STRIDE questions where relevant.
Do not force every category if it does not make sense.
Phase 9 — Apply AI-Specific Security Guidance
Section titled “Phase 9 — Apply AI-Specific Security Guidance”Step 25 — Review OWASP GenAI Risks
Section titled “Step 25 — Review OWASP GenAI Risks”Validate relevant current OWASP guidance during the assessment.
Look for threat patterns involving:
-
Prompt injection
-
Sensitive data disclosure
-
Data / model poisoning
-
Supply chain
-
Improper output handling
-
Excessive agency
-
Vector / embedding security
-
Resource consumption
Do not simply copy category names.
Map them to the actual architecture.
Step 26 — Write Real Scenarios
Section titled “Step 26 — Write Real Scenarios”Avoid:
Prompt Injection — High
Prefer:
A malicious document contributor may insert adversarial instructions into an indexed knowledge source. If the content is retrieved and influences a privileged AI agent without independent authorization, the agent may perform an unauthorized enterprise action.
Phase 10 — Use MITRE ATLAS
Section titled “Phase 10 — Use MITRE ATLAS”Step 27 — Shift to Adversary Behavior
Section titled “Step 27 — Shift to Adversary Behavior”Ask:
What Is the Attacker Trying to Achieve?
How Would They Discover the AI System?
What Do They Need to Access?
What Could They Manipulate?
What Would They Do Next?Step 28 — Map Relevant ATLAS Behaviors
Section titled “Step 28 — Map Relevant ATLAS Behaviors”Use current MITRE ATLAS mappings where they improve understanding.
Typical areas may include:
-
Reconnaissance
-
Discovery
-
AI resource manipulation
-
Data manipulation
-
Credential abuse
-
Collection
-
Impact
Do not force ATLAS mappings into every finding.
Phase 11 — Build Threat Scenarios
Section titled “Phase 11 — Build Threat Scenarios”Step 29 — Create the Threat Register
Section titled “Step 29 — Create the Threat Register”A strong threat scenario contains:
Threat Actor +Attack Surface +Security Weakness +Target Asset +ImpactUse:
# T-XX — Threat Name
## Threat Actor
## Attack Surface
## Trust Boundary
## Target Asset
## Preconditions
## Threat Scenario
## Existing Controls
## Security Gap
## Business ImpactStep 30 — Focus on Realistic Threats
Section titled “Step 30 — Focus on Realistic Threats”Remove threats that:
-
Do not apply
-
Require unrealistic assumptions
-
Have no meaningful impact
Threat modeling should prioritize quality over quantity.
Phase 12 — Threat Model RAG
Section titled “Phase 12 — Threat Model RAG”If the architecture uses RAG, review the complete lifecycle.
Step 31 — Review RAG Sources
Section titled “Step 31 — Review RAG Sources”Ask:
Where Does Knowledge Come From?
Who Can Modify It?
What Is Its Classification?
Can External Content Enter?Step 32 — Review Ingestion
Section titled “Step 32 — Review Ingestion”Verify:
-
Source identity
-
Provenance
-
ACL preservation
-
Logging
-
Integrity
Step 33 — Review Retrieval Authorization
Section titled “Step 33 — Review Retrieval Authorization”The key question:
Can users only retrieve documents they are authorized to access?
Validate:
User Identity ↓Authorization ↓Allowed Data Scope ↓Vector SearchStep 34 — Review RAG-Specific Threats
Section titled “Step 34 — Review RAG-Specific Threats”Consider:
-
Unauthorized retrieval
-
Cross-tenant access
-
Knowledge poisoning
-
Indirect prompt injection
-
Metadata tampering
-
Vector DB compromise
-
Stale data
-
Deletion failure
Phase 13 — Threat Model AI Agents
Section titled “Phase 13 — Threat Model AI Agents”If agents are present, treat them as action systems.
Step 35 — Inventory Agent Tools
Section titled “Step 35 — Inventory Agent Tools”Record:
| Tool | Capability | Privilege | Risk |
|---|---|---|---|
| Search | Read | Low | Low |
| Ticket | Write | Medium | Medium |
| Cloud Change | Write | High | Critical |
Step 36 — Review Agent Identity
Section titled “Step 36 — Review Agent Identity”Ask:
Which Identity Does the Agent Use?
What Permissions Does It Have?
How Is It Authenticated?
How Is It Revoked?Step 37 — Review User-Context Authorization
Section titled “Step 37 — Review User-Context Authorization”Do not ask only:
Can Agent Perform Action?Ask:
Can User Perform Action?+Can Agent Perform Action?+Is Target Allowed?Step 38 — Review Tool Parameters
Section titled “Step 38 — Review Tool Parameters”Validate:
-
Operation
-
Target
-
Environment
-
Scope
-
Destination
Step 39 — Review Agent Memory
Section titled “Step 39 — Review Agent Memory”Ask:
-
Is memory persistent?
-
Is it isolated?
-
Can it be poisoned?
-
Does it contain confidential information?
Step 40 — Review Approval
Section titled “Step 40 — Review Approval”Classify high-impact actions.
Examples:
Read Data → No Approval
Create Standard Ticket → Usually No Approval
Restart Production → Approval
Modify Network → Approval
Modify IAM → Consider Removing CapabilityStep 41 — Review Agent Containment
Section titled “Step 41 — Review Agent Containment”Ask:
How do we stop the agent immediately?
Required options may include:
-
Disable workload identity
-
Revoke tokens
-
Block tools
-
Stop runtime
-
Emergency deny policy
Phase 14 — Build Abuse Cases
Section titled “Phase 14 — Build Abuse Cases”Step 42 — Select High-Value Abuse Cases
Section titled “Step 42 — Select High-Value Abuse Cases”Examples:
Unauthorized RAG Retrieval
Knowledge Poisoning
Privileged Agent Abuse
Agent Credential Compromise
Sensitive Data ExfiltrationUse:
# AC-XX — Abuse Case
## Threat Actor
## Objective
## Starting Position
## Preconditions
## Abuse Scenario
## Target Assets
## Business ImpactPhase 15 — Build Attack Paths
Section titled “Phase 15 — Build Attack Paths”Step 43 — Connect the Threats
Section titled “Step 43 — Connect the Threats”Example:
Compromised Employee ↓AI Agent ↓Missing User Authorization ↓Privileged Tool ↓Production ResourceStep 44 — Include Preconditions
Section titled “Step 44 — Include Preconditions”Record what must be true.
Example:
Employee identity compromised
Agent reachable
Agent has production permissions
User authorization missingStep 45 — Identify Control Breakpoints
Section titled “Step 45 — Identify Control Breakpoints”Example:
Compromised User ↓[MFA] ↓AI Agent ↓[User Authorization] ↓Tool ↓[Least Privilege] ↓ProductionStep 46 — Consider Alternative Paths
Section titled “Step 46 — Consider Alternative Paths”Use attack trees when several routes lead to the same asset.
Example:
Production Impact OR ┌──────────────┼──────────────┐ ▼ ▼ ▼ Agent Abuse Credential Theft RAG AttackPhase 16 — Assess Risk
Section titled “Phase 16 — Assess Risk”Step 47 — Assess Likelihood
Section titled “Step 47 — Assess Likelihood”Consider:
-
Exposure
-
Required access
-
Attack complexity
-
Threat actor capability
-
Existing controls
Step 48 — Assess Impact
Section titled “Step 48 — Assess Impact”Consider:
-
Asset criticality
-
Data sensitivity
-
Agent privilege
-
Business disruption
-
Regulatory impact
-
Safety where applicable
Step 49 — Assess AI-Specific Factors
Section titled “Step 49 — Assess AI-Specific Factors”Include:
Autonomy
Agent Privilege
Blast Radius
Persistence
Number of Users
External Tools
DetectabilityStep 50 — Assign Inherent Risk
Section titled “Step 50 — Assign Inherent Risk”Use the organization’s risk methodology.
For simple assessments:
Likelihood × Impact = RiskStep 51 — Assess Existing Controls
Section titled “Step 51 — Assess Existing Controls”Classify control effectiveness:
-
None
-
Weak
-
Partial
-
Strong
Step 52 — Determine Residual Risk
Section titled “Step 52 — Determine Residual Risk”Reassess the scenario after controls.
Record reasoning.
Phase 17 — Map Security Controls
Section titled “Phase 17 — Map Security Controls”Step 53 — Map Preventive Controls
Section titled “Step 53 — Map Preventive Controls”Examples:
-
Authentication
-
Authorization
-
Least privilege
-
Source governance
-
Data segmentation
-
Tool restrictions
-
Parameter validation
-
Approval
-
Rate limits
Step 54 — Map Detective Controls
Section titled “Step 54 — Map Detective Controls”Examples:
-
Identity alerts
-
Retrieval monitoring
-
RAG source-change monitoring
-
Agent tool monitoring
-
Model registry alerts
-
Cloud audit alerts
Step 55 — Map Responsive Controls
Section titled “Step 55 — Map Responsive Controls”Examples:
-
Disable user
-
Disable agent
-
Revoke token
-
Remove malicious RAG source
-
Rebuild index
-
Roll back model
-
Restore cloud configuration
Step 56 — Build the Threat-to-Control Matrix
Section titled “Step 56 — Build the Threat-to-Control Matrix”Use:
| Threat | Prevent | Detect | Respond |
|---|
Phase 18 — Create Security Requirements
Section titled “Phase 18 — Create Security Requirements”Step 57 — Convert Controls Into Testable Requirements
Section titled “Step 57 — Convert Controls Into Testable Requirements”Avoid:
Secure RAG.
Prefer:
RAG-SEC-001
The RAG retrieval service must enforce source-documentauthorization using the initiating user's identity.Avoid:
Secure the agent.
Prefer:
AGENT-SEC-001
Production AI agents must use dedicated least-privilegeworkload identities.Step 58 — Assign Owners
Section titled “Step 58 — Assign Owners”Each requirement should have:
-
Owner
-
Priority
-
Target date
-
Validation method
Phase 19 — Define Detection Requirements
Section titled “Phase 19 — Define Detection Requirements”Step 59 — Convert Threats Into Detection Use Cases
Section titled “Step 59 — Convert Threats Into Detection Use Cases”Example:
Threat:
Compromised user collects sensitive RAG data.
Detection:
Alert when a user retrieves an abnormal volume of confidential content or accesses multiple restricted knowledge domains.
Another:
Threat:
Agent performs unexpected production operation.
Detection:
Alert on privileged AI-agent tool activity inconsistent with the initiating user’s role.
Phase 20 — Define Response Requirements
Section titled “Phase 20 — Define Response Requirements”Step 60 — Define Containment for Critical Threats
Section titled “Step 60 — Define Containment for Critical Threats”For each critical threat ask:
How Do We Stop It?
How Do We Revoke Access?
How Do We Preserve Evidence?
How Do We Restore the System?Document required runbooks where needed.
Phase 21 — Validate the Threat Model
Section titled “Phase 21 — Validate the Threat Model”Step 61 — Validate Assumptions
Section titled “Step 61 — Validate Assumptions”Turn assumptions into checks.
Example:
Assumption:Agent is read-only.
Validation:Review actual IAM policy.Step 62 — Validate Security Controls
Section titled “Step 62 — Validate Security Controls”Example:
Requirement:
General employees cannot retrieve HR data.
Test:
General Employee → HR Document → DENIEDStep 63 — Validate Agent Permissions
Section titled “Step 63 — Validate Agent Permissions”Expected:
Allowed Read → ALLOWED
Unauthorized Write → DENIEDStep 64 — Validate Kill Switches
Section titled “Step 64 — Validate Kill Switches”In an authorized non-production environment:
Agent Enabled → Tool Works
Agent Disabled → Tool DeniedPhase 22 — Document Evidence
Section titled “Phase 22 — Document Evidence”Step 65 — Build an Evidence Register
Section titled “Step 65 — Build an Evidence Register”Use:
| ID | Evidence | Supports | Date |
|---|---|---|---|
| EV-01 | Agent IAM Policy | AGENT-SEC-001 | YYYY-MM-DD |
| EV-02 | RAG ACL Test | RAG-SEC-001 | YYYY-MM-DD |
Examples of evidence:
-
IAM policies
-
Architecture diagrams
-
Access-test results
-
Logging screenshots
-
Configuration extracts
Phase 23 — Produce the Final Report
Section titled “Phase 23 — Produce the Final Report”Step 66 — Prepare the Executive Summary
Section titled “Step 66 — Prepare the Executive Summary”Include:
System Reviewed
Scope
Overall Risk
Critical Risks
High Risks
Positive Controls
Priority Recommendations
Residual RiskKeep it concise and decision-focused.
Step 67 — Prepare the Technical Findings
Section titled “Step 67 — Prepare the Technical Findings”Each high-risk finding should contain:
Threat
Attack Path
Asset
Evidence
Impact
Existing Controls
Control Gap
Recommendation
RiskStep 68 — Prioritize Recommendations
Section titled “Step 68 — Prioritize Recommendations”Prefer:
Priority 1 — Critical Architecture Issues
Priority 2 — High-Risk Controls
Priority 3 — Monitoring and Hardening
Priority 4 — ImprovementsAvoid presenting 50 recommendations with equal priority.
Phase 24 — Conduct the Review
Section titled “Phase 24 — Conduct the Review”Step 69 — Architecture Review
Section titled “Step 69 — Architecture Review”Engineering confirms:
-
Architecture
-
Identities
-
Data flows
-
Permissions
-
Integrations
Step 70 — Security Review
Section titled “Step 70 — Security Review”Security validates:
-
Threats
-
Attack paths
-
Risk
-
Controls
-
Requirements
Step 71 — Business / Risk Review
Section titled “Step 71 — Business / Risk Review”Business owners validate:
-
Business impact
-
Remediation priority
-
Residual risk
-
Risk acceptance where required
Phase 25 — Maintain the Threat Model
Section titled “Phase 25 — Maintain the Threat Model”Step 72 — Define Review Triggers
Section titled “Step 72 — Define Review Triggers”Reassess when:
New LLM Added
New RAG Source Added
Sensitive Data Added
New Agent Tool Added
Agent Permissions Increased
Production Write Access Added
External Provider Changed
Architecture Changed
Security Incident OccurredStep 73 — Version the Threat Model
Section titled “Step 73 — Version the Threat Model”Record:
Threat Model Version:
Architecture Version:
Last Review:
Next Review:
Status:Step 74 — Archive Superseded Versions
Section titled “Step 74 — Archive Superseded Versions”Keep historical versions where required for:
-
Audit
-
Risk decisions
-
Architecture history
-
Incident investigation
Quick AI Threat Modeling Checklist
Section titled “Quick AI Threat Modeling Checklist”Business
Section titled “Business”-
Business purpose understood
-
Users identified
-
Criticality defined
-
In-scope documented
-
Out-of-scope documented
-
Assumptions documented
-
Unknowns documented
Architecture
Section titled “Architecture”-
Architecture validated
-
Components inventoried
-
External services identified
-
Identities identified
Assets
Section titled “Assets”-
Assets identified
-
Data classified
-
Crown jewels identified
Data Flows
Section titled “Data Flows”-
Data flows mapped
-
External flows identified
-
Sensitive-data paths identified
-
Trust boundaries identified
-
Privilege boundaries identified
-
Third-party boundaries identified
Threat Actors
Section titled “Threat Actors”-
External attackers considered
-
Compromised users considered
-
Insiders considered
-
Supply-chain attackers considered
-
Indirect content attackers considered
Attack Surface
Section titled “Attack Surface”-
Public interfaces reviewed
-
Internal interfaces reviewed
-
RAG reviewed
-
Agents reviewed
-
Infrastructure reviewed
Threat Analysis
Section titled “Threat Analysis”-
STRIDE applied
-
OWASP GenAI considered
-
MITRE ATLAS considered
-
Threat scenarios documented
-
Source permissions reviewed
-
Ingestion reviewed
-
Retrieval authorization reviewed
-
Poisoning reviewed
-
Indirect injection reviewed
-
Vector security reviewed
Agents
Section titled “Agents”-
Agent identities reviewed
-
Tools inventoried
-
Permissions reviewed
-
User authorization reviewed
-
Parameters reviewed
-
Memory reviewed
-
Approval reviewed
-
Kill switch reviewed
Attack Paths
Section titled “Attack Paths”-
Abuse cases documented
-
Attack paths documented
-
Preconditions documented
-
Control breakpoints identified
-
Likelihood evaluated
-
Impact evaluated
-
Blast radius evaluated
-
Inherent risk documented
-
Residual risk documented
Controls
Section titled “Controls”-
Preventive controls mapped
-
Detective controls mapped
-
Response controls mapped
Requirements
Section titled “Requirements”-
Security requirements created
-
Owners assigned
-
Validation methods defined
Operations
Section titled “Operations”-
Logging requirements defined
-
Detection requirements defined
-
Containment procedures defined
Reporting
Section titled “Reporting”-
Executive summary completed
-
Top risks prioritized
-
Recommendations prioritized
-
Review triggers defined
Decision Questions for AI Security Engineers
Section titled “Decision Questions for AI Security Engineers”Whenever you feel stuck during an AI threat modeling engagement, return to these questions:
What Is the Business Trying to Do? ↓What Can the AI Access? ↓What Can the AI Change? ↓Who Can Influence the AI? ↓What Content Can Influence the AI? ↓Which Identities Does It Use? ↓Where Does Privilege Increase? ↓Which Assets Are Behind That Privilege? ↓How Could an Attacker Reach Them? ↓Where Can We Break the Path? ↓How Would We Detect It? ↓How Would We Stop It?Common Failure Patterns
Section titled “Common Failure Patterns”Starting With Framework Checklists
Section titled “Starting With Framework Checklists”Do not start by listing every OWASP or ATLAS category.
Start with the architecture.
Focusing Only on the LLM
Section titled “Focusing Only on the LLM”The critical risk may exist in:
-
IAM
-
RAG
-
Tools
-
Agent identity
-
Cloud permissions
-
CI/CD
Assuming Internal Content Is Trusted
Section titled “Assuming Internal Content Is Trusted”Internal documents can be malicious, compromised or incorrect.
Trusting Model Output
Section titled “Trusting Model Output”Model output should not automatically authorize actions.
Ignoring User Authorization
Section titled “Ignoring User Authorization”Agent permissions should not replace user authorization.
Overprivileged Agents
Section titled “Overprivileged Agents”Broad agent permissions dramatically increase blast radius.
Ignoring Traditional Cybersecurity
Section titled “Ignoring Traditional Cybersecurity”AI systems still require:
-
Identity security
-
Cloud security
-
Network security
-
Application security
-
Supply-chain security
No Detection or Response
Section titled “No Detection or Response”Threat modeling should produce operational requirements.
No Validation
Section titled “No Validation”A documented control may not be an effective control.
Threat Modeling Deliverable Template
Section titled “Threat Modeling Deliverable Template”Use this structure for a professional assessment:
AI-Threat-Model/│├── 01 Executive Summary.md├── 02 Business Context and Scope.md├── 03 Architecture.md├── 04 Component Inventory.md├── 05 Asset Register.md├── 06 Data Flows.md├── 07 Trust Boundaries.md├── 08 Threat Actors.md├── 09 Attack Surface.md├── 10 STRIDE Analysis.md├── 11 OWASP GenAI Mapping.md├── 12 MITRE ATLAS Mapping.md├── 13 Threat Register.md├── 14 Abuse Cases.md├── 15 Attack Paths.md├── 16 Risk Register.md├── 17 Threat-to-Control Matrix.md├── 18 Security Requirements.md├── 19 Detection Requirements.md├── 20 Validation Register.md├── 21 Evidence Register.md├── 22 Residual Risk.md└── 23 Review History.mdRunbook Completion Criteria
Section titled “Runbook Completion Criteria”The assessment is complete when you can clearly demonstrate:
Business Context ↓Architecture ↓Asset ↓Threat ↓Attack Path ↓Risk ↓Security Control ↓Requirement ↓ValidationFor every major risk, you should be able to answer:
What asset is at risk?
Who could attack it?
How could they reach it?
Which controls prevent or limit the attack?
How would we detect the activity?
How would we contain it?
What residual risk remains?
Example Traceability
Section titled “Example Traceability”AssetProduction Cloud
↓
ThreatPrivileged AI Agent Abuse
↓
Attack PathCompromised User→ AI Agent→ Missing User Authorization→ Cloud Tool→ Production
↓
RiskCritical
↓
ControlsLeast PrivilegeUser AuthorizationParameter ValidationHuman Approval
↓
RequirementAGENT-SEC-001
↓
ValidationUnauthorized User Production Action → DENIEDThis traceability is one of the strongest indicators of a mature threat model.
Runbook Summary
Section titled “Runbook Summary”The complete methodology is:
1. Define Scope ↓2. Understand Business Context ↓3. Validate Architecture ↓4. Identify Assets ↓5. Map Data Flows ↓6. Identify Trust Boundaries ↓7. Identify Threat Actors ↓8. Map Attack Surfaces ↓9. Apply STRIDE ↓10. Apply OWASP GenAI Guidance ↓11. Use MITRE ATLAS ↓12. Threat Model RAG ↓13. Threat Model Agents ↓14. Build Abuse Cases ↓15. Build Attack Paths ↓16. Assess Risk ↓17. Map Security Controls ↓18. Define Requirements ↓19. Define Detection and Response ↓20. Validate Controls ↓21. Document Evidence ↓22. Report and Review ↓23. Maintain the Threat ModelThe core principle is:
Threat modeling is not about predicting every possible AI attack. It is about understanding the system well enough to identify realistic attack paths to important assets and designing controls that make those paths difficult, detectable and containable.
What’s Next?
Section titled “What’s Next?”➡️ Runbook 02 — AI Risk Assessment Methodology
You now have a repeatable process for identifying and analyzing AI threats.
The next runbook focuses specifically on how to assess and prioritize the resulting risks.
You will create a repeatable methodology for evaluating:
-
Threat likelihood
-
Business impact
-
Asset criticality
-
Exposure
-
Attack complexity
-
Required attacker access
-
AI agent privilege
-
Autonomy
-
Blast radius
-
Persistence
-
Detectability
-
Existing control effectiveness
-
Inherent risk
-
Residual risk
-
Risk treatment
-
Risk acceptance
The goal is to move from:
“I can identify and document AI threats.”
to:
“I can consistently determine which AI risks require immediate remediation, which can be reduced through controls and which may be accepted by the appropriate risk owner.”
➡️ Next: Runbook 02 — AI Risk Assessment Methodology