Lab 01 Build an AI Threat Model

Mission Information

In this lab, you will build your first complete AI Threat Model from the ground up.

You will work through a realistic enterprise AI scenario and apply the methodology learned throughout the AI Threat Modeling module.

You will identify:

Business context
Architecture components
Assets
Sensitive data
Data flows
Trust boundaries
Threat actors
Attack surfaces
STRIDE threats
OWASP GenAI risks
MITRE ATLAS adversary behaviors
Abuse cases
Attack paths
Risk ratings
Preventive controls
Detective controls
Responsive controls
Security requirements

The goal is not simply to produce a list of AI threats.

The goal is to build a structured threat model that explains:

What can go wrong, how it could happen, what business impact could occur, and what controls should reduce the risk.

Lab Difficulty

Beginner to Intermediate

Estimated Time

90–120 minutes

Lab Type

Architecture Review + Threat Modeling

Prerequisites

Before starting this lab, you should understand:

AI system architecture
LLM fundamentals
RAG architecture
AI agents
Assets and data classification
Trust boundaries
Threat actors
STRIDE
OWASP GenAI security concepts
MITRE ATLAS concepts
AI attack paths
Risk assessment
Threat-to-control mapping

What You Will Build

By the end of this lab, you will create:

AI-Threat-Model/
│
├── 01 Business Context.md
├── 02 Architecture.md
├── 03 Asset Register.md
├── 04 Data Flow Register.md
├── 05 Trust Boundary Register.md
├── 06 Threat Actor Register.md
├── 07 Attack Surface Register.md
├── 08 Threat Register.md
├── 09 Attack Paths.md
├── 10 Risk Register.md
├── 11 Threat-to-Control Matrix.md
├── 12 Security Requirements.md
└── 13 Executive Summary.md

You may keep these as separate Obsidian notes or combine them into one lab document.

Scenario

You are working as an AI Security Engineer for an enterprise organization.

The company is developing an internal AI assistant called:

Enterprise Knowledge Assistant

Employees will use the assistant to search internal company information using natural language.

The system uses:

Enterprise SSO
Web application
AI orchestration service
External LLM provider
RAG
Vector database
Internal document repositories
AI agent
IT ticketing system

The AI assistant can:

Answer employee questions using enterprise documents.
Retrieve information from approved knowledge repositories.
Generate summaries.
Create IT support tickets through an AI agent.

The organization wants a security review before production deployment.

Your mission is to threat model the system.

Target Architecture

Use the following architecture for the lab.

                        Employee
                           │
                           │ SSO + MFA
                           ▼
                    Web Application
                           │
                           ▼
                    AI Orchestrator
                           │
           ┌───────────────┼────────────────┐
           │               │                │
           ▼               ▼                ▼
     External LLM         RAG            AI Agent
                           │                │
                           ▼                ▼
                    Vector Database    Ticketing API
                           │
                           ▼
                  Enterprise Documents

Enterprise Document Sources

The RAG system indexes:

General company policies
IT procedures
Security documentation
HR policies
Engineering documentation

Some documents are accessible to all employees.

Step 1 — Understand the Business Context

Start by documenting why the system exists.

Create:

01 Business Context.md

Use:

# Business Context

## System Name

Enterprise Knowledge Assistant

## Business Purpose

Provide employees with an AI-powered interface for retrieving approved enterprise knowledge and creating IT support tickets.

## Primary Users

- Employees
- IT Support
- Security Team

## Business Criticality

Medium to High

## Sensitive Functions

- Enterprise document retrieval
- Confidential knowledge access
- IT ticket creation

## External Dependencies

- External LLM provider
- Enterprise identity provider
- Ticketing platform

Student Task

Answer:

What business problem does the AI assistant solve?
Which users interact with it?
Which business processes depend on it?
What could happen if it exposes confidential information?
What could happen if the ticketing agent is abused?

Document your answers.

Step 2 — Define Scope

Document what you are reviewing.

In Scope

Web Application
AI Orchestrator
External LLM
RAG Service
Vector Database
Enterprise Knowledge Sources
AI Agent
Ticketing API
Relevant Identities

Out of Scope

For this lab:

Corporate Endpoint Security
Physical Security
Identity Provider Internal Architecture
Ticketing Platform Internal Infrastructure

Student Task

Create:

## Scope

### In Scope

### Out of Scope

### Assumptions

### Unknowns

Step 3 — Document Assumptions

Use the following assumptions:

Employees authenticate using SSO.

MFA is enabled.

The vector database is private.

The AI agent uses a dedicated service identity.

Only enterprise employees can access the assistant.

The external LLM is accessed through an authenticated API.

Do not assume:

RAG authorization is correct.
Agent permissions follow least privilege.
All RAG documents are trustworthy.
All AI activity is logged.

These must be investigated as part of the threat model.

Step 4 — Build the Component Inventory

Create:

02 Architecture.md

Start with:

ID	Component	Purpose	Exposure	Identity
C-01	Web Application	Employee interface	Internal	Employee Session
C-02	AI Orchestrator	Coordinates AI workflow	Private	Workload Identity
C-03	External LLM	Generates responses	External	API Credential
C-04	RAG Service	Retrieves knowledge	Private	Service Identity
C-05	Vector Database	Stores embeddings	Private	Service Identity
C-06	AI Agent	Creates tickets	Private	Agent Identity
C-07	Ticketing API	Creates support tickets	Enterprise SaaS	Agent Credential

Student Task

For each component answer:

What does it do?
Who owns it?
What data does it process?
Is it externally accessible?
What identity does it use?

Step 5 — Identify Assets

Create:

03 Asset Register.md

Identify everything that requires protection.

Start with:

ID	Asset	Type	Classification	Criticality
A-01	Employee Identity	Identity	Internal	High
A-02	Enterprise Documents	Data	Confidential	High
A-03	HR Documents	Data	Confidential	High
A-04	Security Documentation	Data	Confidential	High
A-05	Vector Database	Data Store	Confidential	High
A-06	LLM API Credential	Credential	Restricted	High
A-07	AI Agent Identity	Identity	Restricted	High
A-08	AI Logs	Security Data	Confidential	High
A-09	Ticketing Platform	Business System	Internal	Medium

Student Task

Identify at least 10 assets.

For each asset determine:

Owner
Classification
Criticality
Confidentiality requirement
Integrity requirement
Availability requirement

Step 6 — Identify Crown Jewels

From your asset register, identify the most important assets.

Possible examples:

CJ-01 Confidential HR Documentation

CJ-02 Security Architecture Documentation

CJ-03 AI Agent Identity

CJ-04 Enterprise LLM Credentials

Ask:

What happens if this asset is fully compromised?

Step 7 — Map Data Flows

Create:

04 Data Flow Register.md

Start with:

ID	Source	Destination	Data	Identity	Classification
DF-01	Employee	Web App	Prompt	Employee	Internal
DF-02	Web App	AI Orchestrator	Prompt	User Session	Internal
DF-03	AI Orchestrator	External LLM	Prompt + Context	API Credential	Confidential
DF-04	AI Orchestrator	RAG	Search Request	Service Identity	Internal
DF-05	RAG	Vector DB	Vector Query	Service Identity	Confidential
DF-06	Vector DB	RAG	Retrieved Content	Service Identity	Confidential
DF-07	AI Agent	Ticketing API	Ticket Details	Agent Identity	Internal

Student Task

Document at least 8 important data flows.

For each flow ask:

What data moves?
Who initiates the flow?
Which identity is used?
Is the data sensitive?
Does it leave the organization?
Is it logged?

Step 8 — Identify Trust Boundaries

Create:

05 Trust Boundary Register.md

Start with:

ID	Source	Destination	Trust Change	Concern
TB-01	Employee	Web App	User → Application	Untrusted Input
TB-02	AI Orchestrator	External LLM	Enterprise → Third Party	Data Exposure
TB-03	RAG	Vector DB	Service → Sensitive Data	Authorization
TB-04	Documents	RAG	Content → AI Knowledge	Poisoning
TB-05	AI Agent	Ticketing API	AI → Enterprise Action	Unauthorized Action

Student Task

Identify at least 5 trust boundaries.

For each one answer:

What crosses this boundary?

Which identity crosses it?

Does privilege change?

Does sensitive data cross?

What control protects it?

What happens if the control fails?

Step 9 — Identify Threat Actors

Create:

06 Threat Actor Register.md

Consider:

TA-01 External Attacker

TA-02 Compromised Employee

TA-03 Malicious Insider

TA-04 Malicious Document Contributor

TA-05 Supply-Chain Attacker

TA-06 Compromised AI Workload

Create:

ID	Threat Actor	Existing Access	Objective	Likely Targets
TA-01	External Attacker	None	Gain access	Identity/API
TA-02	Compromised Employee	Employee Account	Collect data	RAG
TA-03	Malicious Insider	Internal Access	Manipulate/Steal	Documents
TA-04	Malicious Contributor	Document Write	Poison AI	RAG
TA-05	Supply-Chain Attacker	Dependency	Compromise Platform	Application

Step 10 — Map the Attack Surface

Create:

07 Attack Surface Register.md

Identify attack surfaces such as:

Login interface
Chat interface
LLM API
RAG query interface
RAG ingestion
Vector database
AI agent interface
Ticketing API
Document repositories
Software dependencies

Example:

ID	Component	Attack Surface	Exposure	Assets at Risk
AS-01	Web App	Chat Interface	Internal	AI Service
AS-02	RAG	Retrieval	Internal	Documents
AS-03	RAG	Ingestion	Internal	Knowledge Integrity
AS-04	AI Agent	Tool Interface	Internal	Ticketing
AS-05	LLM	API	External Boundary	Enterprise Data

Student Task

Identify at least 8 attack surfaces.

Also classify each as:

Public
Internal
Privileged
Indirect

Step 11 — Apply STRIDE

Create:

08 Threat Register.md

Apply STRIDE to important components and data flows.

Spoofing

Consider:

Employee account takeover
Agent identity theft
Service identity impersonation

Tampering

Consider:

RAG poisoning
Metadata modification
Agent configuration changes

Repudiation

Consider:

Missing AI action attribution
Missing retrieval logging
Missing ticket creation audit trail

Information Disclosure

Consider:

Unauthorized RAG access
External LLM exposure
Cross-user leakage

Denial of Service

Consider:

AI request flooding
Vector query exhaustion
LLM cost abuse

Elevation of Privilege

Consider:

Employee influencing privileged agent
Agent using excessive ticketing permissions

Step 12 — Create Initial Threat Scenarios

Create at least 10 threats.

Example:

## T-01 — Unauthorized HR Document Retrieval

### STRIDE

Information Disclosure

### Threat Actor

Compromised Employee

### Target

RAG Retrieval Service

### Target Asset

Confidential HR Documents

### Threat Scenario

A compromised employee uses the AI assistant to retrieve HR documents outside the employee's normal authorization because the RAG service does not enforce source-document permissions.

Step 13 — Apply OWASP GenAI Guidance

Review each threat against relevant current OWASP GenAI guidance.

Look for areas involving:

Prompt injection
Sensitive information disclosure
Data poisoning
Supply chain
Improper output handling
Excessive agency
Vector/embedding security
Resource consumption

Do not simply write an OWASP category.

Explain why it applies.

Step 14 — Consider MITRE ATLAS

Now look at the attack from the adversary perspective.

For each high-priority scenario ask:

What does the adversary want?

What must they discover?

How do they gain influence?

What AI component do they target?

What happens next?

Map relevant current ATLAS tactics and techniques where useful.

You do not need to memorize technique IDs.

Validate them against the current official ATLAS knowledge base during real assessments.

Step 15 — Build Abuse Cases

Create at least 3 abuse cases.

Abuse Case 1 — Unauthorized RAG Retrieval

Normal behavior:

Employee
 ↓
AI Assistant
 ↓
Authorized Documents

Abuse:

Compromised Employee
       ↓
AI Assistant
       ↓
Weak RAG Authorization
       ↓
Restricted HR Documents

Abuse Case 2 — Knowledge Poisoning

Malicious Contributor
       ↓
Modified Document
       ↓
RAG
       ↓
Manipulated AI Response

Abuse Case 3 — AI Agent Abuse

Compromised Employee
       ↓
AI Agent
       ↓
Unauthorized Ticket Action

Step 16 — Build Attack Path 1

Create:

09 Attack Paths.md

AP-01 — Unauthorized RAG Retrieval

Compromised Employee
       ↓
Enterprise AI Assistant
       ↓
RAG Query
       ↓
Authentication Succeeds
       ↓
Document Authorization Missing
       ↓
Restricted HR Document
       ↓
LLM
       ↓
Sensitive Information Disclosure

Student Task

Identify:

Threat actor
Preconditions
Entry point
Trust boundaries
Security weakness
Target asset
Final impact

Step 17 — Build Attack Path 2

AP-02 — RAG Poisoning

Malicious Contributor
       ↓
Enterprise Document Repository
       ↓
Modified Document
       ↓
RAG Ingestion
       ↓
Vector Database
       ↓
Employee Query
       ↓
Poisoned Context
       ↓
Manipulated AI Answer

Ask:

Where could this attack path be broken?

Step 18 — Build Attack Path 3

AP-03 — Indirect Prompt Injection Through RAG

Malicious Contributor
       ↓
Document
       ↓
RAG
       ↓
Retrieved Context
       ↓
LLM
       ↓
AI Agent
       ↓
Ticketing Tool
       ↓
Unauthorized Action

Potential controls:

Source governance
Treat RAG content as untrusted
Agent tool restrictions
User authorization
Parameter validation

Step 19 — Build Attack Path 4

AP-04 — Agent Identity Compromise

Attacker
   ↓
Agent Credential
   ↓
Ticketing API
   ↓
Unauthorized Ticket Creation / Modification

Notice that this attack bypasses the LLM.

This is an important lesson:

AI threat modeling must include traditional identity and application security.

Step 20 — Assess Risk

Create:

10 Risk Register.md

Use a simple model:

Likelihood × Impact = Risk

Example:

ID	Threat	Likelihood	Impact	Inherent Risk
R-01	Unauthorized RAG Retrieval	High	High	Critical
R-02	RAG Poisoning	Medium	High	High
R-03	Agent Abuse	Medium	High	High
R-04	Agent Credential Theft	Medium	High	High
R-05	AI Resource Abuse	High	Medium	High

Student Task

For each risk consider:

Exposure
Required access
Attack complexity
Asset sensitivity
Privilege
Blast radius
Existing controls
Detectability

Step 21 — Identify Control Breakpoints

For AP-01:

Compromised Employee
       ↓
[Conditional Access]
       ↓
AI Assistant
       ↓
[Document Authorization]
       ↓
RAG
       ↓
Sensitive Data

For AP-03:

Malicious Document
       ↓
[Source Governance]
       ↓
RAG
       ↓
LLM
       ↓
[Restricted Agent Tools]
       ↓
[User Authorization]
       ↓
Ticketing API

Add control breakpoints to every important attack path.

Step 22 — Build the Threat-to-Control Matrix

Create:

11 Threat-to-Control Matrix.md

Use:

Threat	Prevent	Detect	Respond
Unauthorized RAG Retrieval	Document authorization	Retrieval monitoring	Disable account
RAG Poisoning	Source governance	Document-change alerts	Remove/reindex
Indirect Prompt Injection	Restricted agent tools	Agent action monitoring	Disable agent
Agent Credential Theft	Managed identity	API anomaly detection	Revoke identity
AI Resource Abuse	Rate limits	Cost alerts	Block requester

Step 23 — Define Security Requirements

Create:

12 Security Requirements.md

Write requirements that can actually be tested.

RAG-SEC-001

The RAG service must enforce source-document authorization using the initiating employee's identity.

RAG-SEC-002

Only approved identities may modify production RAG knowledge sources.

AGENT-SEC-001

The AI ticketing agent must use a dedicated least-privilege workload identity.

AGENT-SEC-002

Ticketing actions must be authorized using the initiating user's context where applicable.

LOG-SEC-001

AI agent tool calls must record the initiating user, agent identity, tool, action, target and result.

IR-SEC-001

Security Operations must be able to disable the AI agent identity during an incident.

Student Task

Create at least 8 security requirements.

Step 24 — Define Validation Criteria

For every important requirement define how you will verify it.

Example:

Requirement

RAG users may only retrieve authorized documents.

Test

Use:

User A — General Employee
User B — HR Employee

Attempt to retrieve an HR-only document.

Expected:

User A → DENIED

User B → ALLOWED

Step 25 — Validate Agent Least Privilege

Requirement:

AI ticketing agent can only create approved ticket types.

Expected:

Create Approved Ticket
        ↓
ALLOWED

Administrative Ticketing Action
        ↓
DENIED

Step 26 — Define Detection Requirements

Create at least 4 detection use cases.

Examples:

DET-01

Alert on unusually high RAG retrieval volume.

DET-02

Alert when users attempt retrieval from restricted knowledge categories.

DET-03

Alert on unusual AI agent ticket creation volume.

DET-04

Alert when production RAG sources are modified by unusual identities.

Step 27 — Define Response Actions

For each high-risk threat define containment.

Compromised User

Disable Account
Revoke Session
Investigate Retrieval

Poisoned RAG

Remove Document
Disable Source if Required
Rebuild Index
Review Related Responses

Agent Compromise

Disable Agent Identity
Revoke Token
Block Ticketing Tool
Review Agent Activity

Step 28 — Determine Residual Risk

Now consider the planned controls.

Example:

Unauthorized RAG Retrieval

Inherent Risk:

Critical

Controls:

Fine-grained retrieval authorization
Data segmentation
Logging

Residual Risk:

Medium

Document your reasoning.

Step 29 — Build the Executive Summary

Create:

13 Executive Summary.md

Use:

# Executive Summary

## System

Enterprise Knowledge Assistant

## Business Purpose

Provide employees with AI-powered access to approved enterprise knowledge and IT support automation.

## Overall Risk

High

## Critical Risks

1. Unauthorized RAG retrieval
2. AI agent privilege and authorization weaknesses
3. Knowledge poisoning and indirect prompt manipulation

## Security Strengths

- Enterprise authentication
- MFA
- Private vector database
- Dedicated services

## Priority Recommendations

1. Implement document-level RAG authorization.
2. Restrict knowledge source modification.
3. Apply least privilege to the AI agent.
4. Independently authorize tool actions.
5. Centralize RAG and agent activity logging.

## Residual Risk

Medium after recommended controls are implemented and validated.

Step 30 — Review Your Threat Model

Before finishing, verify that your threat model answers:

What Is the System?

What Are the Important Assets?

Where Does Sensitive Data Flow?

Where Does Trust Change?

Who Might Attack?

Where Can They Interact?

What Could They Do?

How Could the Attack Progress?

Which Asset Is Affected?

What Is the Business Impact?

What Controls Break the Attack Path?

How Would We Detect It?

How Would We Respond?

What Risk Remains?

If you can answer all of these, you have built a meaningful threat model.

Expected Deliverables

Your final lab should contain:

Business context
Scope
Assumptions and unknowns
Architecture diagram
Component inventory
Asset register
Crown jewels
Data flow register
Trust boundary register
Threat actor register
Attack surface register
STRIDE analysis
OWASP GenAI mapping
MITRE ATLAS considerations
At least 10 threat scenarios
At least 3 abuse cases
At least 4 attack paths
Risk register
Threat-to-control matrix
At least 8 security requirements
Detection requirements
Response actions
Executive summary

Lab Success Criteria

You have successfully completed the lab when you can demonstrate clear traceability between:

Asset
  ↓
Threat
  ↓
Attack Path
  ↓
Risk
  ↓
Control
  ↓
Security Requirement

For example:

Asset
Confidential HR Documents

       ↓

Threat
Unauthorized RAG Retrieval

       ↓

Attack Path
Employee → AI → RAG → HR Data

       ↓

Risk
Critical

       ↓

Control
Document-Level Authorization

       ↓

Requirement
RAG-SEC-001

That traceability is one of the most important skills in professional threat modeling.

Security Engineer Challenge

After completing the basic lab, imagine the architecture changes.

The organization adds:

AI Agent
   ↓
Cloud Administration API

The AI assistant can now:

Restart cloud workloads
Modify network configurations
Change IAM permissions

Revisit your existing threat model.

Determine:

Which new assets appear?
Which new trust boundaries exist?
Which attack surfaces expand?
Which existing threats become more severe?
Which new attack paths appear?
How does the blast radius change?
Which controls become mandatory?
Does the overall risk change?

You should quickly recognize that the system has moved from:

AI Knowledge Assistant

to:

AI-Enabled Privileged Operations Platform

The threat model must evolve accordingly.

Real-World Takeaway

This lab represents the basic workflow you may follow when asked:

“Can you perform a security review of our new AI application?”

Do not begin with:

Here's the OWASP AI Top 10.

Begin with:

What Is the Business Use Case?
        ↓
Show Me the Architecture.
        ↓
What Data Does It Access?
        ↓
Which Identities Does It Use?
        ↓
What Can the AI Actually Do?

Then systematically build the threat model.

This is how AI Security Engineers convert security frameworks into practical enterprise security decisions.

Lab Review Questions

Before moving forward, make sure you can explain:

Question 1

Why do we begin with business context rather than STRIDE?

Question 2

Why are assets important for threat modeling?

Question 3

Why should data flows and trust boundaries be documented separately?

Question 4

What makes RAG retrieval authorization different from authentication?

Question 5

How can a malicious document become an AI attack vector?

Question 6

Why must an AI agent have a dedicated identity?

Question 7

Why can prompt injection severity vary dramatically between systems?

Question 8

What is the difference between a threat and an attack path?

Question 9

What is the purpose of a control breakpoint?

Question 10

Why should security requirements be testable?

Question 11

Why are detection and response part of the threat model?

Question 12

What does residual risk represent?

Key Takeaways

In this lab, you transformed:

Enterprise AI Architecture

into:

Business Context
      ↓
Assets
      ↓
Data Flows
      ↓
Trust Boundaries
      ↓
Threat Actors
      ↓
Attack Surfaces
      ↓
Threat Scenarios
      ↓
Attack Paths
      ↓
Risk
      ↓
Security Controls
      ↓
Requirements

The most important lesson is:

AI threat modeling is not about creating the largest possible list of AI vulnerabilities. It is about understanding how the actual system could be abused and designing controls that protect the assets that matter most.

You have now built the foundation required for more specialized AI threat-modeling exercises.

What’s Next?

➡️ Lab 02 — Threat Model an Enterprise RAG Application

In the next lab, you will focus specifically on an enterprise RAG architecture.

You will investigate:

Multiple enterprise knowledge sources
Data classification
RAG ingestion
Source permissions
Embeddings
Vector databases
Retrieval authorization
Tenant and department isolation
Knowledge poisoning
Indirect prompt injection
Sensitive information disclosure
RAG logging and monitoring
RAG-specific attack paths

The goal is to move from:

“I can threat model a complete AI application.”

to:

“I can perform a deep security analysis of an enterprise RAG architecture.”

➡️ Next: Lab 02 — Threat Model an Enterprise RAG Application