10 Threat Modeling AI Agents

AI agents represent a major shift in modern AI systems.

A traditional LLM application may:

User
 ↓
Prompt
 ↓
LLM
 ↓
Response

An AI agent may instead:

User
 ↓
Prompt
 ↓
AI Agent
 ↓
Reasoning / Planning
 ↓
Tool Selection
 ↓
Enterprise API
 ↓
Real Action

This changes the security model significantly.

The system is no longer only generating information.

It may now:

Read enterprise data
Send email
Create tickets
Modify cloud resources
Update databases
Execute workflows
Call external APIs
Trigger business actions

This means threat modeling AI agents must focus not only on:

What can influence the AI?

but also:

What can the AI influence?

That distinction is critical.

Learning Objectives

By the end of this lesson, you should be able to:

Explain how AI agents differ from traditional LLM applications.
Identify important AI agent assets.
Map agent identities and permissions.
Identify agent trust boundaries.
Understand tool and API attack surfaces.
Identify confused deputy scenarios.
Analyze excessive agency.
Threat model agent memory.
Analyze indirect prompt injection against agents.
Identify risks involving external content.
Understand tool parameter manipulation.
Threat model agent-to-agent communication.
Understand human approval controls.
Identify production privilege risks.
Define logging and accountability requirements.
Understand emergency disable and kill-switch controls.
Build an AI Agent Threat Register.

1. What Is an AI Agent?

An AI agent is an AI-enabled system that can perform tasks using one or more tools.

Conceptually:

User Goal
   ↓
AI Agent
   ↓
Understand Task
   ↓
Select Tool
   ↓
Perform Action
   ↓
Observe Result
   ↓
Continue / Finish

Tools may include:

Search
Email
Databases
Cloud APIs
Ticketing systems
Code repositories
Browsers
File systems
Enterprise applications

The more tools and permissions an agent has, the more security impact it can potentially create.

2. AI Agent vs Chatbot

A chatbot may provide:

Information

An AI agent may provide:

Information
+
Decision
+
Action

For example:

Chatbot

User:
"How do I create an IAM role?"

AI:
Provides instructions.

Agent

User:
"Create an IAM role."

AI Agent:
Calls Cloud API
Creates IAM Role

The second architecture introduces much stronger security requirements.

3. Agentic AI Security Principle

The core security principle is:

AI-generated intent should never automatically equal authorization.

The agent may determine:

“This action seems appropriate.”

But trusted systems must determine:

“Is this action actually permitted?”

A safer pattern is:

User Request
     ↓
AI Agent
     ↓
Proposed Action
     ↓
Policy Validation
     ↓
Authorization
     ↓
Approval if Required
     ↓
Tool Execution

4. Understand the Agent Architecture

A typical enterprise agent may look like:

                     Employee
                         │
                         ▼
                    AI Agent
                         │
             ┌───────────┼───────────┐
             │           │           │
             ▼           ▼           ▼
            LLM        Memory      Tool Router
                                     │
                       ┌─────────────┼─────────────┐
                       │             │             │
                       ▼             ▼             ▼
                    Email API    Cloud API     Database

Threat modeling should examine each component separately.

5. Identify Agent Assets

Important assets may include:

Agent identity
Tool credentials
Tool configuration
Agent instructions
Agent memory
Conversation context
User identity
API tokens
Enterprise data
Logs
Approval records

A useful register might contain:

Asset	Why It Matters
Agent Identity	Determines what the agent can do
Tool Credentials	Provide access to enterprise systems
Agent Configuration	Controls available tools
Memory	May contain sensitive information
Logs	Support accountability and investigation

6. Agent Identity

Every agent should have a clearly defined identity where possible.

Example:

AI Agent
   ↓
Dedicated Workload Identity
   ↓
Enterprise API

Avoid:

Multiple Agents
      ↓
Shared Admin Credential

Shared credentials make it difficult to:

Apply least privilege
Attribute actions
Revoke access
Investigate incidents

7. Agent Permissions

For each agent, ask:

What Can It Read?

What Can It Write?

What Can It Delete?

What Can It Execute?

What Can It Approve?

What Can It Administer?

Example:

Cloud Security Agent
   ├── Read Security Findings
   ├── Read IAM Policies
   └── Read CloudTrail Logs

may be reasonable.

But:

Cloud Security Agent
   ├── Administrator Access
   ├── IAM Modification
   ├── Network Modification
   └── Resource Deletion

may introduce excessive risk.

8. Least Privilege for Agents

AI agents should follow the same least-privilege principle as human and workload identities.

Grant:

Only what is required for the specific business task.

Prefer:

Read Findings

over:

AdministratorAccess

when only read capability is required.

9. Tool Inventory

Create an inventory of every tool the agent can use.

Example:

Tool	Purpose	Permission	Risk
Search	Find documentation	Read	Low
Ticketing	Create incidents	Write	Medium
Email	Send notifications	Send	Medium
Cloud API	Modify resources	Admin	Critical

This helps identify the agent’s effective capability.

10. Tool Risk Classification

A practical model is:

Low Risk

Read public data
Search documentation

Medium Risk

Read internal data
Create tickets
Send internal notifications

High Risk

Modify business data
Write configuration
Execute code

Critical Risk

Modify IAM
Delete production resources
Transfer funds
Access highly sensitive systems
Perform administrative actions

Agent controls should increase with tool risk.

11. Agent Trust Boundaries

A typical agent has several trust boundaries:

User
 ↓
====================
TB-A01
====================
 ↓
Agent
 ↓
====================
TB-A02
====================
 ↓
Tool
 ↓
====================
TB-A03
====================
 ↓
Enterprise System

Another boundary may exist between:

External Content
      ↓
====================
TB-A04
====================
      ↓
Agent Context

These are critical threat modeling points.

12. User-to-Agent Boundary

Potential threats include:

Compromised user identity
Malicious instructions
Privilege abuse
Social engineering through natural language

Security controls may include:

Authentication
Authorization
Session protection
Request validation
Logging

13. Agent-to-Tool Boundary

Potential threats include:

Unauthorized tool selection
Parameter manipulation
Excessive permissions
Tool spoofing

Security controls may include:

Tool allowlists
Parameter validation
Policy enforcement
Agent identity
Logging

14. Tool-to-System Boundary

Even if the agent has chosen a tool, the destination system should enforce its own security.

Example:

AI Agent
   ↓
Tool
   ↓
Cloud API

The cloud API should independently validate:

Identity
Authorization
Resource permissions

Do not rely only on the agent’s logic.

15. The Confused Deputy Problem

AI agents can become a classic confused deputy.

Example:

Low-Privilege User
        ↓
AI Agent
        ↓
High-Privilege Credential
        ↓
Sensitive System

The user cannot directly access the system.

But if the agent acts using higher privilege, the user may indirectly gain unauthorized capabilities.

This is one of the most important agent threat scenarios.

16. User-Context Authorization

A strong design considers:

User Identity
      +
Agent Identity
      +
Requested Action
      +
Target Resource

before execution.

Example:

User
 ↓
Agent
 ↓
Proposed Action
 ↓
Authorization Check
 ↓
Allowed?

The agent’s permissions alone should not determine whether the operation is authorized.

17. Excessive Agency

Excessive agency occurs when an AI system has more authority, autonomy or tools than required.

Examples:

Too many tools
Excessive IAM permissions
Automatic execution
No approval for sensitive operations
Broad data access

A threat model should ask:

Does the agent need this capability at all?

18. Capability Reduction

Instead of trying only to improve model behavior, reduce capability where possible.

Example:

Instead of:

General Cloud Admin Agent

consider:

Read-Only Cloud Security Analysis Agent

This reduces blast radius significantly.

19. Direct Prompt Manipulation

A user may attempt to influence agent behavior through direct instructions.

Example:

User
 ↓
Manipulated Prompt
 ↓
Agent
 ↓
Tool

The real impact depends on:

Tool permissions
Authorization
Policy enforcement
Approval

Prompt injection becomes far more serious when the agent has powerful tools.

20. Indirect Prompt Injection

Agents may process untrusted external content.

Example:

Attacker
   ↓
Malicious Email
   ↓
AI Agent
   ↓
Tool

or:

Attacker
   ↓
Website
   ↓
Browsing Agent
   ↓
Tool

The attacker may influence agent behavior without directly interacting with the agent.

21. Example — Malicious Email Agent Attack

Consider an AI email assistant that:

Reads incoming messages
Creates support tickets
Sends responses

Attack path:

External Attacker
      ↓
Malicious Email
      ↓
AI Email Agent
      ↓
Interprets Malicious Content
      ↓
Creates Unauthorized Ticket / Sends Data

The email itself becomes an attack input.

22. External Content Should Be Untrusted

A core agentic AI principle:

External content should be treated as data, not authority.

Examples:

Emails
Websites
PDFs
Documents
Search results
Third-party API responses

These may contain adversarial instructions.

23. Tool Parameter Manipulation

Even if the correct tool is selected, parameters may still be dangerous.

Example:

Tool:
DeleteFile

Parameter:
"/production/data"

The problem may not be the tool.

It may be the selected target.

Therefore validate:

Target resource
Operation
Scope
Quantity
Destination

24. Parameter Policy Enforcement

A safer architecture:

AI Agent
   ↓
Tool + Parameters
   ↓
Policy Validator
   ↓
Allowed Scope?
   ↓
Execution

The validation layer should be deterministic where possible.

25. High-Risk Tool Approval

Some actions should require stronger controls.

Examples:

IAM modification
Production changes
Data deletion
External data transfer
Financial transactions

Use:

Agent
 ↓
Proposed Action
 ↓
Human Approval
 ↓
Execution

when appropriate.

26. Human-in-the-Loop

Human approval can reduce risk, but only when implemented meaningfully.

The reviewer should see:

Requested action
Target resource
Reason
Impact
Agent identity
User identity

Avoid approval prompts such as:

Approve? Yes / No

without enough context.

27. Human-on-the-Loop

Some environments use oversight rather than per-action approval.

Conceptually:

Agent Operates
      ↓
Continuous Monitoring
      ↓
Human Supervisor

This may be appropriate for lower-risk actions.

Higher-risk operations may still require explicit approval.

28. Agent Memory

Agents may maintain memory.

Memory may include:

Previous conversations
User preferences
Workflow information
Retrieved content
Tool results

This becomes another security asset.

29. Memory Confidentiality

Ask:

Does memory contain sensitive information?
Is memory separated between users?
How long is it retained?
Who can access it?

A multi-user agent must avoid:

User A Memory
      ↓
User B

30. Memory Integrity

Attackers may attempt to insert persistent malicious information into memory.

Example:

Attacker Input
      ↓
Agent Memory
      ↓
Future Sessions

This can create persistent influence.

Threat modeling should therefore consider:

Who can write memory?
What gets stored?
Can users clear it?
Is memory validated?

31. Short-Term vs Long-Term Memory

Different memory types may have different risk.

Short-Term Memory

Current conversation context.

Long-Term Memory

Persistent information stored across sessions.

Long-term memory deserves stronger:

Access control
Retention
Integrity protection
Privacy review

32. Agent-to-Agent Communication

Modern architectures may contain multiple agents.

Example:

User
 ↓
Coordinator Agent
      ↓
  ┌───┼─────────┐
  │   │         │
  ▼   ▼         ▼
Cloud Security  Ticketing
Agent           Agent

Now trust becomes more complicated.

33. Multi-Agent Trust

Ask:

Does one agent trust another automatically?
What identity does each agent use?
Can one agent instruct another to perform privileged actions?
Is authorization preserved between agents?

Avoid:

Agent A
   ↓
"Do this"
   ↓
Agent B Executes Without Validation

Prefer:

Agent A
   ↓
Proposed Task
   ↓
Policy / Authorization
   ↓
Agent B

34. Agent Impersonation

In multi-agent environments, spoofing becomes important.

If Agent B receives a request:

“Agent A told me to perform this action.”

how is that identity verified?

Agent-to-agent authentication may therefore be required.

35. Delegation Chains

Consider:

User
 ↓
Agent A
 ↓
Agent B
 ↓
Tool
 ↓
Production

The system should preserve:

Original User
      +
Agent A
      +
Agent B
      +
Final Action

for authorization and accountability.

36. Delegated Authorization

A common challenge is determining what authority should be delegated.

A useful principle is:

Delegation should never increase the original user’s authorization unless explicitly approved.

Conceptually:

User Permission
      ≥
Delegated Agent Permission

not:

User Permission
      <
Agent Permission

without additional controls.

37. Agent Credential Compromise

Agents may rely on:

API keys
OAuth tokens
Cloud identities
Certificates

If compromised:

Attacker
   ↓
Agent Credential
   ↓
Enterprise Tool

the attacker may bypass the AI layer completely.

This is why traditional credential security remains essential.

38. Secret Management

Use:

Managed identities
Secret stores
Short-lived tokens
Rotation
Scoped permissions

Avoid:

Hardcoded credentials
Credentials in prompts
Secrets in logs
Shared administrator accounts

39. Tool Output Risks

Tool output also becomes model input.

Example:

Agent
 ↓
Tool
 ↓
Tool Response
 ↓
LLM

If the tool returns untrusted external content, it may influence subsequent reasoning.

Therefore:

Tool output may also represent an untrusted input boundary.

40. Tool Response Manipulation

Consider:

External API
      ↓
Malicious / Manipulated Response
      ↓
Agent
      ↓
Next Action

This is another form of indirect influence.

41. Browser Agents

Browser-capable agents have large attack surfaces.

They may process:

HTML
JavaScript-rendered content
Forms
Search results
Downloads

Threats include:

Malicious web content
Indirect prompt injection
Unsafe downloads
Credential exposure
Unauthorized actions

Browser agents should operate with carefully limited privileges.

42. Code Execution Agents

Some agents can execute:

Python
Shell commands
Scripts
Infrastructure commands

This creates significant risk.

Example:

User
 ↓
AI Agent
 ↓
Generated Command
 ↓
Shell

Do not use:

LLM Output → Direct Execution

for high-risk environments.

43. Safer Code Execution Pattern

Prefer:

LLM
 ↓
Generated Proposal
 ↓
Validation
 ↓
Sandbox
 ↓
Restricted Execution
 ↓
Review

Use isolation and least privilege.

44. Sandboxing

Execution tools may require sandboxing.

Potential controls include:

Container isolation
Restricted filesystem
No production credentials
Network restrictions
Resource limits
Timeouts

Sandboxing reduces blast radius.

45. Production Access

One of the most important threat modeling questions is:

Does this agent have production access?

If yes, determine:

Read vs write
Administrative capability
Environment scope
Approval requirements
Logging

Production access should significantly increase review depth.

46. Environment Separation

Prefer:

Development Agent
      ↓
Development Resources

and:

Production Agent
      ↓
Production Resources

Avoid allowing development agents broad production access.

47. Transaction Limits

Agents performing business actions may require quantitative limits.

Examples:

Maximum 10 Tickets per Request

Maximum 1 Resource Change per Approval

Maximum ₹X Transaction Value

Maximum 5 External Emails

Limits reduce impact from unexpected behavior.

48. Rate Limits

Rate limiting helps prevent:

Tool flooding
Resource exhaustion
Repeated actions
Cost abuse

Apply rate limits at:

User
Agent
Tool
API

levels where appropriate.

49. Agent Loops

Autonomous agents may enter loops.

Example:

Think
 ↓
Tool
 ↓
Observe
 ↓
Think
 ↓
Tool
 ↓
Observe
 ↓
...

Potential consequences:

Excessive cost
API exhaustion
repeated changes
service disruption

50. Loop Controls

Possible controls:

Maximum iterations
Maximum tool calls
Time limits
Token budgets
Cost budgets
Failure thresholds

51. Logging and Accountability

Agentic actions require strong auditability.

Logs should help answer:

Who Initiated the Request?

Which Agent Processed It?

Which Tool Was Selected?

What Parameters Were Used?

Which Resource Was Targeted?

Was Approval Required?

Who Approved It?

What Was the Result?

This supports both security and operational investigation.

52. Agent Audit Trail

A useful sequence:

User ID
  ↓
Request ID
  ↓
Agent ID
  ↓
Model Interaction
  ↓
Tool Call
  ↓
Authorization Decision
  ↓
Approval
  ↓
Enterprise Action
  ↓
Result

Maintain correlation identifiers where practical.

53. Sensitive Logging

Do not log everything blindly.

Prompts and tool results may contain:

Credentials
Personal information
Customer data
Source code

Balance:

Security Visibility
       +
Privacy
       +
Data Minimization

54. Agent Kill Switch

High-impact agents should have a way to quickly stop activity.

Example:

Suspicious Agent Behavior
        ↓
Security Decision
        ↓
Disable Agent Identity
        ↓
Block Tool Access

Possible kill mechanisms include:

Disable workload identity
Revoke token
Disable tool integration
Stop agent service
Apply deny policy

55. Why Kill Switches Matter

During an incident, security teams may not have time to understand every detail before containment.

They need to be able to answer:

How do we stop this agent now?

The architecture should have an answer.

56. Fail-Safe Behavior

Ask what happens when:

LLM is unavailable
Authorization system fails
Policy engine fails
Tool returns unexpected data
Approval system is unavailable

Prefer secure failure modes.

For example:

Authorization Service Failure
        ↓
DENY ACTION

rather than:

Authorization Service Failure
        ↓
ALLOW ACTION

57. Apply STRIDE to AI Agents

Spoofing

User impersonation
Agent identity theft
Tool impersonation
Agent-to-agent impersonation

Tampering

Agent instruction modification
Memory manipulation
Tool parameter modification
Tool configuration changes

Repudiation

Missing user attribution
Missing tool logs
Missing approval evidence

Information Disclosure

Agent reveals sensitive data
Tool output exposes secrets
Cross-user memory leakage

Denial of Service

Agent loops
Excessive tool calls
Resource exhaustion

Elevation of Privilege

Low-privileged user controls privileged agent
Agent gains excessive tool permissions

58. Apply OWASP GenAI Guidance

Relevant areas may include:

Prompt injection
Sensitive information disclosure
Improper output handling
Excessive agency
Supply-chain risk

Use current official OWASP guidance during real assessments.

59. Apply MITRE ATLAS

Adversary behaviors may include:

Discovering agent capabilities
Manipulating AI context
Abusing AI resources
Collecting information
Causing impact

Validate current ATLAS tactics and techniques when mapping the scenario.

60. Agent Attack Path 1 — Privileged Tool Abuse

Compromised Employee
       ↓
AI Agent
       ↓
Manipulated Request
       ↓
Privileged Tool
       ↓
Production API
       ↓
Unauthorized Change

Primary weakness:

User authorization is not enforced independently of agent privilege.

Primary controls:

User-context authorization
Least privilege
Tool restrictions
Approval
Logging

61. Agent Attack Path 2 — Indirect Prompt Injection

External Attacker
      ↓
Malicious Email
      ↓
AI Agent
      ↓
Interprets Content as Instruction
      ↓
Enterprise Tool
      ↓
Unauthorized Action

Controls:

Treat external content as untrusted
Restrict tools
Policy enforcement
Approval for sensitive actions
Monitor content-to-action chains

62. Agent Attack Path 3 — Credential Compromise

Attacker
   ↓
Agent Credential
   ↓
Enterprise API
   ↓
Sensitive Resource

This attack may completely bypass the LLM.

Controls:

Managed identity
Short-lived credentials
Secret protection
Least privilege
Behavioral monitoring

63. Agent Attack Path 4 — Memory Poisoning

Attacker Input
      ↓
Persistent Agent Memory
      ↓
Future Session
      ↓
Influenced Agent Behavior

Controls:

Restrict persistent memory writes
Validate stored information
User isolation
Retention limits
Ability to reset memory

64. Agent Attack Path 5 — Multi-Agent Privilege Escalation

User
 ↓
Low-Privilege Agent
 ↓
Privileged Agent
 ↓
Production Tool

If delegation rules are weak, low-privileged workflows may reach high-privileged capabilities.

Controls:

Agent-to-agent authentication
Delegation authorization
Preserve initiating-user context
Restrict privileged agents

65. Agent Attack Path 6 — Tool Response Manipulation

Agent
 ↓
External Tool
 ↓
Manipulated Response
 ↓
LLM
 ↓
Next Tool Action

This illustrates why tool responses can also become untrusted inputs.

66. Blast Radius Analysis

For every agent ask:

If this agent were fully compromised, what could it affect?

Example:

Agent Compromised
      ↓
What Can It Read?
      ↓
What Can It Modify?
      ↓
What Can It Delete?
      ↓
What External Systems Can It Reach?

This defines the potential blast radius.

67. Reduce Agent Blast Radius

Use:

Least privilege
Separate identities
Restricted tools
Data segmentation
Network restrictions
Environment separation
Read-only access where possible
Transaction limits
Approval gates

68. Agent Security Requirements

Threat models should produce actionable requirements.

Examples:

AGENT-SEC-001
Every production AI agent must use a dedicated workload identity.

AGENT-SEC-002
Agent permissions must be limited to operations required by the approved business workflow.

AGENT-SEC-003
Sensitive tool requests must be independently authorized using the initiating user's context.

AGENT-SEC-004
High-impact production actions must require explicit approval where defined by policy.

AGENT-SEC-005
All production agent tool calls must be auditable.

AGENT-SEC-006
Security operations must be able to immediately disable agent access during an incident.

69. Build an AI Agent Threat Register

Create an Obsidian note:

AI Agent Threat Register.md

Use:

ID	Threat Actor	Attack Surface	Threat	Target Asset	Impact	Existing Controls	Required Controls	Risk
AG-T01	Compromised User	Prompt Interface	Privileged tool abuse	Production	Critical	Authentication	User authz + least privilege	Critical
AG-T02	External Attacker	Email Content	Indirect prompt injection	Agent Tools	High	Mail Security	Policy enforcement	Critical
AG-T03	External Attacker	Agent Credential	Credential compromise	Enterprise API	Critical	Secret Store	Managed identity	Critical
AG-T04	User	Persistent Memory	Memory poisoning	Agent Behavior	High	Basic ACL	Memory governance	High
AG-T05	Low-Privilege Agent	Agent API	Privilege delegation	Production Tools	Critical	Agent Auth	Delegation policy	Critical

70. Agent Threat Scenario Template

Create:

# AG-TXX — Threat Name

## Threat Actor

## Business Scenario

## Target Agent

## Attack Surface

## Trust Boundary

## Target Asset

## Agent Identity

## Agent Tools

## Agent Permissions

## STRIDE Mapping

## OWASP Mapping

## MITRE ATLAS Mapping

## Preconditions

## Attack Path

## Existing Controls

## Security Gap

## Detection Opportunities

## Recommended Controls

## Containment Method

## Business Impact

## Risk Rating

71. Example — Unauthorized Production Change

# AG-T01 — Unauthorized Production Change Through AI Agent

## Threat Actor

Compromised employee account

## Business Scenario

Employees use an AI cloud assistant to analyze and manage cloud resources.

## Target Agent

Cloud Operations Agent

## Attack Surface

Natural-language agent interface

## Trust Boundary

Employee → AI Agent → Cloud API

## Target Asset

Production cloud infrastructure

## Agent Identity

Privileged cloud workload identity

## Agent Tools

Cloud administration API

## Agent Permissions

Resource modification permissions

## STRIDE Mapping

Elevation of Privilege

## OWASP Mapping

Excessive agency / prompt injection-related guidance

## MITRE ATLAS Mapping

Validate relevant current adversary technique.

## Preconditions

The attacker controls a valid employee session.

## Attack Path

Compromised User
      ↓
AI Agent
      ↓
Manipulated Request
      ↓
Cloud Administration Tool
      ↓
Privileged Workload Identity
      ↓
Production Resource Modification

## Existing Controls

- Enterprise authentication
- Agent workload identity

## Security Gap

The agent authorizes actions using its own privileges without sufficiently validating whether the initiating employee is authorized for the requested operation.

## Detection Opportunities

- Privileged tool calls initiated by low-privileged users
- Unusual production changes
- Agent-to-user privilege mismatch

## Recommended Controls

- User-context authorization
- Least-privilege agent permissions
- Restricted tool set
- Parameter validation
- Approval for high-risk operations
- Centralized agent logging

## Containment Method

Disable agent workload identity and revoke active tokens.

## Business Impact

Unauthorized production changes could result in service disruption, security control modification or data exposure.

## Risk Rating

Critical

72. AI Agent Security Checklist

Identity

Dedicated agent identity exists
Shared credentials avoided
Credentials are short-lived where possible
Agent authentication is logged

Permissions

Least privilege applied
Production access justified
Administrative permissions minimized
Environment access separated

Tools

Tools inventoried
Tool risks classified
Unnecessary tools removed
Tool access restricted
Parameters validated

Authorization

Initiating-user authorization preserved
Agent privilege does not automatically override user privilege
Target-resource authorization enforced

Input

User input treated as untrusted
External content treated as untrusted
File and email content considered
Tool responses treated appropriately

Memory

Memory access controlled
Users isolated
Persistent memory governed
Retention defined
Memory reset available

Multi-Agent

Agents authenticate each other
Delegation rules defined
User context preserved
Privileged agents restricted

Approval

High-impact actions identified
Approval requirements defined
Approval contains enough context
Approval recorded

Execution

Code execution isolated where applicable
Resource limits defined
Tool-call limits defined
Loop limits configured

Logging

Initiating user recorded
Agent identity recorded
Tool calls logged
Parameters logged appropriately
Approval logged
Results logged

Incident Response

Agent can be disabled quickly
Credentials can be revoked
Tools can be blocked
Investigation telemetry exists

73. Common Agent Threat Modeling Mistakes

Focusing Only on Prompt Injection

Prompt injection is one threat.

Agent privilege determines the potential impact.

Treating Agent Identity as the User’s Authority

The agent may have more permissions than the initiating user.

Always consider both.

Giving Agents Too Many Tools

Every tool increases attack surface.

Using Shared Administrative Credentials

This increases blast radius and reduces accountability.

Ignoring Memory

Persistent memory can become both a data asset and a manipulation surface.

Ignoring Tool Responses

Tools may return untrusted data that influences future agent actions.

Ignoring Multi-Agent Delegation

Multiple agents create additional identity and trust relationships.

Assuming Human Approval Solves Everything

Poor approval design may result in users approving actions without understanding them.

Missing Containment

Every high-risk agent should have a clear answer to:

How do we stop it?

74. Real-World AI Security Engineer Workflow

When reviewing an AI agent:

Understand Business Purpose
        ↓
Identify Agent Identity
        ↓
Inventory Tools
        ↓
Review Permissions
        ↓
Map User Authorization
        ↓
Map Trust Boundaries
        ↓
Identify External Inputs
        ↓
Review Memory
        ↓
Review Agent Delegation
        ↓
Identify High-Risk Actions
        ↓
Review Approval Controls
        ↓
Review Logging
        ↓
Define Kill Switch
        ↓
Build Threat Scenarios
        ↓
Reduce Blast Radius

75. AI Agent Threat Modeling Mindset

Whenever you see an AI agent, ask:

Who Can Instruct the Agent?
        ↓
What Content Can Influence It?
        ↓
Which Identity Does It Use?
        ↓
Which Tools Can It Call?
        ↓
What Permissions Do Those Tools Have?
        ↓
Can the Agent Act Beyond User Authority?
        ↓
Does It Have Persistent Memory?
        ↓
Can Other Agents Instruct It?
        ↓
Which Actions Require Approval?
        ↓
What Happens If the Agent Is Fully Compromised?
        ↓
How Do We Detect Abuse?
        ↓
How Do We Stop It?

76. Knowledge Check

Question 1

What is the main security difference between a chatbot and an AI agent?

Question 2

Why should AI-generated intent not equal authorization?

Question 3

What is excessive agency?

Question 4

Why should agents use dedicated identities?

Question 5

What is the confused deputy problem in an AI agent architecture?

Question 6

Why should user-context authorization be preserved?

Question 7

How can indirect prompt injection affect an AI agent?

Question 8

Why should tool parameters be independently validated?

Question 9

What risks can persistent agent memory introduce?

Question 10

Why does agent-to-agent communication create new trust boundaries?

Question 11

Why should high-impact agent actions sometimes require human approval?

Question 12

Why should tool responses sometimes be considered untrusted?

Question 13

What is an agent kill switch?

Question 14

How does least privilege reduce agent blast radius?

Key Takeaways

Threat modeling AI agents requires understanding the entire action chain:

User
  ↓
Agent
  ↓
Model
  ↓
Decision
  ↓
Tool
  ↓
Authorization
  ↓
Enterprise System
  ↓
Real-World Impact

The most important agent security areas include:

Identity
     +
Permissions
     +
Tools
     +
User Authorization
     +
Untrusted Context
     +
Memory
     +
Agent Delegation
     +
Approval
     +
Logging
     +
Containment

The core principle is:

Assume that AI behavior can sometimes be manipulated or fail unexpectedly, and design the surrounding architecture so that those failures cannot automatically become high-impact enterprise actions.

A secure agent should have:

Limited authority
Explicit identity
Restricted tools
Independent authorization
Strong logging
Controlled approval
Small blast radius
Clear containment mechanisms

What’s Next?

➡️ 11 — Building AI Attack Paths and Abuse Cases

You now understand how to threat model individual AI components and architectures, including:

LLM applications
RAG systems
AI agents
Trust boundaries
Threat actors
STRIDE threats
OWASP GenAI risks
MITRE ATLAS adversary behaviors

The next step is to connect these individual threats together.

In the next lesson, you will learn how to build:

AI abuse cases
Threat scenarios
Multi-stage attack paths
RAG-to-agent attack chains
Identity-to-agent privilege escalation paths
Supply-chain attack paths
Data-exfiltration paths
Production-impact scenarios
Prevent → Detect → Respond mappings

The goal is to move from:

“I can identify individual AI threats.”

to:

“I can explain exactly how an attacker could move through an AI architecture from initial access to business impact.”

➡️ Next: 11 — Building AI Attack Paths and Abuse Cases