04 Enterprise AI Architecture

Enterprise AI is rarely a single chatbot connected directly to a model.

In real organizations, AI systems are usually part of a much larger environment involving:

Users
Applications
Identity systems
APIs
Large Language Models
Retrieval-Augmented Generation
Vector databases
Enterprise data
AI agents
Cloud infrastructure
Containers
Kubernetes
Secrets
Logging
Monitoring
Governance controls

As an AI Security Engineer, one of your most important skills is learning how to look at this complete architecture and understand:

What are the components?

How do they communicate?

Where does sensitive data flow?

Which trust boundaries exist?

What permissions are required?

Where should security controls be applied?

This lesson gives you that enterprise architecture perspective.

Learning Objectives

By the end of this lesson, you should be able to:

Understand the major layers of an enterprise AI architecture.
Recognize common AI deployment patterns.
Understand how users, applications, models and enterprise data connect.
Understand the role of IAM in AI environments.
Understand common RAG architecture patterns.
Understand AI agent architecture.
Recognize model hosting options.
Understand cloud and network security considerations.
Understand secrets management for AI workloads.
Identify important trust boundaries.
Understand logging and monitoring requirements.
Recognize where security controls should be placed.
Understand shared responsibility across AI teams.
Perform a basic security review of an enterprise AI architecture.

Why Architecture Matters

Security controls only make sense when you understand the system they protect.

Imagine someone asks:

Is this AI application secure?

Without understanding the architecture, this question is almost impossible to answer.

You first need to know:

Who Uses It?
     ↓
How Do They Authenticate?
     ↓
What Application Do They Access?
     ↓
Which Model Is Used?
     ↓
What Data Can the Model Access?
     ↓
Are RAG or Agents Used?
     ↓
What Infrastructure Supports It?
     ↓
What Actions Can It Perform?
     ↓
What Is Logged?

The architecture gives security context.

A Simple Enterprise AI Architecture

A basic enterprise AI application may look like:

Employee
   │
   ▼
Identity Provider
   │
   ▼
AI Application
   │
   ▼
Application Backend
   │
   ├── LLM API
   ├── RAG
   └── Logging

This might be sufficient for a basic internal assistant.

But enterprise environments often evolve into something larger.

A More Complete Enterprise AI Architecture

                            Users
                              │
                              ▼
                      Identity Provider
                              │
                              ▼
                        AI Application
                              │
                              ▼
                      Application Backend
                              │
        ┌─────────────────────┼──────────────────────┐
        │                     │                      │
        ▼                     ▼                      ▼
   LLM Service              RAG Layer             AI Agent
        │                     │                      │
        │                     ▼                      ├── Email
        │              Vector Database               ├── Database
        │                     │                      ├── Cloud API
        │                     ▼                      └── Ticketing
        │              Enterprise Data
        │
        ▼
    Model Runtime
        │
        ▼
Cloud / Container / Kubernetes Infrastructure
        │
        ▼
Logging / Monitoring / SIEM

This is closer to how enterprise AI systems should be viewed from a security perspective.

Think in Architecture Layers

A useful way to understand enterprise AI is by breaking it into layers.

Layer 1 — Users & Identity
Layer 2 — Application
Layer 3 — AI Orchestration
Layer 4 — Models
Layer 5 — Data & Knowledge
Layer 6 — Agents & Tools
Layer 7 — Infrastructure
Layer 8 — Security Operations
Layer 9 — Governance

Each layer creates different security concerns.

Layer 1 — Users and Identity

Every AI system begins with an identity.

Users may include:

Employees
Customers
Developers
Administrators
Partners
Applications
Service accounts
Automated workloads

Security should begin by understanding:

Who is accessing the system?

Authentication

Authentication answers:

Who are you?

Common enterprise authentication methods may include:

Username and password
Multi-Factor Authentication
Single Sign-On
Federation
Certificates
API credentials
Workload identities

For enterprise applications, centralized identity is generally preferable to creating separate unmanaged accounts for every AI application.

Example

Employee
   │
   ▼
Enterprise Identity Provider
   │
   ▼
AI Application

This allows the AI application to use existing enterprise identity controls.

Authorization

Authentication is not enough.

Authorization answers:

What are you allowed to do?

For example:

HR Employee
   │
   ├── HR Policy Documents
   └── HR Knowledge

Developer
   │
   ├── Engineering Documentation
   └── Source Code Guidance

One user should not automatically receive access to everything the AI platform knows.

Identity Must Flow Through the Architecture

Suppose a user authenticates correctly at the web application.

That identity may need to influence downstream systems.

User Identity
     ↓
Application
     ↓
Authorization
     ↓
RAG Retrieval
     ↓
Allowed Documents Only

If identity is lost between layers, the AI system may retrieve information the user should not access.

This is a major enterprise architecture concern.

Layer 2 — AI Application

The AI application is the interface users interact with.

Examples include:

Internal chatbot
Security copilot
Customer assistant
Coding assistant
Knowledge assistant
Document analyzer

The application may include:

Frontend
   +
Backend
   +
API
   +
Authentication
   +
Authorization
   +
Session Management

Traditional application security remains important here.

Application Security Questions

Ask:

Is authentication implemented correctly?
Are sessions protected?
Are APIs secured?
Is access control enforced?
Is input treated as untrusted?
Are errors handled safely?
Are secrets exposed?
Is output safely rendered?

An AI application can still have traditional application vulnerabilities.

Layer 3 — AI Orchestration Layer

Many enterprise AI applications include an orchestration layer.

This layer may determine:

Which model to use
Which prompt to construct
Whether RAG is required
Which tools the AI can access
What policies apply
What actions require approval

A simplified architecture:

User Request
     ↓
Orchestration Layer
     │
     ├── Prompt Builder
     ├── RAG Retriever
     ├── Model Router
     ├── Tool Selector
     ├── Policy Engine
     └── Logging

This layer is highly security-sensitive.

Why Orchestration Matters

Suppose the application asks:

"Show me the latest cloud incident."

The orchestration layer may:

Identify the user.
Check permissions.
Query the incident knowledge base.
Select relevant documents.
Build the model context.
Call the LLM.
Validate the response.
Return the result.

Every step introduces security decisions.

Layer 4 — Model Layer

The model layer contains the AI models used by the application.

Organizations may use:

Hosted commercial models
Cloud-managed models
Open models
Self-hosted models
Fine-tuned models
Multiple models for different purposes

A model routing architecture may look like:

AI Application
      │
      ▼
Model Router
   ┌──┼────────┐
   │  │        │
   ▼  ▼        ▼
Model A     Model B     Model C

Different models may have different:

Cost
Performance
Security
Data handling
Availability
Regulatory requirements

Model Security Questions

Ask:

Where is the model hosted?
Who owns it?
Who can access it?
Is customer data sent to the provider?
Are prompts retained?
Can the model be changed without approval?
How is model integrity verified?
Is the model version tracked?

Model Versioning

Enterprise environments should know which model is running.

For example:

Application Version: 3.2
Model Version: 2026-08
Prompt Version: 1.7
RAG Index Version: 4.1

This helps with:

Incident investigation
Rollback
Testing
Auditability

Without version tracking, it may be difficult to understand why an AI system behaved differently over time.

Layer 5 — Data and Knowledge

AI systems are only as trustworthy as the data they use.

Enterprise AI may access:

Policies
Customer data
Source code
Security logs
Internal documentation
Financial records
HR information
Technical architecture
Email
Tickets

Data should be treated as a core security asset.

Data Classification

Not all data should be treated equally.

A simple classification model may include:

Public
Internal
Confidential
Restricted

AI applications should understand what types of data they are allowed to process.

Data Minimization

One of the strongest architecture principles is:

Provide the AI only the data required for the task.

Bad design:

User asks one policy question
          ↓
Entire document repository sent to model

Better:

User Question
      ↓
Authorized Retrieval
      ↓
Relevant Approved Content
      ↓
LLM

This reduces risk.

Layer 6 — RAG Architecture

Retrieval-Augmented Generation is common in enterprise AI.

A typical RAG architecture looks like:

Enterprise Documents
       ↓
Document Processing
       ↓
Embedding Model
       ↓
Vector Database
       ↓
Retriever
       ↓
User Question
       ↓
Relevant Content
       ↓
LLM
       ↓
Response

This architecture introduces several security layers.

Document Ingestion

Before documents reach the vector database, they may go through:

Source
  ↓
Validation
  ↓
Parsing
  ↓
Chunking
  ↓
Embedding
  ↓
Indexing

Security questions include:

Who can upload documents?
Are documents scanned?
Can malicious content enter the pipeline?
Is document ownership tracked?
Is data classification preserved?

Retrieval Security

The retrieval layer must enforce authorization.

Bad architecture:

User
  ↓
Search Entire Vector Database

Better:

User
  ↓
Identity
  ↓
Authorization Filter
  ↓
Allowed Documents
  ↓
Vector Search

The vector database should not become a shortcut around enterprise access controls.

Multi-Tenant RAG

Some systems support multiple departments or customers.

For example:

Shared AI Platform
      │
      ├── Customer A Data
      ├── Customer B Data
      └── Customer C Data

Security requires strong tenant isolation.

A failure may create cross-tenant data exposure.

Layer 7 — AI Agents and Tools

AI agents extend an AI platform from:

Answer Questions

to:

Take Actions

This changes the security risk significantly.

Basic Agent Architecture

User
 │
 ▼
Agent
 │
 ├── LLM
 ├── Planner
 ├── Memory
 └── Tools
      │
      ├── Email
      ├── Database
      ├── Cloud
      └── Ticketing

The tool layer becomes critical.

Tool Security

Each tool should have clearly defined permissions.

For example:

AI Security Agent
      │
      ├── Read SIEM Alerts
      ├── Read Cloud Logs
      └── Create Draft Ticket

This is safer than:

AI Security Agent
      │
      └── Full Administrator Access Everywhere

Agent Permission Architecture

A strong model is:

User Permission
      ↓
Application Authorization
      ↓
Agent Policy
      ↓
Tool Permission
      ↓
Action

Each stage restricts what the next layer can do.

Human Approval

High-risk actions may require approval.

For example:

AI Agent
   ↓
Proposes:
Disable User Account
   ↓
Human Approval
   ↓
Identity Platform

This provides another control layer.

Reversible vs Irreversible Actions

Security architecture should distinguish between:

Lower-Risk Actions

Search documentation
Read alerts
Draft reports
Summarize logs

Higher-Risk Actions

Delete resources
Disable users
Send external messages
Modify firewall rules
Execute code

Higher-risk actions should receive stronger controls.

Layer 8 — Infrastructure

All AI systems depend on infrastructure.

This may include:

AWS
Azure
Google Cloud
Virtual machines
Containers
Kubernetes
Storage
Databases
GPU infrastructure
Serverless services

Traditional infrastructure security is essential.

Cloud AI Architecture

A simplified cloud architecture might be:

Internet / Corporate Network
          │
          ▼
      Load Balancer
          │
          ▼
     AI Application
          │
          ▼
       Private API
          │
          ▼
      AI Services
          │
     ┌────┼─────┐
     │    │     │
     ▼    ▼     ▼
   RAG   Model  Agent

Supporting services may include:

IAM
Secrets Manager
Object Storage
Databases
Monitoring
SIEM
KMS
Network Controls

Network Segmentation

Not every AI component needs direct internet exposure.

For example:

Internet
   │
   ▼
Public Frontend
   │
════╪════════ Trust Boundary
   ▼
Private Backend
   │
   ├── Private Database
   ├── Private Vector Store
   └── Private AI Service

Segmentation reduces attack surface.

Private Connectivity

Where supported, organizations may use private connectivity between:

Applications
Databases
AI services
Storage
Model endpoints

This can reduce exposure to public networks.

Layer 9 — Secrets Management

AI applications frequently need credentials.

Examples include:

Model API keys
Database passwords
Cloud credentials
Search API tokens
Service credentials

These should not be stored directly in code.

Bad:

source-code/
    app.py
        API_KEY="secret"

Better:

Application
    │
    ▼
Secrets Manager
    │
    ▼
Temporary / Controlled Credential

Secret Lifecycle

Secrets should be:

Created
  ↓
Stored Securely
  ↓
Accessed Only When Required
  ↓
Rotated
  ↓
Revoked

Secret management is part of AI Security architecture.

Layer 10 — Logging and Monitoring

An enterprise AI platform must be observable.

Without visibility, security teams may not know:

Who used the system
What data was accessed
What actions were performed
Which model was involved
Whether suspicious behavior occurred

Security Telemetry

Useful telemetry may include:

Authentication Event
+
User Identity
+
Application Request
+
Model Request Metadata
+
Retrieval Activity
+
Tool Invocation
+
Agent Action
+
Security Policy Decision
+
Application Response

The exact content of prompts and responses should be handled carefully because they may contain sensitive information.

Centralized Monitoring

A common architecture is:

AI Application Logs
        │
RAG Logs
        │
Agent Logs
        │
Cloud Audit Logs
        │
Identity Logs
        │
        ▼
Central Logging Platform
        │
        ▼
SIEM / Detection
        │
        ▼
Security Operations

This gives security teams a consolidated view.

Logging vs Privacy

Logging every prompt may appear useful for security.

However, prompts may contain:

Customer data
Credentials
Source code
Personal information
Internal business data

Therefore:

Logging must balance security visibility with privacy and data minimization.

Layer 11 — Governance

Enterprise AI also requires governance.

Governance defines:

Who can create AI applications
Which models are approved
Which data may be used
Which vendors are allowed
What testing is required
What risks require approval
What monitoring is mandatory

A mature environment may have:

Enterprise AI Policy
      ↓
Approved Models
      ↓
Approved Architecture Patterns
      ↓
Security Requirements
      ↓
Deployment Controls
      ↓
Continuous Monitoring

Governance creates consistency.

Shadow AI

One enterprise concern is Shadow AI.

This happens when employees or teams adopt AI services without formal approval.

Examples may include:

Employee
   ↓
Uploads Internal Data
   ↓
Unapproved External AI Service

Potential risks include:

Sensitive data exposure
Unknown data retention
Vendor risk
Compliance issues
Lack of monitoring

AI governance should therefore address approved and prohibited use.

Common Enterprise Deployment Patterns

Let’s look at several common patterns.

Pattern 1 — External LLM API

Enterprise Application
         │
         ▼
     External LLM API

Security considerations:

API credentials
Vendor trust
Data transfer
Data retention
Network security
Logging

Pattern 2 — Cloud-Managed AI

Enterprise Cloud Account
         │
         ▼
Managed AI Service

Security considerations:

Cloud IAM
Private networking
Resource policies
Service configuration
Logging

Pattern 3 — Self-Hosted Model

Enterprise Infrastructure
        │
        ▼
Model Runtime
        │
        ▼
Self-Hosted Model

Security considerations:

Host security
GPU security
Patch management
Model protection
Scaling
Availability

Pattern 4 — Enterprise RAG

AI Application
      │
      ├── LLM
      │
      └── RAG
            │
            └── Enterprise Knowledge

Security considerations:

Document authorization
Data classification
Vector database security
Prompt injection
Tenant isolation

Pattern 5 — Enterprise AI Agent

User
 │
 ▼
AI Agent
 │
 ├── LLM
 ├── Knowledge
 └── Tools
       │
       ├── Cloud
       ├── Email
       ├── Database
       └── Security Tools

Security considerations:

Tool permissions
Human approval
Prompt manipulation
Logging
Agent identity
Action boundaries

Trust Boundaries

One of the most important architecture concepts is the trust boundary.

A trust boundary exists whenever data moves between areas with different security assumptions.

Example

Internet User
    │
════╪════════════
    ▼
Enterprise Application

The application should not automatically trust input from the internet.

Another example:

Enterprise Application
    │
════╪════════════
    ▼
Third-Party LLM Provider

Data has now crossed an organizational boundary.

Typical AI Trust Boundaries

Common boundaries include:

User → Application

Application → LLM

Application → RAG

RAG → Document Store

Agent → Tool

Cloud → Third Party

Development → Production

Tenant A → Tenant B

These are important points for security controls.

Security Control Placement

Architecture security is not only about having controls.

Controls must be placed where they can actually enforce security.

Example: Document Authorization

Weak:

RAG retrieves confidential document
          ↓
LLM receives it
          ↓
System prompt says:
"Do not reveal confidential data"

Strong:

User Identity
      ↓
Authorization
      ↓
Confidential Document Blocked
      ↓
LLM Never Receives It

This is much stronger because unauthorized information never reaches the model.

Example: Agent Permissions

Weak:

Agent has admin access
      ↓
Prompt says:
"Only perform safe actions"

Strong:

Agent
   ↓
Restricted Tool
   ↓
Minimum Required Permissions

Security should be enforced technically where possible.

Defense in Depth Architecture

A secure AI platform may use:

Identity
   ↓
Authorization
   ↓
Network Security
   ↓
Application Security
   ↓
Input Controls
   ↓
RAG Authorization
   ↓
Model Controls
   ↓
Tool Restrictions
   ↓
Human Approval
   ↓
Logging
   ↓
Monitoring

No single control should carry the entire security responsibility.

Shared Responsibility

AI systems are usually built by multiple teams.

For example:

AI Platform Team
     │
Application Team
     │
Cloud Team
     │
Security Team
     │
Data Team
     │
Risk & Compliance
     │
Business Team

Security responsibilities must be clearly defined.

Example Responsibility Split

Application Team

Responsible for:

Application code
Prompt orchestration
API integration

Cloud Team

Responsible for:

Infrastructure
Networking
IAM foundations

Data Team

Responsible for:

Data pipelines
Data quality
Data classification

AI Team

Responsible for:

Models
Evaluation
AI workflows

Security Team

Responsible for:

Threat modeling
Security requirements
Security testing
Monitoring requirements

Governance Team

Responsible for:

Policy
Risk
Compliance

Strong AI security requires collaboration.

Development vs Production

AI environments should separate development and production.

Bad:

Developer
   ↓
Direct Production Access

Better:

Development
    ↓
Testing
    ↓
Security Validation
    ↓
Production Deployment

Production access should be controlled.

Model Promotion

Models may follow:

Development Model
      ↓
Evaluation
      ↓
Security Testing
      ↓
Approved
      ↓
Production Model

This is similar to secure software release processes.

AI Architecture Review Process

An AI Security Engineer may review architecture using a process like:

1. Understand Business Use Case
        ↓
2. Identify Users
        ↓
3. Map Components
        ↓
4. Map Data Flows
        ↓
5. Identify Trust Boundaries
        ↓
6. Identify Assets
        ↓
7. Review IAM
        ↓
8. Review Data Access
        ↓
9. Review RAG / Agents
        ↓
10. Review Infrastructure
        ↓
11. Review Logging
        ↓
12. Identify Security Gaps

This is the beginning of an enterprise security assessment.

Enterprise Scenario

Imagine a company builds an AI assistant for internal cloud engineers.

The assistant can:

Explain cloud architecture
Search internal runbooks
Read cloud alerts
Query cloud configuration
Draft remediation steps

Architecture:

Cloud Engineer
      │
      ▼
Enterprise SSO
      │
      ▼
AI Cloud Assistant
      │
      ├── LLM
      │
      ├── RAG
      │     └── Runbooks
      │
      └── AI Agent
             │
             ├── Read Cloud Config
             └── Read Security Alerts

Now review it.

Identity

Are engineers authenticated?
Is MFA enforced?
Are service identities managed?

Authorization

Can every engineer query every cloud account?
Are permissions inherited from the user?

RAG

Can users retrieve confidential security documents?
Are document permissions preserved?

Agent

Does the agent have read-only access?
Can it modify resources?

Secrets

Where are cloud and model credentials stored?

Network

Are internal services private?

Logging

Are tool calls recorded?
Are agent actions attributable to a user?

Monitoring

Can unusual activity trigger alerts?

This is the mindset of enterprise AI architecture review.

Common Architecture Risks

1 — Excessive Permissions

AI workloads or agents receive more privileges than required.

2 — Missing Authorization

Users authenticate correctly but downstream data access is not restricted.

3 — Shared Credentials

Multiple services use one powerful credential.

4 — Publicly Exposed Services

Databases, vector stores or model endpoints are exposed unnecessarily.

5 — Uncontrolled RAG

The AI can retrieve documents regardless of user permissions.

6 — Insecure Secrets

API keys are stored in code.

7 — Excessive Context

The model receives more sensitive information than needed.

8 — Unsafe Agent Tools

Agents can perform high-risk actions without approval.

9 — Missing Logging

Security teams cannot reconstruct AI activity.

10 — Third-Party Risk

Sensitive information is sent to external AI providers without sufficient controls.

AI Security Engineer Architecture Checklist

Business

What business problem does the AI solve?
How critical is the system?
What happens if it is compromised?

Users

Who can access it?
How are users authenticated?
Is MFA required?

Authorization

What can each user access?
Are permissions enforced downstream?
Are service accounts least privileged?

Models

Which models are used?
Where are they hosted?
Are model versions tracked?
Are external providers involved?

Data

What data does the system process?
How is it classified?
Is data minimized?
Does data leave the organization?

RAG

What sources are indexed?
Are document permissions enforced?
Is document provenance known?
Can malicious content enter the index?

Agents

What tools can agents use?
What permissions do those tools have?
Are high-risk actions approved by a human?

Infrastructure

Where does the system run?
Are components privately accessible where possible?
Are network boundaries defined?

Secrets

Where are credentials stored?
Are they rotated?
Are they committed to source code?

Logging

Are authentication events logged?
Are RAG retrievals logged?
Are tool calls logged?
Can agent actions be traced?

Monitoring

Are suspicious patterns detected?
Are alerts integrated with security operations?
Can incidents be investigated?

Interview Perspective

You may be asked:

What would you review first in an enterprise AI architecture?

A strong answer is:

I would first understand the business use case and map the architecture, including users, applications, models, data sources, RAG components, agents, infrastructure and external dependencies. I would then identify trust boundaries, data flows and identities before reviewing security controls such as IAM, authorization, network isolation, secrets management and monitoring.

Another common question is:

What is one of the biggest security risks in enterprise RAG architecture?

A strong answer is:

A major risk is failing to preserve authorization during retrieval. If a user is authenticated but the RAG layer searches all enterprise documents without enforcing document-level permissions, the LLM may receive information the user was never authorized to access.

Another question may be:

Why are AI agents more security-sensitive than standard LLM applications?

A strong answer is:

Agents can interact with tools and perform actions. This means a manipulated model response may translate into a real operation such as querying a database, changing a cloud resource or sending a message. Agent identities, tool permissions, least privilege, human approval and audit logging therefore become critical controls.

Key Takeaways

Enterprise AI is an architecture, not just a model.

A complete environment may include:

Users
+
Identity
+
Applications
+
Orchestration
+
Models
+
RAG
+
Vector Databases
+
Enterprise Data
+
Agents
+
Tools
+
Cloud Infrastructure
+
Secrets
+
Logging
+
Governance

Security depends on understanding how these components interact.

The most important architecture principles are:

Understand the business use case.
Map the complete architecture.
Preserve identity through the system.
Enforce authorization before sensitive data reaches the model.
Apply least privilege to AI agents.
Minimize model context.
Protect secrets.
Reduce unnecessary network exposure.
Log important security events.
Use defense in depth.
Clearly define responsibilities.

The most important mindset is:

Do not ask only whether the model is secure. Ask whether the complete AI system is securely designed.

What’s Next?

➡️ 05 — AI Security Attack Surface

You now understand how enterprise AI systems are structured.

The next step is learning how attackers look at that architecture.

In the next lesson, you will learn how to systematically identify the AI attack surface, including:

Users and identities
AI applications
APIs
Prompts
Models
RAG pipelines
Vector databases
AI agents
Tools and plugins
Data sources
Cloud infrastructure
Model repositories
CI/CD pipelines
Third-party AI services
Logging and management interfaces

You will also learn how to map:

Asset
   ↓
Entry Point
   ↓
Trust Boundary
   ↓
Attack Path
   ↓
Potential Impact

This will prepare you for later lessons on the AI threat landscape, OWASP risks, MITRE ATLAS and AI threat modeling.

➡️ Next: 05 — AI Security Attack Surface