05 Responsible and Ethical AI Security Testing

AI Security Engineers frequently work with technologies that can expose sensitive data, automate actions and interact with critical enterprise systems.

Because of that, security testing must be performed within clearly defined professional and ethical boundaries.

The ability to test a system does not automatically mean you are authorized to test it.

A professional AI Security Engineer must understand:

• Authorization
• Scope
• Rules of engagement
• Data handling
• Testing limitations
• Evidence protection
• Responsible disclosure
• Legal and contractual requirements
• Ethical decision-making

This lesson establishes the operating boundaries for all future AI Security labs, assessments and red team activities.

Learning Objectives

By the end of this lesson, you should be able to:

• Understand why authorization is mandatory for security testing.
• Define testing scope and boundaries.
• Understand rules of engagement.
• Recognize the difference between authorized testing and unauthorized activity.
• Handle AI security testing data responsibly.
• Understand special risks introduced by AI systems.
• Perform safe AI red team exercises.
• Understand responsible vulnerability disclosure.
• Protect collected evidence.
• Know when to stop, escalate or report during a security assessment.
• Apply professional ethics throughout the AI security lifecycle.

The Most Important Rule

Before performing any security testing, ask:

Do I have explicit authorization to test this system?

If the answer is unclear, do not proceed.

Security testing should only be performed against:

• Systems you own
• Dedicated lab environments
• Intentionally vulnerable training platforms
• Authorized sandbox environments
• Systems covered by an approved security assessment
• Systems where written permission has been provided

Never assume authorization.

Publicly Accessible Does Not Mean Authorized

A system may be publicly accessible on the internet.

That does not mean you are allowed to test it.

For example:

Public AI Chatbot
        ≠
Authorized Security Target

Professional security testing requires permission.

Why AI Security Testing Requires Extra Care

Traditional security testing may already involve:

• Sensitive systems

• Credentials

• Production services

• Customer information

• Business-critical infrastructure

AI systems introduce additional risks.

An AI system may be connected to:

Enterprise Data
Customer Information
Source Code
Email
Cloud Storage
Internal Databases
Ticketing Systems
Business Applications
Administrative Tools
External APIs

AI agents may also have the ability to perform actions.

For example:

User Prompt
     │
     ▼
AI Agent
     │
     ├── Read Email
     ├── Query Database
     ├── Create Ticket
     ├── Access Cloud Storage
     └── Execute API Action

A poorly controlled test could therefore trigger unintended business actions.

This makes strict testing boundaries especially important.

Authorization

Authorization is formal permission to perform security testing.

Authorization should clearly define:

• Who is allowed to perform the testing

• Which systems can be tested

• Which activities are permitted

• When testing may occur

• What testing methods are prohibited

• Who should be contacted during an incident

• How findings should be reported

Whenever possible, authorization should be documented.

Example Authorization Questions

Before beginning an assessment, ask:

Who approved the assessment?

What systems are in scope?

What systems are out of scope?

What data may be accessed?

Can production systems be tested?

Can automated tools be used?

Can accounts be created?

Can prompts attempt to bypass controls?

Can AI agents execute actions?

Are denial-of-service tests allowed?

Who should be contacted if something goes wrong?

These questions help prevent ambiguity.

Understanding Scope

Scope defines the boundaries of a security assessment.

An AI security scope may include:

In Scope
──────────────
AI Web Application
LLM API
RAG System
Vector Database
Test Data
AI Agent
Test Cloud Account

and exclude:

Out of Scope
──────────────
Production Database
Corporate Email
Customer Systems
Third-Party SaaS
Production Cloud Account
Employee Devices

Testing must remain within the agreed scope.

Scope Is More Than a List of Systems

Scope may also define:

• Allowed test techniques

• Allowed accounts

• Allowed IP addresses

• Testing time windows

• Data handling restrictions

• Request rate limits

• Social engineering restrictions

• Agent execution restrictions

• Production safety limits

For example:

Target:
Internal AI Assistant

Allowed:
Prompt injection testing
Authorization testing
RAG security testing

Not Allowed:
Denial-of-service testing
Production data extraction
Employee impersonation
External system testing

This is a much stronger scope definition.

Rules of Engagement

Rules of Engagement define how testing should be performed.

They may include:

• Start and end dates

• Testing hours

• Approved testers

• Approved tools

• Communication channels

• Emergency contacts

• Data handling requirements

• Stop conditions

• Evidence storage requirements

• Reporting expectations

Think of the relationship as:

Authorization
     ↓
Scope
     ↓
Rules of Engagement
     ↓
Security Testing

All three are important.

Example Rules of Engagement

A simple example may include:

Assessment:
Enterprise AI Assistant Security Review

Testing Window:
09:00–18:00

Environment:
Dedicated staging environment

Allowed:
Prompt security testing
RAG manipulation testing
Access-control validation
Agent permission review

Restricted:
No destructive actions
No denial-of-service activity
No employee social engineering
No external infrastructure testing

Stop Condition:
Unexpected production access or sensitive data exposure

Escalation:
Contact security lead immediately

This provides clear operational boundaries.

Safe AI Red Teaming

AI red teaming involves deliberately testing how an AI system behaves when faced with adversarial input or scenarios.

Possible areas include:

• Prompt injection

• Jailbreak resistance

• Sensitive data exposure

• RAG manipulation

• AI agent misuse

• Unauthorized tool invocation

• Improper access control

• Unsafe output handling

The objective is not simply to make the AI behave incorrectly.

The objective is to determine:

Can this weakness create a meaningful security impact?

AI Red Teaming Example

Suppose an internal AI assistant uses RAG to answer employee questions.

Authorized testing might investigate:

Can a user access documents they are not authorized to see?

Can malicious content inside a document influence the model?

Can prompts cause the model to reveal hidden instructions?

Can an AI agent perform actions outside the user's permissions?

Can the system expose sensitive metadata?

These are legitimate security questions when tested within an approved environment.

Avoid Unnecessary Impact

Professional security testing should produce the minimum impact required to validate a finding.

For example:

If you can prove that unauthorized access exists by retrieving:

TEST-CONFIDENTIAL-RECORD

there is usually no reason to extract hundreds of real records.

The principle is:

Prove the security issue without creating unnecessary harm.

Use Synthetic Data Whenever Possible

AI security testing often involves data.

Whenever possible, use:

• Test identities

• Synthetic customer information

• Dummy API keys

• Sample documents

• Test databases

• Fake credentials

• Controlled datasets

Example:

Customer Name: Test User
Account Number: TEST-0001
Classification: Confidential-Test

This allows security behavior to be validated without exposing real information.

Sensitive Data Handling

During legitimate security testing, you may unexpectedly encounter sensitive information.

Examples include:

• Personal information

• Authentication tokens

• API keys

• Internal documents

• Source code

• Business information

• Customer records

• Security configurations

If this happens:

1. Stop unnecessary access.

2. Do not collect additional information without need.

3. Preserve only required evidence.

4. Protect the evidence.

5. Notify the appropriate contact.

6. Follow the organization’s incident or assessment procedure.

Do not copy sensitive data into personal notes or public repositories.

Minimize Data Collection

Security professionals should collect only the evidence required to support a finding.

Bad practice:

Download complete database

when the issue can be proven with:

One controlled test record

Better security testing minimizes exposure.

Evidence Handling

Security evidence can itself become sensitive.

Examples include:

• Screenshots

• Logs

• API responses

• Tokens

• Prompt transcripts

• Configuration files

• Architecture diagrams

• Vulnerability details

Store evidence in approved locations.

Avoid:

Desktop screenshots
Personal cloud storage
Public Git repositories
Messaging apps
Unencrypted USB devices

Use the organization’s approved evidence-handling process.

Redact Sensitive Information

Before sharing evidence, remove information that is not required.

For example:

API Key:
sk-live-abcdefgh123456

should become:

API Key:
sk-live-**************

Similarly:

Customer:
John Smith

may become:

Customer:
[REDACTED]

when identity is irrelevant to the security finding.

Protect Credentials During Testing

Security testing often requires temporary credentials.

Apply the following principles:

• Use dedicated test accounts.

• Use least privilege.

• Avoid using personal accounts.

• Rotate temporary credentials after testing.

• Do not store secrets in scripts.

• Do not commit secrets to Git.

• Avoid sharing credentials through unsecured channels.

Prefer:

Security Test Account
        ↓
Minimum Permissions
        ↓
Temporary Usage
        ↓
Credential Rotation

AI Agent Testing Requires Special Controls

AI agents introduce additional risk because they can interact with external systems.

For example:

Prompt
   ↓
Agent
   ↓
Tool
   ↓
Enterprise System

A successful manipulation might cause actions rather than just generate text.

Examples include:

• Sending email

• Creating tickets

• Modifying files

• Querying databases

• Triggering cloud operations

• Calling external APIs

When testing AI agents, use controlled tools whenever possible.

For example:

Production Email System
        ↓
Replace With
        ↓
Test Mailbox

or:

Production Database
        ↓
Replace With
        ↓
Synthetic Lab Database

This reduces unintended impact.

Define Agent Action Boundaries

Before testing an AI agent, determine:

What tools can the agent access?

What permissions does each tool have?

What data can the agent read?

What actions can it perform?

Can actions be reversed?

Is human approval required?

Are actions logged?

These questions should be answered before adversarial testing begins.

Human-in-the-Loop Controls

For higher-risk AI actions, organizations may require human approval.

Example:

AI Recommendation
      ↓
Human Review
      ↓
Approved?
   ┌──┴──┐
  Yes    No
   │      │
Execute  Stop

Testing should respect these workflows.

A red team should not intentionally bypass safety mechanisms outside the authorized testing scope.

Testing Third-Party AI Services

Organizations frequently use external AI services.

Examples include:

• Hosted LLM APIs

• SaaS AI assistants

• External AI agents

• AI coding assistants

• Cloud AI platforms

Before testing a third-party service, confirm:

• The organization’s authorization

• The provider’s terms

• Permitted testing methods

• Rate limits

• Data handling requirements

• Vulnerability disclosure process

Your organization may own the application, but it may not own the underlying AI platform.

This distinction matters.

Avoid Testing Other Users

Do not attempt to access:

• Other customers’ conversations

• Other organizations’ data

• Other users’ accounts

• Shared infrastructure belonging to unrelated parties

If you discover evidence suggesting cross-tenant access, stop and escalate according to the assessment procedure.

The goal is to validate the risk without expanding the exposure.

Denial-of-Service Testing

AI systems can be expensive and resource intensive.

Examples of potentially disruptive testing include:

• Very high request volume

• Extremely large prompts

• Resource exhaustion

• Repeated model invocation

• Automated API flooding

• Excessive tool execution

These activities may:

• Increase cloud costs

• Exhaust quotas

• Affect availability

• Trigger service restrictions

Do not perform availability-impacting testing unless it is explicitly authorized and controlled.

Cost Is Also a Security Consideration

AI systems often charge based on:

• API requests

• Tokens

• Compute time

• GPU usage

• Storage

• Data processing

A poorly designed test could generate unexpected costs.

Therefore security testing should also consider:

Security Impact
+
Operational Impact
+
Financial Impact

Prompt Injection Testing

Prompt injection testing should be performed in controlled environments.

The objective is to determine whether untrusted input can influence system behavior.

For example:

Untrusted Document
        ↓
RAG Pipeline
        ↓
LLM
        ↓
Unexpected Behavior

A responsible assessment asks:

• What trust boundary was crossed?

• What control failed?

• What data became exposed?

• What action became possible?

• How should the architecture be improved?

The objective is not simply to create unusual model output.

Jailbreak Testing

Jailbreak testing evaluates whether expected restrictions can be bypassed.

When performing authorized tests:

• Use defined test cases.

• Document the expected behavior.

• Record actual behavior.

• Avoid unnecessary harmful content.

• Focus on security impact.

• Provide reproducible evidence.

A professional report should explain:

What security control failed and what business risk results from that failure?

RAG Security Testing

RAG systems deserve particular attention because they connect models to enterprise information.

A simplified architecture:

User
 │
 ▼
AI Application
 │
 ▼
RAG
 │
 ├── Vector Database
 ├── Documents
 └── Enterprise Data

Security testing may evaluate:

• Access controls

• Document authorization

• Data segregation

• Malicious document ingestion

• Prompt injection through retrieved content

• Metadata exposure

• Sensitive information leakage

Testing should use approved documents and datasets.

Model and Dataset Testing

Models and datasets may also have contractual or licensing restrictions.

Before testing:

• Confirm usage rights.

• Understand data classification.

• Confirm whether model files can be copied.

• Confirm whether datasets can leave the environment.

• Understand third-party licensing restrictions.

Security engineers must respect both security and data governance requirements.

AI Supply Chain Testing

AI systems rely on many components:

Model
+
Dataset
+
Python Packages
+
Containers
+
Libraries
+
APIs
+
Repositories

Security assessments may identify issues involving:

• Untrusted models

• Malicious packages

• Vulnerable dependencies

• Compromised repositories

• Unsafe container images

When testing supply-chain scenarios, avoid introducing malicious software into uncontrolled environments.

Use isolated labs.

Safe Malware-Related Testing

Some AI security research may involve malware analysis or detection.

Use:

• Isolated environments

• Non-production systems

• Dedicated sandboxes

• Controlled samples

• Approved procedures

Never execute unknown malicious code directly on your normal workstation.

Stop Conditions

Professional testers should know when to stop.

Possible stop conditions include:

• Unexpected production access

• Sensitive data belonging to real users

• Service instability

• Unexpected administrative access

• Cross-tenant data exposure

• Destructive actions

• System outage

• Significant cost increase

• Evidence of an active real-world compromise

If a stop condition occurs:

Stop Testing
     ↓
Preserve Evidence
     ↓
Notify Contact
     ↓
Follow Escalation Procedure

Do not continue experimenting.

Escalation

Some findings require immediate escalation.

Examples include:

• Active credential compromise

• Exposed production secrets

• Large-scale data exposure

• AI agent access to critical systems

• Remote administrative control

• Active attacker activity

• Security controls completely bypassed

Your rules of engagement should identify the appropriate escalation contact.

Responsible Vulnerability Disclosure

Sometimes you may discover a vulnerability outside a formal engagement.

Responsible disclosure means reporting the vulnerability through appropriate channels rather than exploiting or publishing it irresponsibly.

A typical process is:

Discover Potential Issue
        ↓
Verify Carefully
        ↓
Minimize Testing
        ↓
Collect Limited Evidence
        ↓
Identify Disclosure Channel
        ↓
Report Privately
        ↓
Allow Remediation
        ↓
Coordinate Publication If Appropriate

Do not publicly disclose sensitive technical details before the affected organization has had a reasonable opportunity to investigate and remediate.

Bug Bounty Programs

Some organizations explicitly authorize certain security testing through bug bounty or vulnerability disclosure programs.

Always read the program rules carefully.

Programs typically define:

• In-scope systems

• Out-of-scope systems

• Permitted techniques

• Prohibited activities

• Reporting requirements

• Reward eligibility

• Safe-harbor conditions

Do not assume one organization’s bug bounty rules apply to another.

Ethical AI Security Research

Ethical AI security research should aim to:

• Improve safety

• Identify weaknesses responsibly

• Help organizations reduce risk

• Protect users

• Improve engineering practices

• Advance defensive understanding

Avoid research designed primarily to:

• Harm users

• Expose sensitive information

• Disrupt services

• Bypass legitimate controls for personal gain

• Create unauthorized access

Your technical capability should be paired with professional judgment.

Understand Business Context

Not every technically interesting behavior represents a major security risk.

For example:

Model Produces Unexpected Response

may be less important than:

Model Can Trigger Unauthorized Financial Action

Security engineers should evaluate:

• Exploitability

• Required access

• Data sensitivity

• Business impact

• Existing controls

• Detection capability

• Recovery complexity

This helps organizations prioritize remediation.

Separate Finding From Impact

When documenting a security issue, clearly distinguish:

Finding

What weakness exists?

Exploitation Scenario

How could it be abused?

Impact

What could happen to the organization?

Example:

Finding:
AI agent can invoke an administrative tool without
secondary authorization.

Attack Scenario:
A manipulated prompt causes the agent to invoke the tool.

Impact:
Unauthorized modification of enterprise resources may occur.

This is much more useful than:

“The AI can be hacked.”

Document Evidence Objectively

Avoid exaggerated language.

Instead of:

CRITICAL AI SYSTEM COMPLETELY BROKEN

write:

The AI agent was able to invoke the administrative test
function without the expected approval step.

Then provide:

• Evidence

• Conditions

• Impact

• Recommended control

Professional reports should be factual and reproducible.

Retesting

After remediation, security teams may perform retesting.

The workflow becomes:

Finding
   ↓
Remediation
   ↓
Retest
   ↓
Control Works?
   ├── Yes → Close
   └── No  → Reopen

Retesting confirms whether the underlying weakness was actually resolved.

AI Security Testing Lifecycle

A professional assessment often follows:

1. Authorization
        ↓
2. Scope
        ↓
3. Rules of Engagement
        ↓
4. Architecture Review
        ↓
5. Threat Modeling
        ↓
6. Test Planning
        ↓
7. Security Testing
        ↓
8. Evidence Collection
        ↓
9. Risk Analysis
        ↓
10. Reporting
        ↓
11. Remediation
        ↓
12. Retesting

This lifecycle will appear repeatedly throughout the learning path.

Professional Ethics

Professional security ethics can be summarized with several principles.

Respect Authorization

Test only what you are allowed to test.

Minimize Harm

Use the least invasive method required.

Protect Data

Treat discovered information responsibly.

Maintain Confidentiality

Do not disclose sensitive assessment information.

Report Truthfully

Do not exaggerate findings or hide mistakes.

Preserve Evidence

Maintain accurate records.

Escalate When Required

Know when a finding requires immediate attention.

Respect Users

Security research should protect people, not exploit them.

Your Decision Framework

Before performing a questionable action, ask:

Am I authorized?

Is this inside scope?

Is this necessary?

Could this cause harm?

Could this expose real data?

Can I prove the issue another way?

Is this action reversible?

Do I know who to contact if something goes wrong?

If any answer creates uncertainty, stop and verify before proceeding.

Example: Safe vs Unsafe Testing

Scenario

You are assessing an AI assistant that can search employee files.

Safer Approach

Create test documents:

TEST-CONFIDENTIAL-HR-DOCUMENT

Assign them to test identities.

Then validate whether access controls work.

Unsafe Approach

Attempt to retrieve actual employee medical, salary or HR information.

The first approach proves the security control without exposing real employees.

That is the professional approach.

Example: AI Agent Testing

Scenario

An AI agent can send email.

Safer Approach

Configure the agent to access:

security-lab@example.test

and validate whether unauthorized instructions can trigger a message.

Unsafe Approach

Attempt to send messages to real customers or employees.

Again:

Validate the security issue with the minimum necessary impact.

Security Testing Is Not About Showing Off

The objective of security testing is not to demonstrate how aggressively you can attack a system.

The objective is to help answer:

What is vulnerable?

Why is it vulnerable?

What is the risk?

How can we prove it safely?

How should it be fixed?

How do we validate the fix?

That is professional security engineering.

Using AI During Security Testing

AI can assist security professionals with activities such as:

• Understanding documentation

• Reviewing logs

• Organizing test cases

• Explaining code

• Drafting findings

• Summarizing evidence

• Generating test data

• Brainstorming threat scenarios

However, AI-generated testing suggestions still require professional judgment.

Before following any suggestion, ask:

Is this authorized?

Is it in scope?

Could it cause disruption?

Could it expose sensitive data?

Do I understand what the command or action does?

Never blindly execute AI-generated commands.

Verify Before You Execute

Suppose an AI assistant generates:

some-security-command --target example

Before executing it:

1. Understand the tool.

2. Understand the flags.

3. Confirm the target.

4. Confirm authorization.

5. Understand the expected impact.

6. Determine how to stop the action.

7. Decide what evidence you need.

This is an essential habit for modern security professionals.

Your AI Security Testing Checklist

Before every security assessment:

Authorization

• I have permission to perform the assessment.

• The authorization is documented where appropriate.

Scope

• I know what systems are in scope.

• I know what systems are out of scope.

• I know what techniques are permitted.

Environment

• I understand whether the target is lab, staging or production.

• I know the expected business impact of testing.

Data

• I understand the data classification.

• I know how evidence should be stored.

• I will minimize sensitive data collection.

Testing

• I understand the activity before performing it.

• I will use the minimum necessary impact.

• I know the stop conditions.

Communication

• I know who to contact during an emergency.

• I know how findings should be reported.

Closure

• Temporary credentials will be removed or rotated.

• Test resources will be cleaned up.

• Evidence will be stored appropriately.

• Findings will be documented.

The Professional AI Security Engineer Mindset

Throughout your career, remember:

Technical Capability
        +
Authorization
        +
Professional Judgment
        +
Responsible Testing
        +
Clear Communication
        =
Trusted Security Engineer

Technical skill alone is not enough.

Organizations trust security engineers with systems, data and knowledge that can create significant impact.

Professional judgment is therefore one of your most important skills.

What You Should Have Learned

You should now understand that responsible AI security testing requires more than technical ability.

Before testing, you should establish:

Authorization
     ↓
Scope
     ↓
Rules of Engagement
     ↓
Testing Boundaries

During testing:

Minimize Impact
     ↓
Protect Data
     ↓
Collect Evidence
     ↓
Know When to Stop

After testing:

Analyze Risk
     ↓
Report Findings
     ↓
Support Remediation
     ↓
Retest

These principles apply throughout the AI Security Engineer Learning Path.

Start Here Complete

You have now completed the Start Here section of the AI Security Engineer Learning Path.

You have covered:

• What an AI Security Engineer does

• How to use this learning path

• Your AI Security Engineer career roadmap

• How to build your AI Security Lab

• Responsible and ethical AI security testing

You now have the context required to begin the technical learning journey.

What’s Next?

➡️ Module 01 — AI Security Foundations

Your technical journey begins by understanding the systems you will eventually protect.

Before attacking, securing or monitoring AI systems, you must understand:

• What Artificial Intelligence actually is

• How Machine Learning works

• How Generative AI works

• How Large Language Models work

• How modern enterprise AI applications are built

• Where AI security risks appear

• How AI changes the cybersecurity attack surface

You will progressively move from:

Understanding AI
       ↓
Understanding Enterprise AI
       ↓
Understanding AI Attack Surfaces
       ↓
Understanding AI Security

The first lesson begins with the foundation for everything that follows.

➡️ Next: Module 01 — AI Security Foundations