Runbook 03 Kubernetes Incident Response

Kubernetes incident response begins when suspicious activity moves from:

Something Happened

to:

We Must Contain the Threat,
Remove the Cause,
and Restore the Environment Safely.

In the previous runbook, you learned how to investigate Kubernetes incidents and reconstruct what happened.

Now the focus changes.

FORENSICS
What happened?

        ↓

INCIDENT RESPONSE
What do we do now?

A Kubernetes incident can evolve quickly because workloads are dynamic, identities are interconnected, containers are ephemeral, and compromised resources may automatically be recreated by controllers.

A professional response therefore requires:

Validate
   ↓
Scope
   ↓
Preserve
   ↓
Contain
   ↓
Eradicate
   ↓
Recover
   ↓
Monitor
   ↓
Improve

Runbook Information

Type: Kubernetes Incident Response Runbook

Difficulty: Advanced

Primary Audience:

SOC Analysts

Incident Responders

Kubernetes Security Engineers

Cloud Security Engineers

Platform Security Engineers

DevSecOps Engineers

SRE Teams

Security Consultants

Security Architects

Primary Skills:

Incident Validation

Incident Classification

Evidence Preservation

Workload Containment

Identity Containment

Network Isolation

Credential Rotation

Node Response

Malicious Resource Removal

Secure Recovery

Post-Incident Monitoring

Root Cause Remediation

Runbook Mission

Your security operations team receives an alert indicating suspicious activity in a production Kubernetes cluster.

Initial investigation suggests:

Application Compromise
        ↓
Unexpected Container Process
        ↓
Interactive Shell Activity
        ↓
ServiceAccount Usage
        ↓
Suspicious Kubernetes API Calls
        ↓
Unexpected Network Communication

Your mission is to:

Confirm the Incident

Determine Severity

Preserve Critical Evidence

Stop Active Threat Activity

Reduce Attacker Access

Remove Malicious Changes

Rotate Exposed Credentials

Restore Trusted Workloads

Validate Security

Monitor for Recurrence

Document Lessons Learned

01 — Understand Kubernetes Incident Response

Traditional incident response often focuses on:

User

Endpoint

Server

Network

Kubernetes introduces additional layers:

Cluster
│
├── Namespace
├── Workload
├── Pod
├── Container
├── ServiceAccount
├── RBAC
├── Secret
├── Network
├── Node
├── Container Image
├── Admission Policy
└── Cloud Identity

Therefore, containment cannot focus only on:

The Suspicious Container

You must determine whether the incident has expanded across these layers.

02 — Incident Response Lifecycle

Use the following lifecycle:

Preparation
     ↓
Detection
     ↓
Validation
     ↓
Scoping
     ↓
Evidence Preservation
     ↓
Containment
     ↓
Eradication
     ↓
Recovery
     ↓
Monitoring
     ↓
Lessons Learned

03 — Establish Incident Authority

Before making production changes, establish:

Incident Commander

Technical Lead

Kubernetes / Platform Owner

Security Lead

Application Owner

Cloud Team

Communications Owner

Business Owner

Document:

Who Can Isolate Workloads?

Who Can Disable Identities?

Who Can Rotate Credentials?

Who Can Quarantine Nodes?

Who Can Approve Downtime?

Who Can Communicate Externally?

Important Principle

During a serious incident:

Technical Capability
      ≠
Authorization

The responder may technically be able to delete a production namespace.

That does not mean they should do so without appropriate incident authority.

04 — Open the Incident Record

Create:

Incident ID:

Incident Title:

Date:

Detection Time:

Environment:

Cluster:

Namespace:

Affected Application:

Incident Commander:

Technical Lead:

Initial Severity:

Current Status:

Example

Incident ID:
K8S-IR-001

Incident:
Suspicious Runtime Activity

Environment:
Production

Cluster:
Production Kubernetes Cluster

Initial Alert:
Interactive shell detected
inside application workload.

Initial Severity:
High

05 — Record the Initial Alert

Capture:

Timestamp

Alert Source

Detection Rule

Cluster

Namespace

Pod

Container

Node

Image

Process

ServiceAccount

Source

Destination

Severity

Preserve the original alert.

Do not rely only on screenshots or memory.

06 — Validate the Alert

Determine whether the activity represents:

Expected Administration

Developer Troubleshooting

Automated Platform Activity

Security Testing

Misconfiguration

Actual Security Incident

Validation Questions

Ask:

Was the activity authorized?

Was there an approved change?

Was there a support ticket?

Does the application normally perform this action?

Was the identity expected?

Was the source expected?

Does the timing make sense?

07 — Classify the Event

Use:

False Positive

Benign Activity

Policy Violation

Suspicious Activity

Confirmed Incident

If confirmed:

Activate Incident Response

08 — Assign Severity

Use the organization’s approved severity model.

Consider:

Production Impact

Internet Exposure

Privilege Obtained

Sensitive Data

ServiceAccount Permissions

Cloud Permissions

Node Access

Persistence

Lateral Movement

Business Criticality

Example Severity Model

LOW
Limited suspicious activity
with minimal impact.

MEDIUM
Confirmed compromise with
limited scope.

HIGH
Privileged access, sensitive
resources, or lateral movement.

CRITICAL
Cluster-wide, node-level,
cloud-level, or major
business/data impact.

09 — Identify the Initial Scope

Start with:

Cluster

Namespace

Pod

Container

Deployment

ServiceAccount

Node

Image

Build:

Resource	Value
Cluster	TBD
Namespace	TBD
Pod	TBD
Container	TBD
Controller	TBD
ServiceAccount	TBD
Node	TBD
Image	TBD

10 — Verify Your Kubernetes Context

Before executing any response action:

kubectl config current-context

Then:

kubectl cluster-info

Record:

Cluster:

Context:

Response Identity:

This simple step helps prevent an incident responder from modifying the wrong cluster.

11 — Preserve Critical Evidence

Before destructive containment, preserve critical evidence where incident urgency permits.

Collect:

Pod YAML

Controller YAML

Events

Container Logs

Previous Logs

ServiceAccount

RBAC

NetworkPolicy

Image ID

Runtime Alerts

Audit Logs

12 — Preserve Pod State

kubectl get pod <pod-name> -n <namespace> -o yaml

Then:

kubectl describe pod <pod-name> -n <namespace>

Record:

Node

Pod IP

Image

Image ID

ServiceAccount

Volumes

Security Context

Restart History

13 — Preserve Logs

Current logs:

kubectl logs <pod-name> -n <namespace>

For multiple containers:

kubectl logs <pod-name> -n <namespace> -c <container-name>

Previous container logs:

kubectl logs <pod-name> -n <namespace> --previous

if available.

14 — Preserve Events

kubectl get events -n <namespace> --sort-by=.metadata.creationTimestamp

Events may disappear relatively quickly.

Capture them early.

15 — Preserve Controller Configuration

Determine ownership:

kubectl describe pod <pod-name> -n <namespace>

If controlled by a Deployment:

kubectl get deployment <deployment-name> -n <namespace> -o yaml

Remember:

Delete Pod
     ↓
Deployment
     ↓
Creates New Pod

Deleting only the compromised Pod may not remove the underlying malicious configuration.

16 — Containment Strategy

Containment should answer:

How Can We Stop the Threat
While Minimizing Business Impact
and Preserving Evidence?

Possible containment layers:

Workload

Network

Identity

Credentials

RBAC

Image

Node

Cloud

17 — Short-Term vs Long-Term Containment

Short-Term

Immediate actions designed to stop active threat activity.

Examples:

Isolate Workload

Disable Identity

Block Network Communication

Scale Workload Down

Quarantine Node

Long-Term

Actions that allow safer continued operations while remediation proceeds.

Examples:

Deploy Clean Image

Apply Restricted RBAC

Implement NetworkPolicy

Rotate Credentials

Enforce Admission Policies

18 — Choose the Least Destructive Effective Action

Possible response:

Threat Active?
      ↓
Can Network Isolation Stop It?
      ↓
Yes
      ↓
Isolate Network First

versus immediately:

Delete Everything

The correct action depends on severity and business risk.

19 — Workload Network Isolation

One containment option is restricting communication around the compromised workload.

Conceptually:

Compromised Pod

     X ← Ingress

     X → Egress

A restrictive NetworkPolicy may help quarantine the workload where the cluster networking implementation enforces NetworkPolicy.

Example Isolation Concept

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: quarantine
  namespace: <namespace>
spec:
  podSelector:
    matchLabels:
      incident-quarantine: "true"
  policyTypes:
    - Ingress
    - Egress

This policy selects labeled Pods and specifies no allowed ingress or egress rules.

Important

Before relying on this technique, validate:

CNI Supports NetworkPolicy

Correct Pod Selected

Ingress Restricted

Egress Restricted

20 — Label the Target Carefully

In an authorized response:

kubectl label pod <pod-name> -n <namespace> incident-quarantine=true

Immediately validate the selected resource.

Warning

Do not accidentally apply broad quarantine controls to unrelated production workloads.

21 — Scale a Compromised Deployment

If the workload must be stopped and the incident commander approves:

kubectl scale deployment <deployment-name> -n <namespace> --replicas=0

Validate:

kubectl get pods -n <namespace>

Security Benefit

This prevents the Deployment from immediately recreating the compromised application Pod while investigation/remediation continues.

Business Impact

Replicas = 0
      ↓
Application May Become Unavailable

Coordinate with application and business owners.

22 — Pod Deletion

Deleting a compromised Pod may be appropriate after evidence preservation.

kubectl delete pod <pod-name> -n <namespace>

But understand:

Pod Deleted
     ↓
Controller
     ↓
Replacement Pod

If the underlying Deployment is compromised, the replacement may also be compromised.

23 — Identify the Underlying Source

Determine whether the compromise exists in:

Running Pod Only

Deployment

ReplicaSet

DaemonSet

StatefulSet

Job

CronJob

Helm Release

Git Repository

CI/CD Pipeline

Container Image

Containment must address the source.

24 — Identity Containment

If a compromised workload identity is being abused, determine:

Which ServiceAccount?

Which RBAC Permissions?

Which Workloads Use It?

Which API Actions Occurred?

Identity Response Model

Compromised Identity
      ↓
Identify Usage
      ↓
Restrict Permissions
      ↓
Revoke/Rotate Credentials
      ↓
Validate

25 — Review ServiceAccount

kubectl get serviceaccount <service-account> -n <namespace> -o yaml

Identify workloads using the same ServiceAccount before making changes.

Important

A ServiceAccount may be shared.

Disabling or changing its access may affect multiple applications.

26 — Review Effective Permissions

Where authorized:

kubectl auth can-i --list --as=system:serviceaccount:<namespace>:<service-account>

Focus on:

Secrets

Pods

Deployments

Jobs

DaemonSets

RBAC

ServiceAccounts

Cluster Resources

27 — Remove Unauthorized RoleBindings

If investigation confirms a malicious or unauthorized RoleBinding:

kubectl get rolebinding <binding-name> -n <namespace> -o yaml

Preserve evidence first.

After approval:

kubectl delete rolebinding <binding-name> -n <namespace>

Validate effective permissions afterward.

28 — Remove Unauthorized ClusterRoleBindings

Cluster-wide bindings require particular care.

Preserve:

kubectl get clusterrolebinding <binding-name> -o yaml

After confirming it is unauthorized and obtaining approval:

kubectl delete clusterrolebinding <binding-name>

Critical Warning

Incorrect removal of legitimate cluster-wide RBAC can disrupt:

Controllers

Security Agents

Platform Components

Automation

Always validate ownership first.

29 — Credential Exposure Assessment

Create:

Credential	Exposure	Action
ServiceAccount identity	Confirmed/Potential	Review/Revoke
Database password	Confirmed/Potential	Rotate
API key	Confirmed/Potential	Rotate
Cloud identity	Confirmed/Potential	Revoke/Rotate
Registry credential	Confirmed/Potential	Rotate
TLS/private key	Confirmed/Potential	Replace if required

30 — Rotate Exposed Secrets

If evidence indicates a credential was exposed:

Assume Compromise
      ↓
Rotate Credential
      ↓
Update Authorized Consumers
      ↓
Invalidate Old Credential
      ↓
Monitor Usage

Do not simply change the Kubernetes Secret object while leaving the external credential valid.

Example

For a database password:

Database
   ↓
Generate New Credential
   ↓
Update Secret
   ↓
Restart/Redeploy Application
   ↓
Invalidate Old Password

31 — ServiceAccount Token Response

Modern Kubernetes ServiceAccount credentials are commonly projected and short-lived, but implementation details vary.

Response should still consider:

RBAC Restriction

Workload Termination

Identity Redesign

Token Exposure Window

Audit Monitoring

Do not assume that deleting one Secret automatically revokes every form of ServiceAccount access.

32 — Cloud Identity Containment

Managed Kubernetes workloads may have access to cloud APIs.

Possible attack path:

Pod
 ↓
Workload Identity
 ↓
Cloud IAM
 ↓
Cloud Resources

If cloud identity compromise is suspected:

Identify Cloud Principal

Review Cloud Audit Logs

Restrict Permissions

Revoke Sessions/Credentials
Where Supported

Rotate Static Credentials

Monitor Cloud Activity

Coordinate with the cloud security team.

33 — Network Containment

Potential containment targets include:

Pod-to-Pod

Namespace-to-Namespace

Internet Egress

Database Access

Cloud API Access

External Destination

Network Response Questions

Can we isolate only the affected workload?

Does it need DNS during containment?

Does the SOC require telemetry connectivity?

Will blocking egress preserve evidence?

Will isolation break critical services?

34 — Block Known Malicious Destinations

If a destination is confirmed malicious, controls may be applied at:

NetworkPolicy

CNI

Firewall

Cloud Network

Proxy

Secure Web Gateway

Choose the layer that provides reliable enforcement for the environment.

35 — Preserve Network Evidence First

Where possible, preserve:

Source IP

Destination IP

Port

Protocol

Timestamp

Bytes Transferred

DNS Query

Connection Duration

before modifying network controls.

36 — Namespace-Level Containment

If compromise appears broader than one workload, consider:

Namespace Isolation

rather than Pod-only containment.

Potential actions:

Restrict Ingress

Restrict Egress

Disable Compromised Identities

Stop Affected Deployments

Restrict New Deployments

37 — Cluster-Level Escalation

Escalate containment scope when evidence indicates:

Cluster-Wide RBAC Abuse

Multiple Namespaces

Compromised Platform Component

Malicious Admission Configuration

Node Compromise

Control Plane Credential Exposure

38 — Node-Level Incident Response

A node should receive additional scrutiny when the affected workload had:

Privileged Access

Writable hostPath

hostPID

hostNetwork

Dangerous Capabilities

Container Escape Indicators

39 — Identify the Node

kubectl get pod <pod-name> -n <namespace> -o wide

Record:

Node:

Node Pool:

Node IP:

Other Workloads:

40 — Cordon a Suspicious Node

If approved:

kubectl cordon <node-name>

This prevents normal scheduling of additional Pods onto the node.

Validate:

kubectl get nodes

Important

Cordon:

Stops New Scheduling

but does not automatically remove existing workloads.

41 — Drain Considerations

Draining a node can be operationally disruptive and may destroy useful volatile state.

Before considering it:

Preserve Evidence

Assess Workloads

Assess Availability

Coordinate with Platform Team

Obtain Approval

Use the organization’s standard node-maintenance and incident procedures rather than blindly applying generic drain options.

42 — Node Quarantine

For serious node compromise, containment may require:

Network Isolation

Cloud Security Group / Firewall Changes

Removal From Load Balancing

Credential Revocation

Forensic Snapshot

Node Replacement

Exact procedures depend on the infrastructure platform.

43 — Prefer Replacement Over Trusting a Compromised Node

For confirmed node compromise, a strong recovery model is often:

Preserve Evidence
      ↓
Remove Node From Service
      ↓
Rebuild From Trusted Image
      ↓
Apply Current Hardening
      ↓
Validate
      ↓
Return Capacity

rather than attempting to manually clean an untrusted host.

44 — Investigate Container Image Compromise

If the running image is malicious:

Stop Deployment
      ↓
Identify Digest
      ↓
Quarantine Image
      ↓
Review Registry Activity
      ↓
Review Build Pipeline
      ↓
Build Trusted Replacement

45 — Prevent Image Reuse

A malicious image should not remain available for accidental redeployment.

Coordinate with registry administrators to:

Restrict Pulls

Quarantine Artifact

Preserve Evidence

Prevent Promotion

Investigate Related Tags/Digests

46 — Supply-Chain Containment

If compromise originated from CI/CD:

Pause Deployment Pipeline

Protect Production

Disable Compromised Pipeline Identity

Rotate Credentials

Review Repository Access

Review Build Infrastructure

Validate Artifacts

Supply-Chain Attack Path

Developer / CI Credential
        ↓
Repository
        ↓
Build
        ↓
Image
        ↓
Registry
        ↓
Kubernetes

Contain the earliest compromised stage you can identify.

47 — GitOps Containment

If GitOps continuously reconciles cluster state:

Manual Fix in Cluster
      ↓
GitOps Controller
      ↓
Malicious Configuration Returns

Therefore investigate:

Git Repository

Desired State

Pull Request

Commit

GitOps Controller

Deployment Credentials

Fix both:

Source of Truth
+
Running Environment

48 — Admission Control Response

Review:

Pod Security Admission

Kyverno

OPA Gatekeeper

Other Admission Controls

Determine:

Why Was the Workload Allowed?

Was Policy Missing?

Was Policy Audit-Only?

Was Namespace Excluded?

Was an Exception Used?

Was the Policy Modified?

49 — Emergency Admission Guardrail

During an incident, the security/platform team may need a temporary preventive policy.

Examples:

Block Compromised Image

Block Privileged Workloads

Restrict Registry

Prevent Specific Configuration

Emergency policies must be:

Tested

Scoped

Approved

Documented

Monitored

An incorrect cluster-wide admission policy can create a production outage.

50 — Eradication Phase

Containment stops active threat activity.

Eradication removes:

Malicious Resources

Compromised Images

Unauthorized RBAC

Persistence

Exposed Credentials

Vulnerable Components

Compromised Nodes

Malicious Pipeline Changes

51 — Search for Persistence

Review for unexpected:

Deployments

DaemonSets

StatefulSets

Jobs

CronJobs

Pods

ServiceAccounts

Roles

RoleBindings

ClusterRoles

ClusterRoleBindings

Persistence Examples

Malicious CronJob

Unauthorized DaemonSet

New ServiceAccount

New ClusterRoleBinding

Modified Deployment

52 — Review CronJobs

kubectl get cronjobs -A

Investigate:

Unexpected Names

Recent Creation

Unknown Images

Suspicious Commands

Privileged Configuration

53 — Review Jobs

kubectl get jobs -A

Determine whether suspicious one-time workloads were created during the incident.

54 — Review DaemonSets

kubectl get daemonsets -A

DaemonSets deserve special attention because they may execute workloads across many nodes.

55 — Review Recently Modified Resources

Use available:

Audit Logs

Git History

GitOps History

CI/CD History

to identify changes during the incident window.

56 — Remove Malicious Persistence

For each confirmed malicious resource:

Preserve Evidence
      ↓
Confirm Ownership
      ↓
Obtain Approval
      ↓
Delete / Revert
      ↓
Validate

57 — Patch the Initial Entry Point

If root cause was:

Application Vulnerability

then deleting malicious resources is insufficient.

Required:

Patch Application
      ↓
Build New Image
      ↓
Security Test
      ↓
Deploy Trusted Image

58 — Remediate Excessive RBAC

If ServiceAccount privilege enabled the attack:

Current Permission
      ↓
Business Requirement
      ↓
Minimum Resource
      ↓
Minimum Verb
      ↓
Minimum Scope

Validate:

kubectl auth can-i --list --as=system:serviceaccount:<namespace>:<service-account>

59 — Harden the Workload

Apply lessons from the Workload Security Lab.

Review:

runAsNonRoot

privileged

allowPrivilegeEscalation

capabilities

readOnlyRootFilesystem

seccompProfile

ServiceAccount

Token Mounting

Resources

Host Access

60 — Restore Network Segmentation

Implement appropriate:

Default Deny

Required Ingress

Required Egress

Namespace Segmentation

Sensitive Service Restrictions

Do not simply remove emergency containment and return to unrestricted networking.

61 — Restore Admission Guardrails

Translate incident lessons into preventive controls.

Examples:

Disallow Privileged Containers

Require Non-Root

Require Seccomp

Restrict hostPath

Restrict Host Namespaces

Require Approved Registry

Require Resource Controls

62 — Rotate All Confirmed Exposed Credentials

Create a credential remediation tracker.

Credential	Owner	Rotation	Validation
DB password	App Team	Pending	Pending
API token	App Team	Complete	Complete
Cloud credential	Cloud Team	Complete	Pending
Registry credential	Platform	Pending	Pending

63 — Consider Potential Credential Exposure

Sometimes you cannot prove a credential was stolen.

Ask:

Was the credential accessible
to the compromised workload?

If yes, risk-based rotation may still be appropriate.

64 — Recovery Phase

Recovery means more than:

Application Is Running

Secure recovery means:

Trusted Code

Trusted Image

Trusted Configuration

Trusted Identity

Restricted Network

Rotated Credentials

Monitoring Enabled

65 — Define Recovery Criteria

Before returning to production, define:

Vulnerability Fixed

Malicious Resources Removed

RBAC Corrected

Credentials Rotated

Network Controls Applied

Image Verified

Admission Controls Active

Runtime Monitoring Active

Logging Verified

66 — Build From Trusted Source

Preferred recovery path:

Trusted Source Repository
      ↓
Reviewed Commit
      ↓
Controlled Build
      ↓
Security Scan
      ↓
Trusted Registry
      ↓
Admission Validation
      ↓
Kubernetes

67 — Avoid Reusing Compromised Artifacts

Do not recover by simply restarting:

Unknown / Compromised Image

Use:

Known-Good Artifact

68 — Validate the Image

Confirm:

Repository

Tag

Digest

Build

Scan

Approval

Record the image digest used for recovery where applicable.

69 — Deploy Gradually

For important applications:

Deploy Small
   ↓
Validate
   ↓
Monitor
   ↓
Increase Traffic

Possible approaches include:

Rolling Deployment

Canary

Blue/Green

depending on platform architecture.

70 — Validate Workload Health

Check:

kubectl get pods -n <namespace>

Then:

kubectl describe pod <pod-name> -n <namespace>

Review:

Readiness

Restarts

Events

Image

ServiceAccount

Node

71 — Validate Application Logs

kubectl logs <pod-name> -n <namespace>

Confirm:

Normal Startup

No Unexpected Errors

No Suspicious Processes

Expected Connections

72 — Validate RBAC

Test the recovered identity.

Where authorized:

kubectl auth can-i --list --as=system:serviceaccount:<namespace>:<service-account>

Confirm that unnecessary privileges have been removed.

73 — Validate Network Controls

Test only required communication paths.

Create a matrix:

Source	Destination	Expected	Result
Frontend	Backend	Allow	Pass/Fail
Frontend	Database	Deny	Pass/Fail
Unknown Pod	Backend	Deny	Pass/Fail
Application	Approved API	Allow	Pass/Fail

74 — Validate Secrets

Confirm:

New Credentials Active

Old Credentials Invalid

Applications Updated

No Old Secret References

Rotation Logged

75 — Validate Admission Controls

Attempt controlled non-compliant test deployments in the authorized training or validation environment.

Examples:

Privileged Pod

Unapproved Image

Missing Required Security Context

Expected:

Rejected

Do not perform disruptive validation against production without authorization.

76 — Validate Runtime Monitoring

Generate an approved benign test event in the training or validation environment.

Example:

Controlled Shell Activity

Confirm:

Event Generated
      ↓
Detection Triggered
      ↓
Alert Received
      ↓
SOC Can Investigate

77 — Post-Recovery Monitoring

After recovery, increase monitoring for:

Previously Compromised Identity

Affected Namespace

Affected Image

Affected Node

Known Indicators

Suspicious API Activity

Unexpected Network Connections

Secret Access

New RBAC Changes

78 — Define Monitoring Window

The duration should be based on:

Incident Severity

Threat Persistence

Business Criticality

Credential Exposure

Attack Complexity

Organizational Policy

Do not assume recovery is complete immediately after redeployment.

Search across:

Other Namespaces

Other Clusters

Other Images

Other ServiceAccounts

Other Nodes

Cloud Accounts

CI/CD

Registry

Important Question

Was This Really
Only One Workload?

80 — Search for Shared Indicators

Potential pivot points:

Image Digest

Source IP

Destination

Domain

ServiceAccount

User Identity

Command Pattern

Registry Account

CI/CD Identity

81 — Check Other Clusters

Organizations frequently operate:

Development

Testing

Staging

Production

If the same compromised:

Image

Credential

Pipeline

Repository

is shared, investigate other clusters.

82 — Incident Timeline

Maintain:

Time	Event	Source	Action
T1	Alert generated	Runtime	Triage
T2	Incident confirmed	SOC	Escalated
T3	Evidence preserved	Kubernetes	Complete
T4	Workload isolated	Network	Contained
T5	Credential rotated	IAM	Complete
T6	Clean image deployed	Platform	Recovery

83 — Containment Log

Document every response action.

Timestamp:

Responder:

Action:

Resource:

Reason:

Approval:

Expected Impact:

Observed Result:

Rollback:

Why This Matters

During major incidents:

Responder Action

can look similar to:

Attacker Action

Accurate documentation prevents confusion.

84 — Communication Management

Define communication paths for:

Security Team

Platform Team

Application Team

Leadership

Legal

Compliance

Privacy

Customer Support

Communication Principle

Confirmed Facts

Current Risk

Actions Taken

Business Impact

Next Decision

Avoid unsupported speculation.

85 — Executive Status Update

Example structure:

Incident:

Severity:

Current Status:

Affected Service:

Confirmed Impact:

Potential Impact:

Containment:

Recovery:

Business Impact:

Next Actions:

Next Update:

86 — Regulatory and Privacy Escalation

If the incident may involve:

Personal Data

Regulated Data

Customer Data

Payment Information

Sensitive Corporate Information

engage the appropriate:

Legal

Privacy

Compliance

Leadership

teams according to organizational policy.

Do not make breach-notification decisions solely from technical assumptions.

87 — Determine When the Incident Is Contained

Containment criteria may include:

Malicious Process Stopped

Compromised Workload Isolated

Compromised Identity Restricted

Known Malicious Connections Blocked

Persistence Controlled

Active Unauthorized API Activity Stopped

88 — Determine When Eradication Is Complete

Eradication criteria:

Root Cause Remediated

Malicious Resources Removed

Persistence Removed

Compromised Images Quarantined

Exposed Credentials Rotated

Unauthorized RBAC Removed

Compromised Nodes Rebuilt Where Required

89 — Determine When Recovery Is Complete

Recovery criteria:

Trusted Workloads Running

Business Function Restored

Security Controls Validated

Monitoring Active

No Evidence of Recurrence

Stakeholders Approve Return to Normal Operations

90 — Incident Closure Criteria

Do not close simply because:

Alert Stopped

Closure should require:

Containment Complete

Eradication Complete

Recovery Complete

Monitoring Reviewed

Findings Assigned

Evidence Preserved

Report Completed

Lessons Learned Scheduled/Completed

91 — Root Cause Analysis

Document:

Initial Access

Execution

Persistence

Privilege Escalation

Credential Access

Discovery

Lateral Movement

Impact

Then determine:

Primary Root Cause

Contributing Factors

Example

Primary Root Cause:
Vulnerable Internet-facing application.

Contributing Factors:

Excessive ServiceAccount permissions

Missing egress restrictions

Insufficient workload hardening

Delayed runtime detection

92 — Build the Attack Path

Example:

Internet
   ↓
Vulnerable Application
   ↓
Container Execution
   ↓
ServiceAccount Credential
   ↓
Kubernetes API
   ↓
Secret Access
   ↓
Internal Service
   ↓
External Connection

93 — Map Response Controls

Now map the response:

Application Exploitation
        ↓
Patch Application

Container Execution
        ↓
Workload Hardening

ServiceAccount Abuse
        ↓
Least-Privilege RBAC

Secret Exposure
        ↓
Credential Rotation

Lateral Movement
        ↓
NetworkPolicy

Runtime Activity
        ↓
Detection Engineering

This converts:

Incident

into:

Security Improvement

94 — Lessons Learned Meeting

Include:

SOC

Incident Response

Platform

Application

Cloud Security

DevSecOps

GRC

Leadership

as appropriate.

Questions

What Worked?

What Failed?

What Was Missing?

What Delayed Detection?

What Delayed Containment?

Which Controls Reduced Impact?

Which Controls Must Be Improved?

95 — Avoid Blame

The goal is not:

Who Made the Mistake?

The better question is:

Why Could One Mistake
Become a Security Incident?

Focus on:

Systems

Controls

Processes

Architecture

Detection

Response

96 — Convert Lessons Into Actions

Every meaningful lesson should become:

Action

Owner

Priority

Due Date

Validation

Example

Lesson	Action	Owner
Excessive RBAC	Redesign workload role	Platform
No egress policy	Implement segmentation	Network
Privileged workload	Admission guardrail	Security
Slow detection	Add runtime detection	SOC
Shared credential	Implement workload identity	Cloud

97 — Update Security Baselines

After the incident, update:

Kubernetes Security Baseline

Workload Standards

RBAC Standards

Network Standards

Admission Policies

Runtime Rules

Logging Requirements

Incident Runbooks

98 — Detection Engineering Improvements

Translate observed attacker behavior into detection opportunities.

Example:

Observed:
Unexpected pods/exec activity

Detection:
Alert on unusual interactive access
to sensitive production workloads.

Another:

Observed:
ServiceAccount accessed Secrets

Detection:
Alert when application identities
perform unexpected Secret operations.

99 — Improve Admission Controls

If the incident involved:

Privileged Container

consider automated enforcement through:

Pod Security Admission

Kyverno

OPA Gatekeeper

where appropriate.

100 — Improve RBAC Governance

Implement:

Periodic Access Review

ServiceAccount Ownership

Least Privilege

Privileged Access Monitoring

ClusterRoleBinding Review

Automated Detection

101 — Improve Network Security

Implement:

Communication Inventory

Default-Deny Strategy

Explicit Application Flows

Egress Governance

Sensitive Namespace Isolation

Network Monitoring

102 — Improve Workload Security

Establish a baseline requiring appropriate controls such as:

Non-Root

No Privilege Escalation

Minimal Capabilities

Seccomp

Read-Only Filesystem

Dedicated ServiceAccount

Resource Controls

Trusted Images

103 — Improve Supply-Chain Security

Strengthen:

Repository Access

Branch Protection

Build Identity

Dependency Scanning

Image Scanning

Image Provenance

Registry Permissions

Admission Verification

104 — Improve Forensic Readiness

Ensure important evidence survives workload deletion.

Centralize:

Audit Logs

Application Logs

Container Logs

Runtime Alerts

Network Telemetry

Cloud Logs

CI/CD Logs

Registry Logs

105 — Improve Response Automation Carefully

Automation can reduce containment time.

Possible automated workflows:

Alert
   ↓
Enrichment
   ↓
Severity Decision
   ↓
Human Approval
   ↓
Containment

Avoid uncontrolled automation that can:

Delete Production Workloads

Disable Critical Identities

Block Business Traffic

without appropriate safeguards.

106 — Kubernetes Incident Response Decision Tree

Suspicious Alert
      ↓
Validate
      ↓
Confirmed?
  ┌───┴────┐
  │        │
 No       Yes
  │        │
Close     Scope
           ↓
     Active Threat?
       ┌───┴───┐
       │       │
      No      Yes
       │       │
 Preserve   Preserve
 Evidence   Critical Evidence
       │       ↓
       │    Contain
       └───────┬───────
               ↓
           Eradicate
               ↓
            Recover
               ↓
            Monitor
               ↓
         Lessons Learned

107 — Workload Containment Decision

Compromised Pod
      ↓
Controller Managed?
   ┌──┴──┐
  No    Yes
  │      │
Isolate  Inspect Controller
  │      ↓
  │   Controller Clean?
  │      ├── Yes → Scale/Replace Pod
  │      │
  │      └── No → Fix Controller/Source
  ↓
Preserve Evidence

108 — Identity Containment Decision

Identity Suspected
      ↓
Determine Permissions
      ↓
Used by Other Workloads?
   ┌──┴──┐
  No    Yes
  │      │
Restrict  Coordinate Impact
  │      │
  └──┬───┘
     ↓
Revoke / Rotate
     ↓
Validate Permissions

109 — Node Response Decision

Workload Compromise
      ↓
Host-Level Exposure?
   ┌──┴──┐
  No    Yes
  │      │
Monitor  Escalate
         ↓
      Cordon
         ↓
 Preserve Evidence
         ↓
 Quarantine / Rebuild

110 — Incident Response Evidence Package

Maintain:

01 Incident Record

02 Initial Alert

03 Evidence Register

04 Pod Evidence

05 Workload Evidence

06 Identity/RBAC Evidence

07 Audit Evidence

08 Runtime Evidence

09 Network Evidence

10 Cloud Evidence

11 Node Evidence

12 Containment Actions

13 Credential Rotation

14 Recovery Validation

15 Timeline

16 Root Cause Analysis

17 Final Report

111 — Incident Response Action Tracker

Action	Owner	Priority	Status	Validation
Isolate workload	Platform	Critical	TBD	Network test
Restrict SA	Security	Critical	TBD	RBAC test
Rotate DB credential	App	High	TBD	Old credential denied
Replace image	DevSecOps	High	TBD	Digest verified
Apply NetworkPolicy	Platform	High	TBD	Connectivity matrix

112 — Incident Report Template

Incident ID:

Title:

Severity:

Environment:

Detection Time:

Containment Time:

Recovery Time:

Incident Commander:

Executive Summary:

Initial Detection:

Affected Resources:

Affected Identities:

Attack Path:

Timeline:

Confirmed Impact:

Potential Impact:

Containment Actions:

Eradication Actions:

Recovery Actions:

Root Cause:

Contributing Factors:

Credential Exposure:

Data Impact:

Business Impact:

Security Control Failures:

Lessons Learned:

Corrective Actions:

Owners:

Target Dates:

Final Status:

113 — Key Incident Response Metrics

Track useful metrics such as:

Time to Detect

Time to Validate

Time to Contain

Time to Eradicate

Time to Recover

Affected Workloads

Affected Identities

Credentials Rotated

Persistence Mechanisms Found

Critical Control Gaps

Repeat Incidents

114 — Common Kubernetes IR Mistakes

Avoid:

Deleting the Pod before evidence collection

Assuming Pod deletion equals containment

Ignoring the controller

Ignoring the ServiceAccount

Ignoring cloud IAM

Ignoring CI/CD

Ignoring GitOps reconciliation

Ignoring the image registry

Rotating only Kubernetes objects
but not external credentials

Restoring from an untrusted image

Removing containment too early

Closing the incident when alerts stop

115 — Kubernetes Incident Response Checklist

Validation

Scope

Evidence

Workload Containment

Identity Containment

ServiceAccount permissions reviewed
Unauthorized RBAC identified
Malicious bindings removed
Exposed credentials identified
Credentials rotated/revoked
Cloud identity reviewed

Network Containment

Node Response

Eradication

Recovery

Monitoring

Closure

Professional Deliverables

At completion, produce:

Kubernetes Incident Response Report

Incident Timeline

Evidence Register

Containment Log

Affected Resource Inventory

Credential Exposure Matrix

Attack Path Diagram

Blast Radius Assessment

Root Cause Analysis

Recovery Validation Report

Corrective Action Tracker

Executive Summary

Interview Questions

What makes Kubernetes incident response different from traditional server incident response?
What are the major phases of Kubernetes incident response?
Why should an alert be validated before containment?
How would you classify Kubernetes incident severity?
Why should evidence be preserved before deleting a Pod?
Why might deleting a Pod fail to contain an incident?
How do Kubernetes controllers affect containment?
How would you quarantine a Kubernetes workload?
What role can NetworkPolicy play during containment?
Why must you verify that NetworkPolicy is actually enforced?
When would you scale a Deployment to zero?
What are the availability implications of scaling to zero?
How would you respond to a compromised ServiceAccount?
Why should effective RBAC permissions be reviewed?
What should you do with an unauthorized RoleBinding?
Why are ClusterRoleBindings especially sensitive?
How would you respond to exposed Kubernetes Secrets?
Why must external credentials be rotated separately?
How can Kubernetes compromise expand into cloud IAM?
How would you contain a compromised cloud workload identity?
What is short-term containment?
What is long-term containment?
When should a Kubernetes node be treated as potentially compromised?
What does kubectl cordon accomplish?
Why can draining a node affect forensic evidence?
Why might a compromised node be rebuilt instead of cleaned?
How would you respond to a malicious container image?
Why should image digests be recorded?
How can CI/CD compromise affect Kubernetes?
How does GitOps complicate manual incident containment?
Why should admission controls be reviewed after an incident?
How can Kyverno or Gatekeeper prevent recurrence?
What Kubernetes resources can provide persistence?
Why are DaemonSets important during incident response?
Why should CronJobs be investigated?
What is eradication?
What is secure recovery?
Why should recovery use known-good artifacts?
What should be validated before restoring production?
Why should RBAC be retested after remediation?
How should network segmentation be validated?
How do you validate credential rotation?
Why should runtime detection be tested after recovery?
What is post-recovery monitoring?
Why should other clusters be searched after an incident?
What is root-cause analysis?
What are contributing factors?
What should happen during lessons learned?
What metrics are useful for Kubernetes incident response?
When should a Kubernetes incident be formally closed?

Runbook Readiness Milestone

You should now be able to receive:

Confirmed Kubernetes Incident

and systematically execute:

VALIDATE
   ↓
CLASSIFY
   ↓
SCOPE
   ↓
PRESERVE
   ↓
CONTAIN WORKLOAD
   ↓
CONTAIN IDENTITY
   ↓
CONTAIN NETWORK
   ↓
ASSESS NODE
   ↓
ROTATE CREDENTIALS
   ↓
ERADICATE
   ↓
REBUILD
   ↓
RECOVER
   ↓
VALIDATE
   ↓
MONITOR
   ↓
IMPROVE

Final Incident Response Mental Model

When responding to Kubernetes incidents, never think only:

Which Pod Do I Delete?

Think:

What Is Compromised?

What Identity Is Involved?

What Permissions Exist?

What Network Paths Exist?

What Credentials Were Exposed?

What Created the Workload?

Could It Reappear?

Could the Node Be Compromised?

Could the Cloud Environment Be Affected?

What Evidence Must Be Preserved?

What Must Be Changed
Before We Can Trust
the Environment Again?

Runbook Outcome

Before this runbook:

You could reconstruct
what happened during
a Kubernetes incident.

After this runbook:

You can validate incidents,

classify severity,

preserve evidence,

contain workloads,

restrict compromised identities,

rotate exposed credentials,

isolate network activity,

respond to node compromise,

remove persistence,

remediate malicious images,

recover from trusted artifacts,

validate security controls,

monitor for recurrence,

and lead structured
post-incident improvement.

You have moved from:

Kubernetes Forensic Investigation

to:

Kubernetes Incident Response
and Secure Recovery.

Kubernetes Security Runbooks Complete

You have now completed:

Runbook 01
Kubernetes Compliance Assessment
        ↓
Assess Security Posture

Runbook 02
Kubernetes Forensics
        ↓
Reconstruct Security Events

Runbook 03
Kubernetes Incident Response
        ↓
Contain, Eradicate and Recover

These runbooks connect directly with your Kubernetes labs:

Kubernetes Fundamentals
        ↓
RBAC
        ↓
Kyverno
        ↓
Network Policies
        ↓
OPA Gatekeeper
        ↓
Runtime Security
        ↓
Workload Security
        ↓
Compliance Assessment
        ↓
Forensics
        ↓
Incident Response

Together, they form a practical Kubernetes security lifecycle:

BUILD
   ↓
HARDEN
   ↓
GOVERN
   ↓
MONITOR
   ↓
ASSESS
   ↓
INVESTIGATE
   ↓
RESPOND
   ↓
IMPROVE

What’s Next?

➡️ Kubernetes Security Path Complete

You have completed the practical Kubernetes security journey covering:

Kubernetes Administration

Identity & RBAC

Workload Security

Network Security

Policy-as-Code

Admission Control

Runtime Security

Compliance Assessment

Digital Forensics

Incident Response

The next progression is to apply these skills across real enterprise environments where Kubernetes interacts with:

Cloud IAM

Cloud Networking

Managed Kubernetes

CI/CD

Container Registries

Secrets Management

SIEM

DevSecOps

Enterprise Governance

At this point, the student should be capable of moving beyond isolated Kubernetes controls and approaching Kubernetes as a complete enterprise security platform.