Runbook 03 — Kubernetes Security Incident Investigation

Runbook Overview

Security incidents within Kubernetes environments can spread rapidly if not detected and contained early.

Unlike traditional virtual machines, Kubernetes workloads are highly dynamic. Containers are frequently created and destroyed, making timely investigation critical.

This runbook provides a structured incident investigation process used by enterprise Security Operations Centres (SOCs), Cloud Security Engineers and Incident Response teams when responding to suspicious activity within Amazon EKS clusters.

The methodology aligns with the NIST Incident Response Lifecycle:

Preparation
Detection & Analysis
Containment
Eradication
Recovery
Lessons Learned

This runbook is designed for real-world enterprise environments where Kubernetes is integrated with AWS security services and centralized monitoring platforms.

Investigation Information

Investigation Objective

Investigate a Kubernetes security incident, determine the root cause, assess business impact, contain the threat and document remediation activities.

Estimated Time

3–5 Hours

Investigation Type

Security Incident Investigation
Digital Forensics
Threat Hunting
Incident Response
SOC Operations
Cloud Security

Business Scenario

A multinational financial services company operates:

850 Amazon EKS clusters
20 AWS Accounts
Multiple production workloads
24×7 Enterprise SOC

At 02:17 UTC, several security tools generate alerts indicating suspicious activity in a production Kubernetes cluster.

Security telemetry indicates:

Kubernetes Secret access
Interactive shell execution
Privilege escalation attempt
Abnormal outbound traffic
High CPU utilisation
Suspicious IAM role assumption

The SOC escalates the incident to the Cloud Security Team for immediate investigation.

Enterprise Investigation Workflow

Alert Generated

↓

SOC Triage

↓

Evidence Collection

↓

Timeline Analysis

↓

Root Cause Identification

↓

Containment

↓

Eradication

↓

Recovery

↓

Lessons Learned

Investigation Scope

Review evidence from:

Kubernetes Audit Logs
Amazon CloudWatch Logs
Falco
Prometheus
Grafana
AWS CloudTrail
Amazon GuardDuty
AWS Security Hub
Amazon Inspector
Enterprise SIEM
SOAR Platform

Investigation Checklist

Investigation Step	Status
Incident Confirmed	☐
Scope Identified	☐
Evidence Collected	☐
Timeline Created	☐
IoCs Identified	☐
Root Cause Confirmed	☐
Threat Contained	☐
Recovery Completed	☐
Report Published	☐

Phase 1 — Incident Validation

Objective

Confirm that the reported security event is genuine.

Review:

SIEM Alerts
Falco Alerts
GuardDuty Findings
Security Hub Findings
CloudTrail Events

Determine:

Severity
Affected Cluster
Namespace
Workload
Business Impact

Initial Incident Record

Field	Value
Incident ID	INC-2026-001
Severity	Critical
Environment	Production
Cluster	production-eks
Namespace	payments
Primary Pod	payment-api
Detection Time	02:17 UTC

Phase 2 — Collect Evidence

Collect evidence before making changes to the environment.

Gather:

Audit Logs
Falco Events
Prometheus Metrics
Grafana Dashboards
CloudTrail Logs
GuardDuty Findings
Security Hub Findings
Kubernetes Events

Commands

View cluster events.

kubectl get events -A

View Pods.

kubectl get pods -A

Describe affected Pod.

kubectl describe pod payment-api

Export Pod definition.

kubectl get pod payment-api -o yaml

Review logs.

kubectl logs payment-api

Phase 3 — Review Kubernetes Audit Logs

Search for:

kubectl exec
get secrets
create clusterrolebinding
patch deployment
create serviceaccount
token requests

Example event:

{
  "verb":"get",
  "resource":"secrets",
  "user":"system:serviceaccount:payments:payment-sa",
  "namespace":"payments"
}

Review:

User
Source IP
Namespace
Timestamp
API Action

Phase 4 — Review Runtime Activity

Review Falco alerts.

Example:

Interactive shell inside container

Privilege escalation detected

Unexpected outbound network connection

Review:

Container
User
Process
Parent Process
Severity
Timestamp

Phase 5 — Review Infrastructure Metrics

Open Grafana.

Review:

CPU spikes
Memory spikes
Pod restarts
Node availability
Network traffic
Resource exhaustion

Correlate metrics with Falco alerts.

Phase 6 — Review AWS Activity

Review CloudTrail.

Look for:

AssumeRole
GetCallerIdentity
IAM changes
EKS API activity
Security Group changes

Review GuardDuty findings.

Example:

Backdoor:EC2/C&CActivity

Review Security Hub findings.

Check Inspector for newly discovered vulnerabilities.

Phase 7 — Determine Attack Timeline

Construct the sequence of events.

Time	Activity
02:15	IAM Role Assumed
02:16	Interactive Shell
02:17	Secret Access
02:18	Privilege Escalation
02:19	Outbound Traffic
02:20	CPU Spike
02:21	GuardDuty Alert
02:22	SIEM Incident Created

Timeline analysis helps identify attacker behaviour and supports forensic reporting.

Phase 8 — Identify Indicators of Compromise (IoCs)

Record observed IoCs.

Indicator	Observed
Interactive Shell	☐
Secret Access	☐
Privilege Escalation	☐
Unknown Source IP	☐
Reverse Shell	☐
High CPU Usage	☐
Suspicious IAM Role	☐
Data Exfiltration	☐

Document all confirmed indicators.

Phase 9 — Assess Business Impact

Determine:

Which applications were affected?
Was customer data exposed?
Were Secrets accessed?
Were IAM credentials compromised?
Was lateral movement observed?
Were production services disrupted?

Assign an impact rating.

Rating	Description
Low	No customer impact
Medium	Limited service degradation
High	Sensitive workloads affected
Critical	Customer or regulated data at risk

Phase 10 — Containment

Immediate containment actions:

Isolate affected Pods
Apply restrictive Network Policies
Disable compromised Service Accounts
Revoke temporary IAM credentials
Block malicious IP addresses
Quarantine worker nodes if required
Capture forensic evidence before terminating workloads

Confirm containment has been completed.

Phase 11 — Eradication

Remove the attacker’s access.

Tasks include:

Delete malicious Pods
Redeploy trusted container images
Patch vulnerable applications
Rotate Kubernetes Secrets
Rotate IAM credentials
Remove unauthorized RBAC permissions
Update Falco detection rules

Phase 12 — Recovery

Validate that production services have recovered.

Verify:

Healthy Pods
Successful deployments
Monitoring restored
No active alerts
Normal CPU and memory usage
Application functionality
Security controls operational

Phase 13 — Root Cause Analysis

Determine:

Initial attack vector
Exploited vulnerability
Misconfigured security control
Identity used
Duration of attacker access
Controls that failed
Controls that successfully detected the attack

Document the root cause.

Phase 14 — Lessons Learned

Conduct a post-incident review.

Discuss:

What worked well?
What delayed detection?
Which alerts should be improved?
Which dashboards require enhancement?
Which runbooks need updating?
Which security controls should be strengthened?

Assign improvement actions and owners.

Investigation Report

Executive Summary

Field	Value
Incident ID
Investigation Lead
Date
Severity
Root Cause
Business Impact
Current Status

Findings

Finding	Severity	Recommendation
Weak RBAC permissions	High	Apply least privilege
Excessive Service Account access	High	Restrict permissions
Missing Network Policy	Medium	Implement Zero Trust policies
Delayed Alert Correlation	Medium	Improve SIEM rules

Recommendations

Immediate (0–30 Days)

Rotate compromised credentials
Patch affected workloads
Improve alert correlation
Review RBAC permissions

Short-Term (30–90 Days)

Expand runtime monitoring
Improve SIEM detection rules
Enhance Grafana dashboards
Automate containment playbooks

Long-Term (3–12 Months)

Implement AI-assisted threat detection
Expand threat hunting capabilities
Standardize Kubernetes security baselines
Conduct quarterly incident response exercises

Enterprise Best Practices

As a Cloud Security Engineer:

Preserve evidence before containment.
Correlate multiple telemetry sources during every investigation.
Document every action taken during the incident.
Use immutable log storage for forensic evidence.
Follow approved incident response procedures.
Validate recovery before closing incidents.
Conduct root cause analysis for every critical incident.
Update detection rules based on lessons learned.
Perform regular tabletop exercises.
Continuously improve security controls after each investigation.

Real-World Scenario

A global retail company experienced a compromise of a production application running on Amazon EKS.

Attackers exploited an unpatched web application vulnerability, gained an interactive shell inside a container and attempted to access Kubernetes Secrets.

Falco detected the shell session, Kubernetes Audit Logs recorded Secret access, Prometheus identified abnormal CPU usage and GuardDuty detected outbound communication with a known malicious command-and-control server.

The SOC correlated these events through the enterprise SIEM and initiated automated SOAR playbooks that isolated the affected Pod, revoked temporary IAM credentials and created an incident ticket.

The Cloud Security Team completed the investigation, identified excessive Service Account permissions as a contributing factor and implemented stricter RBAC, Network Policies and continuous runtime monitoring to prevent similar incidents.

Investigation Summary

Congratulations!

You have completed an enterprise Kubernetes Security Incident Investigation.

During this runbook you:

Validated a production security incident
Collected forensic evidence
Analysed Kubernetes Audit Logs
Investigated runtime activity
Reviewed AWS security telemetry
Built an attack timeline
Identified Indicators of Compromise (IoCs)
Assessed business impact
Contained and eradicated the threat
Documented findings and lessons learned

These investigation techniques closely reflect the daily responsibilities of Cloud Security Engineers, Incident Responders and SOC Analysts securing enterprise Amazon EKS environments.

Key Takeaways

Successful investigations depend on complete and reliable telemetry.
Correlating Kubernetes, AWS and SIEM data provides the best visibility into attacker behaviour.
Rapid containment limits business impact and reduces attacker dwell time.
Root cause analysis and lessons learned strengthen long-term security posture.
Well-defined runbooks enable consistent, repeatable and effective incident response.

What’s Next?

You have now completed Module 06 — Kubernetes Logging, Monitoring & Security Operations, including all lessons, hands-on labs and enterprise runbooks.

➡️ Next Module: Module 07 — Kubernetes Backup, Disaster Recovery & Business Continuity