Lab 03 — Container Escape Investigation

Mission Information

Item	Details
Lab ID	K8S-IR-LAB-03
Difficulty	Expert
Estimated Time	2–3 Hours
Platform	Kubernetes
Environment	Kind / Minikube / Amazon EKS
Type	Incident Response & Container Forensics
Focus Area	Container Escape Investigation
Primary Tools	kubectl, crictl, journalctl, Falco, jq, Linux Commands
Recommended Tools	Trivy, Sysdig, AWS CLI (Amazon EKS)
Prerequisites	Kubernetes Security, Linux Fundamentals, Runtime Security

Mission Brief

CloudNova Technologies’ Security Operations Centre (SOC) has detected highly suspicious activity from a production Kubernetes cluster.

Runtime monitoring has generated alerts indicating:

A privileged container is running.
Host filesystem directories are mounted inside the container.
Processes have been executed outside the expected container namespace.
Kernel-level system calls triggered multiple runtime alerts.
The affected worker node has initiated unusual outbound network connections.
A new privileged Pod was deployed shortly before the alerts began.

Security engineers suspect a container escape attack.

Your responsibility is to investigate the incident, determine whether the attacker escaped the container, assess the impact on the worker node, collect forensic evidence, and recommend containment actions.

Learning Objectives

After completing this lab, you will be able to:

Investigate suspected container escape attacks
Identify insecure Pod configurations
Analyze Linux namespaces
Inspect host filesystem mounts
Review runtime security events
Investigate worker node activity
Detect privilege escalation
Identify persistence mechanisms
Collect forensic evidence
Recommend containment and recovery actions

Enterprise Scenario

Falco Alert

      │

      ▼

Privileged Container

      │

      ▼

Host Filesystem Access

      │

      ▼

Kernel Interaction

      │

      ▼

Worker Node Investigation

      │

      ▼

Evidence Collection

      │

      ▼

Containment & Recovery

Lab Objectives

Investigate:

Privileged containers
Linux namespaces
Host filesystem access
Runtime security alerts
Mounted host volumes
Running processes
Worker node activity
Network connections
Indicators of compromise
Potential persistence

Lab Environment

Resources available:

Kubernetes Cluster
Worker Nodes
Falco Runtime Security
kubectl
Linux Shell
Kubernetes Audit Logs
Sample vulnerable workload

Task 1 — Identify Privileged Pods

List all Pods:

kubectl get pods -A

Inspect Pod Security Context:

kubectl get pod suspicious-pod \
-o yaml \
-n production

Review:

privileged
hostPID
hostIPC
hostNetwork
allowPrivilegeEscalation
capabilities
runAsUser
readOnlyRootFilesystem

Questions:

Is the Pod running as privileged?
Does it violate security best practices?

Task 2 — Review Mounted Host Volumes

Describe the Pod:

kubectl describe pod suspicious-pod \
-n production

Review mounted volumes:

HostPath
Docker socket
containerd socket
Root filesystem
kubelet directories
Sensitive host directories

Questions:

Can the container access the host filesystem?
Which directories are mounted?

Task 3 — Investigate Linux Namespaces

Access the container:

kubectl exec \
-it suspicious-pod \
-n production \
-- lsns

Review:

PID namespace
Network namespace
Mount namespace
IPC namespace
UTS namespace

Questions:

Is the container sharing the host namespaces?
Are namespace boundaries intact?

Task 4 — Review Runtime Alerts

Display Falco alerts:

kubectl logs \
-n falco \
deployment/falco

Look for alerts such as:

Terminal shell detected
Host filesystem access
Privileged container
Container escape attempt
Sensitive file access
Unexpected kernel activity

Document:

Alert name
Timestamp
Severity
Pod
Namespace
Worker node

Task 5 — Review Running Processes

Inspect running processes:

kubectl exec \
-it suspicious-pod \
-n production \
-- ps aux

Look for:

Bash
sh
curl
wget
nc
Python
Perl
Reverse shells
Unknown binaries

Questions:

Are unexpected processes running?
Are any processes interacting with the host?

Task 6 — Review Host Filesystem Access

Inspect mounted directories:

kubectl exec \
-it suspicious-pod \
-n production \
-- mount

Review:

HostPath mounts
Root filesystem
Docker socket
containerd socket
kubelet directory
/proc
/sys

Questions:

Does the container have direct host access?
Can the attacker modify host files?

Task 7 — Investigate Network Activity

Display active connections:

kubectl exec \
-it suspicious-pod \
-n production \
-- ss -tunap

Review:

External IPs
Listening ports
Established sessions
Unknown destinations

Questions:

Is the container communicating externally?
Are there suspicious connections?

Task 8 — Investigate Worker Node

Identify the worker node:

kubectl get pod suspicious-pod \
-o wide

Record:

Node name
Internal IP
Worker group
Availability Zone (Amazon EKS)

Review:

Node health
Runtime alerts
Recent Events

Task 9 — Review Kubernetes Events

Display Events:

kubectl get events \
-A \
--sort-by=.metadata.creationTimestamp

Look for:

Privileged Pod creation
Failed authentication
Secret access
Node warnings
Security policy violations

Questions:

Were there Events immediately before the alert?
Was the Pod recently deployed?

Task 10 — Review Kubernetes Audit Logs

If enabled, investigate:

exec requests
Pod creation
Secret access
ClusterRoleBinding changes
Service Account usage

Questions:

Which identity initiated the activity?
Which namespace was involved?

Task 11 — Investigate Worker Node Logs

If node access is available:

Review:

journalctl -xe

Inspect:

kubelet logs
container runtime logs
authentication failures
kernel messages
unexpected service activity

Questions:

Are there signs of host compromise?
Were privileged operations executed?

Task 12 — Investigate Persistence

Review Kubernetes resources:

kubectl get daemonsets -A

kubectl get cronjobs -A

kubectl get clusterrolebindings

Look for:

Unknown DaemonSets
Malicious CronJobs
Unexpected ClusterRoleBindings
Suspicious Service Accounts

Questions:

Has the attacker established persistence?

Task 13 — Identify Indicators of Compromise

Document observed IOCs including:

Privileged Pod
Host filesystem access
HostPath mount
Reverse shell
Unknown processes
Suspicious binaries
Unauthorized Service Account
Unknown external IP
Container escape indicators
Kernel alerts

Without destroying evidence, recommend actions:

Isolate affected worker node
Cordon the node
Scale Deployment to zero
Remove network access
Rotate exposed Secrets
Disable compromised Service Account
Preserve forensic evidence
Notify Incident Response team

Avoid deleting the Pod before evidence collection.

Validation Checklist

Verify that you successfully:

Identified privileged Pods
Reviewed Security Context
Investigated HostPath mounts
Analyzed Linux namespaces
Reviewed runtime alerts
Investigated running processes
Reviewed network activity
Investigated worker node
Reviewed Kubernetes Events
Investigated Audit Logs
Reviewed node logs
Investigated persistence
Documented IOCs
Recommended containment actions

Expected Findings

You should identify:

Privileged container configuration
Host filesystem exposure
Namespace sharing
Runtime security alerts
Suspicious processes
External network communication
Worker node indicators
Potential persistence mechanisms
Evidence supporting or disproving a container escape

Best Practices

Avoid immediate deletion of compromised Pods.
Preserve forensic evidence before containment.
Restrict privileged containers wherever possible.
Disable HostPath mounts unless absolutely required.
Prevent host namespace sharing.
Enable runtime threat detection.
Use read-only root filesystems.
Continuously monitor worker nodes for suspicious activity.
Regularly review Kubernetes Audit Logs and runtime alerts.

Challenge Exercise

Extend the investigation by:

Reviewing GuardDuty Runtime Monitoring findings (Amazon EKS)
Mapping attacker activity to the MITRE ATT&CK framework
Scanning the container image with Trivy
Correlating CloudTrail and Kubernetes Audit Logs
Capturing worker node forensic evidence
Building a complete attack timeline from initial access to suspected container escape

Key Takeaways

After completing this lab, you can:

Investigate suspected container escape attacks
Analyze Pod security configurations
Detect insecure HostPath mounts
Review Linux namespaces
Investigate runtime security alerts
Examine worker node evidence
Identify persistence mechanisms
Document indicators of compromise
Recommend containment and recovery actions
Perform enterprise-grade Kubernetes container escape investigations

Next Steps

Next Lab: Lab 04 — Kubernetes Threat Hunting

In the next lab, you will proactively hunt for hidden threats across a Kubernetes cluster by analyzing workloads, identities, runtime telemetry, audit logs, network activity, and configuration changes to identify malicious behavior that may have bypassed automated security controls.

Lab 03 — Container Escape Investigation

Mission Information

Mission Brief

Learning Objectives

Enterprise Scenario

Lab Objectives

Lab Environment

Task 1 — Identify Privileged Pods

Task 2 — Review Mounted Host Volumes

Task 3 — Investigate Linux Namespaces

Task 4 — Review Runtime Alerts

Task 5 — Review Running Processes

Task 6 — Review Host Filesystem Access

Task 7 — Investigate Network Activity

Task 8 — Investigate Worker Node

Task 9 — Review Kubernetes Events

Task 10 — Review Kubernetes Audit Logs

Task 11 — Investigate Worker Node Logs

Task 12 — Investigate Persistence

Task 13 — Identify Indicators of Compromise

Task 14 — Recommend Containment

Validation Checklist

Expected Findings

Best Practices

Challenge Exercise

Key Takeaways

Next Steps