Lab 04 — Investigate a Kubernetes Security Incident

Lab Overview

A Security Operations Centre (SOC) rarely investigates alerts from a single security tool.

Instead, analysts correlate information from multiple sources to understand:

• What happened
• When it happened
• Who performed the activity
• Which workloads were affected
• Whether the activity was malicious
• How to contain the incident

In this lab, you will perform a complete Kubernetes security investigation using the monitoring platform you built in the previous labs.

You will analyse security events from:

• Kubernetes Audit Logs
• Falco
• Prometheus
• Grafana
• Amazon CloudTrail
• Amazon GuardDuty (simulated findings)

This lab follows a real enterprise incident response workflow used by Cloud Security Engineers and SOC Analysts.

---

Mission Information

Mission Objective

Investigate a simulated Kubernetes security incident, identify the root cause, determine the attack timeline and recommend containment actions.

---

Estimated Time

90–120 Minutes

---

Difficulty

Advanced

---

Lab Type

• Amazon EKS
• Security Operations Centre (SOC)
• Incident Investigation
• Runtime Security
• Threat Detection
• Digital Forensics

---

Learning Objectives

After completing this lab, you will be able to:

• Investigate Kubernetes security alerts
• Build an attack timeline
• Analyse Kubernetes Audit Logs
• Analyse Falco runtime alerts
• Correlate Prometheus metrics
• Review CloudTrail events
• Identify Indicators of Compromise (IoCs)
• Recommend containment actions

---

Business Scenario

You are an L2 Cloud Security Engineer working in a global financial institution.

At 09:32 UTC, the SOC receives several alerts.

Within minutes:

• Falco reports an interactive shell inside a production container.
• Prometheus detects abnormal CPU utilisation.
• Kubernetes Audit Logs show repeated Secret access.
• CloudTrail records suspicious IAM role activity.
• GuardDuty generates a finding for communication with a known malicious IP address.

Management suspects that a production payment application has been compromised.

Your responsibility is to investigate the incident, determine the root cause and recommend immediate containment actions.

---

Lab Architecture

Compromised Container

↓

Falco Alert

↓

Kubernetes Audit Logs

↓

Prometheus Metrics

↓

CloudTrail Events

↓

GuardDuty Finding

↓

Enterprise SIEM

↓

SOC Investigation

---

Enterprise Investigation Workflow

Alert Received

↓

Validate Alert

↓

Collect Evidence

↓

Correlate Events

↓

Determine Root Cause

↓

Assess Business Impact

↓

Contain Threat

↓

Document Findings

---

Lab Tasks

Task	Description
Task 1	Review Incident Details
Task 2	Analyse Kubernetes Audit Logs
Task 3	Investigate Falco Runtime Alerts
Task 4	Review Prometheus & Grafana Metrics
Task 5	Analyse CloudTrail Activity
Task 6	Review GuardDuty Findings
Task 7	Build the Attack Timeline
Task 8	Recommend Containment Actions

---

Task 1 — Review Incident Details

The SOC has provided the following incident summary.

Item	Value
Cluster	production-eks
Namespace	payments
Pod	payment-api-58d97f
Severity	Critical
Detection Time	09:32 UTC
Status	Under Investigation

Questions to consider:

• Which workload is affected?
• Is the workload internet-facing?
• What business service does it support?

---

Task 2 — Analyse Kubernetes Audit Logs

Search CloudWatch Logs for recent Kubernetes API activity.

Look for:

• get secrets
• list secrets
• kubectl exec
• create clusterrolebinding
• patch deployment
• delete pod

Example audit event:

{
  "verb":"get",
  "resource":"secrets",
  "user":"system:serviceaccount:payments:payment-sa",
  "namespace":"payments",
  "responseStatus":200
}

Record:

• User
• Namespace
• Resource
• Timestamp
• Source IP
• Response code

---

Investigation Questions

• Who accessed the Secret?
• Which Service Account was used?
• Was the request expected?
• Has this account accessed Secrets previously?

---

Task 3 — Investigate Falco Alerts

View Falco logs.

kubectl logs \
-n falco \
-l app.kubernetes.io/name=falco

Example alert:

Priority: Critical

Rule:

Terminal shell in container

Container:

payment-api

User:

root

Command:

/bin/bash

Review:

• Rule triggered
• Severity
• Container
• User
• Process
• Time

---

Additional Falco Events

Example findings:

Sensitive file opened

/etc/shadow

Unexpected outbound connection

Privilege escalation attempt

These indicate possible post-compromise activity.

---

Task 4 — Review Prometheus & Grafana

Open Grafana and inspect the following dashboards:

• Cluster Overview
• Node Metrics
• Pod Metrics
• Namespace Overview

Look for:

• CPU spikes
• Memory spikes
• Restart count
• Network traffic
• Pod lifecycle events

Questions:

• Did resource usage change during the incident?
• Did any Pods restart?
• Which node hosted the affected workload?

---

Task 5 — Analyse CloudTrail Events

Review CloudTrail activity around the incident time.

Look for:

• AssumeRole
• GetCallerIdentity
• UpdateClusterConfig
• CreateAccessEntry
• IAM changes

Example:

Event:

AssumeRole

Role:

payments-admin

Source IP:

198.51.100.20

Determine:

• Was the IAM activity expected?
• Was MFA used?
• Was the source IP recognised?

---

Task 6 — Review GuardDuty Findings

Review simulated GuardDuty findings.

Example:

Finding:

Backdoor:EC2/C&CActivity

Severity:

High

Affected Resource:

EKS Worker Node

Questions:

• Does this align with Falco alerts?
• Is outbound communication expected?
• Should the node be isolated?

---

Task 7 — Build the Attack Timeline

Using all collected evidence, construct the sequence of events.

Example:

Time	Event
09:31	Suspicious IAM role assumed
09:32	Interactive shell detected
09:33	Kubernetes Secret accessed
09:34	Sensitive file opened
09:35	CPU utilisation increased
09:36	Outbound network connection detected
09:37	GuardDuty generated High finding

This timeline provides investigators with a complete picture of attacker activity.

---

Indicators of Compromise (IoCs)

Identify all observed IoCs.

Possible examples:

• Interactive shell
• Secret access
• Privilege escalation
• Unknown IP address
• High CPU usage
• Reverse shell
• Unexpected network traffic
• Suspicious IAM role assumption

Document each indicator.

---

Task 8 — Recommend Containment Actions

As the investigating Cloud Security Engineer, recommend actions to the Incident Response team.

Possible actions include:

• Isolate the affected Pod
• Apply restrictive Network Policies
• Rotate Kubernetes Secrets
• Disable the compromised Service Account
• Revoke temporary IAM credentials
• Quarantine the worker node
• Capture forensic evidence
• Redeploy workloads from trusted container images
• Patch the vulnerable application
• Increase monitoring for related workloads

Prioritise actions that minimise business impact while preventing further attacker movement.

---

Validation Checklist

Verify that you successfully:

• Analysed Audit Logs
• Reviewed Falco alerts
• Investigated Prometheus metrics
• Examined Grafana dashboards
• Reviewed CloudTrail activity
• Analysed GuardDuty findings
• Built an attack timeline
• Identified IoCs
• Recommended containment actions

---

Common Investigation Challenges

Too Many Alerts

Focus first on:

• Critical severity
• Production workloads
• Privileged activity
• Secret access
• Interactive shells

---

Missing Logs

Verify:

• Audit Logging enabled
• Falco running
• Prometheus scraping metrics
• CloudTrail logging enabled
• GuardDuty enabled

---

False Positives

Validate:

• Maintenance windows
• Approved administrator activity
• Change requests
• Deployment schedules

Always confirm before escalating an incident.

---

Enterprise Best Practices

As a Cloud Security Engineer:

• Correlate events across multiple telemetry sources.
• Never rely on a single alert.
• Preserve forensic evidence before containment.
• Build an accurate attack timeline.
• Prioritise business-critical workloads.
• Validate user identities and IAM activity.
• Use threat intelligence to enrich investigations.
• Document every investigative step.
• Conduct post-incident reviews.
• Update detection rules based on lessons learned.

---

Real-World Scenario

A global payment processing company experiences intermittent failures in its production payment platform.

The SOC receives alerts indicating:

• Interactive shell access within a production Pod
• Repeated Kubernetes Secret access
• A privileged IAM role assumption from an unfamiliar IP address
• Increased CPU utilisation
• Outbound traffic to a known command-and-control server

By correlating Kubernetes Audit Logs, Falco alerts, Prometheus metrics, CloudTrail events and GuardDuty findings, analysts determine that an attacker exploited a vulnerable application, established an interactive shell and attempted to steal application credentials.

The incident response team isolates the workload, revokes compromised credentials, blocks network communication and restores the application from a trusted container image.

The investigation confirms that customer payment data was protected due to rapid detection and response.

---

Lab Summary

Congratulations!

You have successfully completed a full Kubernetes SOC investigation.

During this lab you learned how to:

• Investigate enterprise Kubernetes alerts
• Analyse Kubernetes Audit Logs
• Review Falco runtime detections
• Correlate Prometheus metrics
• Investigate CloudTrail activity
• Interpret GuardDuty findings
• Build an attack timeline
• Recommend containment actions

These skills closely reflect the day-to-day responsibilities of Cloud Security Engineers, Incident Responders and SOC Analysts operating enterprise Amazon EKS environments.

---

Key Takeaways

• Effective investigations require correlation across multiple telemetry sources.
• Kubernetes Audit Logs explain who performed an action.
• Falco reveals runtime behaviour inside containers.
• Prometheus and Grafana highlight operational anomalies.
• CloudTrail provides AWS API activity.
• GuardDuty adds managed threat intelligence.
• Attack timelines are essential for understanding the full scope of an incident.
• Rapid containment reduces business impact and limits attacker movement.

---

What’s Next?

In the next lab, you will integrate your Kubernetes monitoring environment with enterprise security tooling by forwarding Audit Logs, Falco alerts and monitoring events into a centralized platform for automated detection, correlation and response.

➡️ Next Lab: Lab 05 — Integrate Kubernetes Monitoring with SIEM