Lab 04 — Kubernetes Threat Hunting

Mission Information

Item	Details
Lab ID	K8S-IR-LAB-04
Difficulty	Expert
Estimated Time	2–3 Hours
Platform	Kubernetes
Environment	Kind / Minikube / Amazon EKS
Type	Threat Hunting
Focus Area	Kubernetes Threat Detection
Primary Tools	kubectl, jq, Falco, Trivy, Linux CLI
Recommended Tools	AWS CLI, CloudTrail, GuardDuty, Prometheus, Grafana
Prerequisites	Kubernetes Security, Runtime Security, Incident Response Fundamentals

Mission Brief

CloudNova Technologies has not received any high-priority security alerts during the last several weeks.

However, recent intelligence reports indicate that advanced attackers are increasingly avoiding detection by using legitimate Kubernetes functionality instead of obvious exploits.

The Security Operations Centre (SOC) has initiated a proactive threat hunting exercise to determine whether an attacker has already established persistence within the Kubernetes environment.

Unlike traditional incident response, there is no confirmed compromise.

Your objective is to proactively search for indicators of malicious activity using Kubernetes telemetry, runtime information, cloud logs, identity records, and workload configurations.

Learning Objectives

By completing this lab you will learn how to:

Perform proactive Kubernetes threat hunting
Identify suspicious workloads
Detect persistence mechanisms
Hunt for compromised identities
Analyze Kubernetes Audit Logs
Review runtime telemetry
Investigate unusual network behavior
Detect unauthorized privilege escalation
Correlate multiple log sources
Build threat hunting hypotheses

Enterprise Hunting Workflow

Threat Intelligence

        │

        ▼

Build Hunting Hypothesis

        │

        ▼

Collect Cluster Evidence

        │

        ▼

Analyze Workloads

        │

        ▼

Analyze Identity

        │

        ▼

Analyze Runtime Activity

        │

        ▼

Analyze Network Traffic

        │

        ▼

Correlate Evidence

        │

        ▼

Threat Assessment

        │

        ▼

Hunting Report

Threat Hunting Objectives

Hunt for:

Suspicious Pods
Unknown images
Privileged containers
Excessive RBAC permissions
Unknown Service Accounts
Suspicious DaemonSets
Malicious CronJobs
Unexpected outbound connections
Unauthorized Secrets access
Hidden persistence mechanisms

Lab Environment

Available resources:

Kubernetes Cluster
kubectl
Kubernetes Audit Logs
Falco
CloudTrail (Amazon EKS)
GuardDuty
CloudWatch Logs
Sample production workloads

Hunting Hypothesis

Hypothesis:

“An attacker has compromised a Kubernetes workload and established persistence without triggering critical security alerts.”

Your task is to validate or disprove this hypothesis.

Task 1 — Build an Asset Inventory

List all namespaces:

kubectl get namespaces

List all workloads:

kubectl get pods -A

kubectl get deployments -A

kubectl get daemonsets -A

kubectl get statefulsets -A

Questions:

Are there unknown namespaces?
Are there unexpected workloads?

Task 2 — Hunt for Privileged Containers

Review Pods:

kubectl get pods \
-A \
-o yaml

Search for:

privileged
hostPID
hostIPC
hostNetwork
HostPath
allowPrivilegeEscalation

Questions:

Which workloads violate security standards?
Are privileged Pods expected?

Task 3 — Hunt for Suspicious Images

List container images:

kubectl get pods -A \
-o=jsonpath='{..image}'

Review:

Public registries
Unknown registries
Mutable tags
:latest images
Test images

Questions:

Are all images approved?
Are images signed?

Task 4 — Hunt for Persistence

Review:

kubectl get daemonsets -A

kubectl get cronjobs -A

kubectl get jobs -A

Look for:

Unknown DaemonSets
Scheduled malware
Unexpected Jobs
Newly created workloads

Task 5 — Hunt for Privileged Service Accounts

List Service Accounts:

kubectl get serviceaccounts -A

Review permissions:

kubectl auth can-i \
--list \
--as=system:serviceaccount:production:default

Questions:

Are Service Accounts over-privileged?
Can they access Secrets?
Can they create Pods?

Task 6 — Hunt for RBAC Abuse

Review:

kubectl get clusterrolebindings

kubectl get rolebindings -A

Look for:

cluster-admin assignments
Wildcard permissions
Anonymous access
Unknown identities

Questions:

Which identities have administrative privileges?
Are there unnecessary bindings?

Task 7 — Hunt for Secret Exposure

List Secrets:

kubectl get secrets -A

Review:

Recently created Secrets
Default tokens
Unused Secrets
Application credentials

Investigate:

Secret mounting
Secret access
Secret ownership

Task 8 — Review Kubernetes Audit Logs

Investigate:

exec requests
Secret access
Pod creation
RBAC changes
Namespace creation
ClusterRoleBinding changes

Questions:

Which user created new workloads?
Were Secrets accessed unexpectedly?

Task 9 — Hunt for Runtime Threats

Review Falco alerts:

kubectl logs \
-n falco \
deployment/falco

Look for:

Shell execution
Reverse shell
Host access
Sensitive file access
Unexpected binaries
Container escape attempts

Task 10 — Hunt for Suspicious Network Activity

Inspect active connections:

kubectl exec \
-it suspicious-pod \
-n production \
-- ss -tunap

Review:

External IPs
Unknown destinations
Long-running sessions
High-volume traffic

If using Amazon EKS, also review:

VPC Flow Logs
GuardDuty findings
CloudWatch metrics

Task 11 — Hunt for Configuration Drift

Review:

kubectl get all -A

Compare:

Current configuration
Approved baseline
Git repository
Infrastructure as Code

Questions:

Have unauthorized changes occurred?
Were changes approved?

Task 12 — Build an Attack Timeline

Correlate:

Kubernetes Events
Audit Logs
CloudTrail
Falco alerts
Container logs
Runtime activity

Create:

Time	Event	Source

Task 13 — Document Indicators of Compromise

Record:

Suspicious images
Unknown Service Accounts
Privileged Pods
Reverse shell
Secret access
Unknown external IPs
Persistence mechanisms
Unauthorized RBAC changes

Task 14 — Threat Assessment

Determine:

Is compromise confirmed?
Is additional investigation required?
Is the activity benign?
Does the evidence support the original hypothesis?

Assign a confidence level:

High
Medium
Low

Prepare recommendations:

Reduce RBAC permissions
Remove privileged Pods
Rotate Secrets
Enforce Pod Security Admission
Enable image signing
Enforce NetworkPolicies
Enable continuous runtime monitoring
Improve log retention
Conduct regular threat hunting exercises

Validation Checklist

Verify that you successfully:

Created an asset inventory
Reviewed privileged workloads
Investigated container images
Identified persistence mechanisms
Reviewed Service Accounts
Reviewed RBAC
Investigated Secrets
Analyzed Audit Logs
Reviewed runtime alerts
Investigated network activity
Detected configuration drift
Built an attack timeline
Documented IOCs
Produced a hunting assessment

Expected Findings

You should identify:

Suspicious workloads
Privileged containers
RBAC misconfigurations
Potential persistence mechanisms
Secret exposure risks
Runtime anomalies
Configuration drift
Indicators of compromise
Areas requiring further investigation

Threat Hunting Report Template

Section	Details
Hunting Hypothesis
Scope
Data Sources
Findings
Indicators of Compromise
Confidence Level
Business Impact
Recommended Actions
Next Investigation Steps

Best Practices

Hunt proactively, not only after alerts.
Correlate multiple telemetry sources before drawing conclusions.
Establish and maintain a secure baseline for comparison.
Continuously review RBAC and Service Account permissions.
Monitor runtime behavior and network activity.
Automate recurring threat hunting tasks where possible.
Document findings, even if no compromise is identified.
Treat threat hunting as a continuous security practice.

Challenge Exercise

Extend your hunt by:

Mapping findings to the MITRE ATT&CK framework
Investigating GuardDuty Runtime Monitoring alerts
Reviewing CloudTrail API activity
Scanning workloads with Trivy
Reviewing admission controller logs
Hunting across multiple Amazon EKS clusters
Creating custom Falco detection rules

Key Takeaways

After completing this lab, you can:

Build and validate threat hunting hypotheses
Perform proactive Kubernetes threat hunting
Identify suspicious workloads and identities
Detect persistence techniques
Correlate runtime, audit, and cloud telemetry
Investigate RBAC abuse and Secret exposure
Detect configuration drift
Produce actionable threat hunting reports
Strengthen Kubernetes security posture through continuous hunting

What’s Next?

Next Lab: Lab 05 — Enterprise Incident Response Simulation

In the next lab, you will participate in a full-scale enterprise incident response exercise, investigating a simulated compromise of an Amazon EKS production environment from initial detection through containment, eradication, recovery, and executive reporting.

Lab 04 — Kubernetes Threat Hunting

Mission Information

Mission Brief

Learning Objectives

Enterprise Hunting Workflow

Threat Hunting Objectives

Lab Environment

Hunting Hypothesis

Task 1 — Build an Asset Inventory

Task 2 — Hunt for Privileged Containers

Task 3 — Hunt for Suspicious Images

Task 4 — Hunt for Persistence

Task 5 — Hunt for Privileged Service Accounts

Task 6 — Hunt for RBAC Abuse

Task 7 — Hunt for Secret Exposure

Task 8 — Review Kubernetes Audit Logs

Task 9 — Hunt for Runtime Threats

Task 10 — Hunt for Suspicious Network Activity

Task 11 — Hunt for Configuration Drift

Task 12 — Build an Attack Timeline

Task 13 — Document Indicators of Compromise

Task 14 — Threat Assessment

Task 15 — Recommend Security Improvements

Validation Checklist

Expected Findings

Threat Hunting Report Template

Best Practices

Challenge Exercise

Key Takeaways

What’s Next?