Project 10 — End-to-End Kubernetes Security Capstone

Project Overview

Welcome to Project 10 — End-to-End Kubernetes Security Capstone.

This is the final project in the Enterprise Kubernetes Security Projects module.

In this capstone, you will combine the skills developed across the complete Kubernetes Security Engineer learning path into one enterprise engagement.

You will act as a:

Kubernetes Security Engineer
Cloud Security Architect
Amazon EKS Security Engineer
DevSecOps Engineer
Security Assessor
Detection Engineer
Incident Responder
Security Consultant

This project does not focus on one isolated security control.

You will assess, design, secure, validate, monitor, investigate and report on an enterprise Amazon EKS environment from beginning to end.

Business Requirements

↓

Environment Discovery

↓

Security Assessment

↓

Threat Modelling

↓

Target Architecture

↓

Security Implementation

↓

Control Validation

↓

Incident Simulation

↓

Investigation and Recovery

↓

Compliance and Reporting

↓

Executive Recommendation

Project Mission

CloudNova Technologies is preparing to launch a new enterprise digital platform on Amazon EKS.

The platform will support:

Customer-facing applications
Payment services
Internal administration workloads
Analytics services
Shared platform components
Security monitoring workloads

The organisation has completed several security initiatives, but the final production environment has not yet undergone a complete end-to-end validation.

Management has requested a full Kubernetes security engagement before production approval.

Your mission is to:

Define the project scope.
Discover the current environment.
Assess the existing security posture.
Identify critical risks.
Design the target security architecture.
Implement priority controls.
Validate control effectiveness.
Simulate a Kubernetes security incident.
Investigate and contain the incident.
Recover the environment securely.
Produce compliance evidence.
Present a final production recommendation.

Business Scenario

CloudNova Technologies operates a multi-account AWS environment.

The new digital platform will run across:

Production AWS account
Security tooling account
Log archive account
Shared services account
CI/CD account

The production platform includes:

Amazon EKS
Private worker nodes
Amazon ECR
AWS Secrets Manager
AWS KMS
Amazon CloudWatch
AWS CloudTrail
Amazon GuardDuty
AWS Security Hub
Amazon Inspector
Falco
Prometheus
Grafana
Kyverno or OPA Gatekeeper
Velero or approved backup tooling
Enterprise SIEM

The organisation has identified several potential risks:

Broad cluster administrator access
Public Kubernetes API exposure
Excessive Service Account permissions
Workloads using node IAM roles
Privileged Pods
Missing default-deny Network Policies
Mutable container tags
Incomplete image scanning
Secrets stored in Kubernetes manifests
Inconsistent audit logging
Runtime-monitoring gaps
Untested incident runbooks
Incomplete backup-restoration evidence
Security exceptions without expiry dates

Target Business Outcome

At the end of this project, CloudNova Technologies should have:

A complete Kubernetes security assessment
A documented current-state risk profile
A secure target-state architecture
Implemented high-priority controls
Validated workload and cluster security
Centralised security telemetry
Tested runtime detections
Tested incident-response workflows
Tested backup and recovery
Compliance evidence
A findings and remediation register
A production-readiness scorecard
An executive security recommendation

Project Objectives

By completing this project, you will be able to:

Plan an enterprise Kubernetes security engagement
Define assessment scope and Rules of Engagement
Discover and inventory Amazon EKS environments
Assess IAM, RBAC and workload identity
Review cluster, node, workload and network security
Identify privileged workloads and high-risk configurations
Assess software supply-chain controls
Review secrets and data protection
Evaluate logging and runtime monitoring
Perform threat modelling
Design a production Kubernetes security architecture
Implement security controls
Test positive and negative security scenarios
Conduct a controlled security incident simulation
Investigate Pod, identity, runtime and network activity
Perform containment and recovery
Conduct Root Cause Analysis
Validate compliance and production readiness
Produce executive and technical reports
Present a final risk-based recommendation

Project Difficulty

Level: Expert

Estimated Time

Recommended Duration: 30–40 hours

This project should be completed over multiple practical sessions.

Project Type

Enterprise capstone
Security assessment
Security architecture
Hands-on implementation
Detection engineering
Incident response
Compliance validation
Executive reporting
Portfolio project

Prerequisites

You should understand:

Kubernetes architecture
Amazon EKS
AWS Organizations
AWS IAM
Kubernetes RBAC
Service Accounts
EKS Access Entries
EKS Pod Identity and IRSA
Amazon VPC
Security Groups
Kubernetes Network Policies
Pod Security Admission
Admission controllers
Container image security
Image scanning and signing
AWS Secrets Manager
AWS KMS
Kubernetes Audit Logs
Falco
GuardDuty Runtime Monitoring
SIEM concepts
Incident response
Pod and node forensics
Root Cause Analysis
Backup and disaster recovery
Compliance frameworks

Recommended Tools

AWS

AWS Management Console
AWS CLI
AWS IAM Identity Center
Amazon EKS
Amazon ECR
AWS Secrets Manager
AWS KMS
AWS CloudTrail
Amazon GuardDuty
AWS Security Hub
Amazon Inspector
AWS Config
Amazon CloudWatch
Amazon EventBridge
AWS Backup

Kubernetes

kubectl
Helm
Kubernetes RBAC
Network Policies
Pod Security Admission
ResourceQuota
LimitRange
Admission controllers
Kubernetes Audit Logs

Security

Falco
Falcosidekick
Kyverno or OPA Gatekeeper
Trivy
Kubescape
kube-bench
kubeaudit
Checkov
Prometheus
Grafana
Enterprise SIEM
Incident-management platform

Engineering

Terraform
Git
GitHub or approved source-control platform
CI/CD pipeline
Diagramming tool
Markdown

Cost Warning

This project may create billable AWS resources, including:

Amazon EKS cluster
EC2 worker nodes
NAT Gateways
Load balancers
CloudWatch Logs
GuardDuty Runtime Monitoring
Security Hub
Inspector
ECR
EBS volumes
Backup storage
Cross-Region copies
SIEM ingestion

Use an approved lab or enterprise account.

Destroy unnecessary resources after completing the project.

Safety and Authorisation

Perform this project only in:

Your own AWS account
An approved training environment
A dedicated lab cluster
An authorised enterprise environment

Do not perform:

Real malware execution
Uncontrolled reverse shells
Destructive production testing
Credential theft
Denial-of-Service tests
Container escape exploits
Production data extraction
Unauthorised security testing

All attack simulations must be safe, isolated and approved.

Capstone Architecture

AWS Organization

├── Security Tooling Account
├── Log Archive Account
├── Shared Services Account
├── CI/CD Account
├── Production Workload Account
└── Non-Production Account

Enterprise Identity Provider

↓

AWS IAM Identity Center

↓

MFA and Temporary Roles

↓

Restricted Amazon EKS API

↓

EKS Access Entries

↓

Kubernetes RBAC

↓

Private Worker Nodes

↓

Secure Namespaces

├── Pod Security Admission
├── ResourceQuota
├── LimitRange
├── Dedicated Service Accounts
├── Workload IAM Roles
└── Default-Deny Network Policies

↓

Secure Software Supply Chain

├── Source Review
├── Secret Scanning
├── Dependency Scanning
├── Image Scanning
├── SBOM
├── Image Signing
└── Admission Verification

↓

External Secrets and Encryption

↓

Logging and Runtime Detection

↓

SIEM and SOC

↓

Incident Response and Recovery

↓

Continuous Compliance and Governance

Security Domains

The project covers the following domains:

1. Governance and Scope

2. AWS Account Security

3. Amazon EKS Architecture

4. Identity and Access

5. Worker-Node Security

6. Namespace and Workload Security

7. Network Security

8. Software Supply Chain

9. Secrets and Data Protection

10. Logging and Monitoring

11. Runtime Security

12. Incident Response

13. Backup and Recovery

14. Compliance and Governance

15. Executive Reporting

Required Deliverables

Planning Deliverables

Project charter
Scope statement
Rules of Engagement
Stakeholder register
Assumptions and constraints
Project plan
Evidence-handling plan

Discovery Deliverables

AWS account inventory
Cluster inventory
Namespace inventory
Workload inventory
Identity inventory
Network inventory
Security-tool inventory
Data-flow inventory

Architecture Deliverables

High-level architecture
AWS account architecture
Cluster architecture
Network architecture
Identity architecture
Workload identity flow
Software supply-chain architecture
Secrets architecture
Logging architecture
Runtime architecture
Incident-response architecture
Backup and recovery architecture
Trust-boundary diagram
Data-flow diagram

Technical Deliverables

Terraform configuration
RBAC manifests
Service Account manifests
Pod Security labels
ResourceQuotas
LimitRanges
Network Policies
Admission policies
Secure workload manifests
Runtime rules
Monitoring configuration
Backup configuration
Safe incident-simulation manifests

Security Deliverables

Threat model
Security requirements
Security assessment
Findings register
Risk register
Exception register
Detection catalogue
Control-validation matrix
Compliance mapping
Root Cause Analysis
Remediation roadmap

Operational Deliverables

Access-review procedure
Secure namespace onboarding
Vulnerability-management process
Runtime alert runbook
Compromised Pod runbook
Credential-compromise runbook
Container-escape runbook
Backup and restore runbook
Production-readiness checklist
Escalation matrix
RACI matrix

Management Deliverables

Executive summary
Current-state risk overview
Target-state security architecture
Security scorecard
Incident summary
Business-impact assessment
Remediation roadmap
Residual-risk statement
Final production recommendation
Final presentation

Recommended Project Folder Structure

10-end-to-end-kubernetes-security-capstone/

├── README.md
│
├── 01-project-management/
│   ├── project-charter.md
│   ├── scope.md
│   ├── rules-of-engagement.md
│   ├── stakeholders.md
│   ├── assumptions.md
│   └── project-plan.md
│
├── 02-discovery/
│   ├── aws-inventory.md
│   ├── cluster-inventory.md
│   ├── namespace-inventory.md
│   ├── workload-inventory.md
│   ├── identity-inventory.md
│   ├── network-inventory.md
│   └── security-tool-inventory.md
│
├── 03-assessment/
│   ├── account-security.md
│   ├── eks-security.md
│   ├── identity-rbac.md
│   ├── node-security.md
│   ├── workload-security.md
│   ├── network-security.md
│   ├── image-security.md
│   ├── secrets-security.md
│   ├── logging-runtime.md
│   └── backup-recovery.md
│
├── 04-threat-model/
│   ├── assets.md
│   ├── threat-actors.md
│   ├── trust-boundaries.md
│   ├── attack-paths.md
│   └── control-mapping.md
│
├── 05-architecture/
│   ├── high-level-architecture.md
│   ├── account-architecture.md
│   ├── cluster-architecture.md
│   ├── network-architecture.md
│   ├── identity-architecture.md
│   ├── workload-identity.md
│   ├── supply-chain-architecture.md
│   ├── secrets-architecture.md
│   ├── logging-architecture.md
│   ├── runtime-architecture.md
│   ├── incident-response-architecture.md
│   └── backup-dr-architecture.md
│
├── 06-implementation/
│   ├── terraform/
│   ├── namespaces/
│   ├── rbac/
│   ├── service-accounts/
│   ├── workloads/
│   ├── network-policies/
│   ├── admission-policies/
│   ├── secrets/
│   ├── logging/
│   ├── runtime/
│   └── backup/
│
├── 07-validation/
│   ├── validation-plan.md
│   ├── identity-tests.md
│   ├── workload-tests.md
│   ├── network-tests.md
│   ├── image-tests.md
│   ├── runtime-tests.md
│   ├── monitoring-tests.md
│   └── recovery-tests.md
│
├── 08-incident-simulation/
│   ├── scenario.md
│   ├── test-manifests/
│   ├── detection-evidence/
│   ├── investigation-notes/
│   ├── containment-actions/
│   ├── recovery-evidence/
│   └── root-cause-analysis.md
│
├── 09-compliance/
│   ├── cis-mapping.md
│   ├── nist-mapping.md
│   ├── policy-reports/
│   ├── compliance-scorecard.md
│   └── exception-register.md
│
├── 10-evidence/
│   ├── commands/
│   ├── manifests/
│   ├── logs/
│   ├── alerts/
│   ├── screenshots/
│   ├── scans/
│   ├── dashboards/
│   └── restore-tests/
│
├── 11-findings/
│   ├── findings-register.md
│   ├── risk-register.md
│   ├── remediation-roadmap.md
│   └── residual-risk.md
│
├── 12-runbooks/
│   ├── access-review.md
│   ├── compromised-pod.md
│   ├── credential-compromise.md
│   ├── runtime-alert.md
│   ├── container-escape.md
│   ├── malicious-image.md
│   ├── backup-restore.md
│   └── cluster-recovery.md
│
└── 13-report/
    ├── executive-summary.md
    ├── technical-report.md
    ├── security-scorecard.md
    ├── production-readiness.md
    ├── final-recommendation.md
    └── final-presentation.md

Project Phases

The project contains 18 phases.

Phase 1 — Project Initiation

↓

Phase 2 — Scope and Rules of Engagement

↓

Phase 3 — Environment Discovery

↓

Phase 4 — Current-State Security Assessment

↓

Phase 5 — Threat Modelling

↓

Phase 6 — Target Security Architecture

↓

Phase 7 — Identity and Access Implementation

↓

Phase 8 — Workload and Node Security

↓

Phase 9 — Network Security

↓

Phase 10 — Software Supply-Chain Security

↓

Phase 11 — Secrets and Data Protection

↓

Phase 12 — Logging and Runtime Detection

↓

Phase 13 — Security Validation

↓

Phase 14 — Incident Simulation

↓

Phase 15 — Investigation and Containment

↓

Phase 16 — Recovery and Root Cause Analysis

↓

Phase 17 — Compliance and Production Readiness

↓

Phase 18 — Final Reporting and Presentation

Phase 1 — Project Initiation

Task 1.1 — Create the Project Charter

Document:

Organisation:

CloudNova Technologies

Project:

End-to-End Kubernetes Security Capstone

Platform:

Amazon EKS

Environment:

Production and approved test environment

Project Sponsor:

Chief Information Security Officer

Project Owner:

Cloud Security

Platform Owner:

Platform Engineering

Application Owner:

Digital Services

Monitoring Owner:

Security Operations Centre

Recovery Owner:

Site Reliability Engineering

Task 1.2 — Define the Project Goals

Goals include:

Assess current Kubernetes risks.
Build the secure target state.
Implement priority controls.
Validate security controls.
Test monitoring and response.
Demonstrate secure recovery.
Produce audit evidence.
Support production approval.

Task 1.3 — Define Success Criteria

The project succeeds when:

All critical assets are inventoried.
Critical risks are identified.
Target architecture is approved.
Priority controls are implemented.
Security tests pass.
Runtime detections operate.
Incident-response workflow is tested.
Recovery is validated.
Compliance evidence is complete.
Final production recommendation is issued.

Phase 2 — Scope and Rules of Engagement

Task 2.1 — Define Scope

Include:

AWS accounts
Regions
EKS clusters
Namespaces
Workloads
IAM roles
Service Accounts
VPC
ECR repositories
Security services
Logging systems
Backup systems
CI/CD pipeline

Task 2.2 — Define Out-of-Scope Activities

Examples:

Real production exploitation
Real malware
Destructive node testing
Denial-of-Service
Credential cracking
Social engineering
Unapproved data extraction
Uncontrolled container escape testing

Task 2.3 — Define Stop Conditions

Stop and escalate if:

Active compromise is discovered
Production availability is affected
Sensitive credentials are exposed
Unknown malware is identified
Node compromise is suspected
Security tests cause instability
Production data becomes accessible unexpectedly

Phase 3 — Environment Discovery

Task 3.1 — Confirm the Kubernetes Context

kubectl config current-context

kubectl cluster-info

Task 3.2 — Inventory the Cluster

kubectl version

kubectl get nodes -o wide

kubectl get namespaces --show-labels

kubectl get deployments,statefulsets,daemonsets,jobs,cronjobs -A

Task 3.3 — Inventory Services and Exposure

kubectl get services -A

kubectl get ingress -A

Record:

LoadBalancer Services
NodePort Services
External IPs
Public hostnames
Administrative endpoints
TLS status

Task 3.4 — Inventory Identity

kubectl get serviceaccounts -A

kubectl get rolebindings -A

kubectl get clusterrolebindings

Review:

EKS access entries
Cluster administrators
Namespace users
Service Accounts
Pod Identity associations
IRSA roles
Node roles
CI/CD identities

Task 3.5 — Inventory Images

kubectl get pods -A -o json \
  | jq -r '
    .items[]
    | .metadata.namespace as $ns
    | .metadata.name as $pod
    | .spec.containers[]
    | [$ns, $pod, .name, .image]
    | @tsv
  '

Identify:

Public registries
Mutable tags
latest
Unknown repositories
Missing digests

Phase 4 — Current-State Security Assessment

Assess the following domains.

AWS Account and Governance

Account separation
Organisational guardrails
CloudTrail
GuardDuty
Security Hub
Inspector
AWS Config
Central logging

Amazon EKS

Version
API endpoint
Logging
Encryption
Access mode
Add-ons
Tags
Infrastructure as Code

Identity

Federation
MFA
Temporary access
Cluster-admin
RBAC
Service Accounts
Workload IAM
Break-glass access

Worker Nodes

Private subnets
Public IPs
AMI
Patch level
IMDSv2
IAM role
Runtime agents
Administration
Encryption

Workloads

Non-root
Privileged mode
Privilege escalation
Capabilities
Seccomp
Host namespaces
HostPath
Runtime sockets
Resource controls
Health probes

Networking

Default-deny policies
Ingress exposure
Egress controls
DNS
Security Groups
Flow Logs
WAF
Database isolation

Images

Registry
Scanning
Signing
SBOM
Provenance
Digest pinning
Admission validation

Secrets

Git exposure
Image exposure
Kubernetes Secrets
Secrets Manager
IAM access
KMS
Rotation
Logging

Logging and Runtime

Audit Logs
Application logs
Falco
GuardDuty Runtime
SIEM
Monitoring health
Detection rules

Backup and Recovery

RPO
RTO
Backup coverage
Encryption
Cross-account copies
Restore testing
Recovery runbooks

Phase 5 — Threat Modelling

Task 5.1 — Identify Assets

Include:

Kubernetes API
Administrator identities
Application Pods
Worker nodes
Container images
Service Account tokens
Workload IAM roles
Node IAM role
Secrets
Databases
Logs
Backups
CI/CD credentials

Task 5.2 — Identify Threat Actors

External attacker
Malicious insider
Compromised developer
Compromised application
Supply-chain attacker
Cloud credential thief
Ransomware operator
Third-party compromise

Task 5.3 — Define Attack Paths

Application Exploitation

Internet

↓

Vulnerable Application

↓

Pod Shell

↓

Service Account Token Access

↓

Secret Enumeration

↓

Internal Service Access

↓

Data Exfiltration

Administrator Compromise

Stolen Identity

↓

Privileged AWS Role

↓

EKS Access Entry

↓

cluster-admin

↓

Malicious Workload

Supply-Chain Compromise

Compromised Pipeline

↓

Malicious Image

↓

Trusted Registry

↓

Production Deployment

↓

Runtime Compromise

Container Escape

Privileged Pod

↓

Runtime Socket

↓

Worker Node

↓

Node IAM Credentials

↓

AWS Services

Phase 6 — Target Security Architecture

Design the target state using:

Separate production AWS account
Restricted EKS API endpoint
Private worker nodes
Federated access
EKS access entries
Namespace-scoped RBAC
Dedicated Service Accounts
Pod Identity or IRSA
Pod Security Admission
Admission policies
Default-deny networking
Approved image registries
Image scanning and signing
External secrets management
Central logging
Runtime monitoring
Tested recovery

Phase 7 — Identity and Access Implementation

Task 7.1 — Define Human Access

Enterprise Identity Provider

↓

IAM Identity Center

↓

MFA

↓

Temporary Role

↓

EKS Access Entry

↓

Kubernetes RBAC

Task 7.2 — Remove Excessive Access

Review:

kubectl get clusterrolebindings

Remove:

Unknown administrators
Unused groups
Shared identities
Unnecessary wildcard roles

Task 7.3 — Implement Namespace RBAC

Create:

Namespace administrator Role
Developer Role
Read-only Role
Auditor Role

Validate:

kubectl auth can-i list pods \
  -n payments \
  --as=<developer-identity>

kubectl auth can-i get secrets \
  -n payments \
  --as=<developer-identity>

Expected:

Pod access allowed.

Secret access denied.

Task 7.4 — Implement Workload Identity

Pod

↓

Dedicated Service Account

↓

EKS Pod Identity or IRSA

↓

Dedicated IAM Role

↓

Approved AWS Resource

Phase 8 — Workload and Node Security

Task 8.1 — Apply Pod Security Admission

Example:

apiVersion: v1
kind: Namespace
metadata:
  name: payments

  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Task 8.2 — Deploy Secure Workloads

Requirements:

Non-root
No privilege escalation
Drop capabilities
RuntimeDefault seccomp
Read-only root filesystem
Dedicated Service Account
Resource limits
Health probes
Approved image digest

Task 8.3 — Harden Nodes

Validate:

Private subnets
Approved AMI
Encrypted volumes
IMDSv2
Restricted node IAM
No unnecessary SSH
Runtime monitoring
Immutable replacement

Phase 9 — Network Security

Task 9.1 — Apply Default-Deny

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments

spec:
  podSelector: {}

  policyTypes:
    - Ingress
    - Egress

Task 9.2 — Allow Required DNS

Create explicit DNS egress.

Task 9.3 — Allow Approved Application Flows

Example:

Customer Portal

↓

Payment API

Allowed

Customer Portal

↓

Payment Database

Denied

Task 9.4 — Validate Public Exposure

Review:

kubectl get services -A

kubectl get ingress -A

Remove or restrict unapproved public endpoints.

Phase 10 — Software Supply-Chain Security

Task 10.1 — Secure the Source Repository

Require:

MFA
Protected branches
Pull-request approval
CODEOWNERS
Secret scanning
Audit logging

Task 10.2 — Secure the Pipeline

Implement:

Source Review

↓

Secret Scan

↓

Dependency Scan

↓

Infrastructure Scan

↓

Container Build

↓

Image Scan

↓

SBOM

↓

Image Signing

↓

Amazon ECR

↓

Admission Verification

Task 10.3 — Enforce Approved Images

Require:

Approved registry
Immutable digest
Passed vulnerability gate
Signature
Provenance
SBOM

Task 10.4 — Block Unsafe Images

Test:

Unsigned image
Public registry image
Mutable tag
Critical-vulnerability image

Expected:

Admission denied.

Phase 11 — Secrets and Data Protection

Task 11.1 — Store Secrets Externally

Use:

AWS Secrets Manager
AWS KMS
Workload IAM
Secrets Store CSI Driver or approved retrieval method

Task 11.2 — Restrict Secret Access

Grant each workload access only to required resources.

Task 11.3 — Validate Rotation

Confirm:

Rotation completes
Application reloads safely
Previous credential is revoked
CloudTrail records access

Task 11.4 — Protect Data

Encrypt:

EBS
EFS
Databases
Logs
Backups
Registry
Evidence

Phase 12 — Logging and Runtime Detection

Task 12.1 — Enable EKS Logs

Enable:

API Server
Audit
Authenticator
Controller Manager
Scheduler

Task 12.2 — Centralise Telemetry

EKS and AWS Sources

↓

CloudWatch and Security Services

↓

Central Log Archive

↓

SIEM

↓

SOC

Task 12.3 — Deploy Runtime Security

Use:

Falco
GuardDuty Runtime Monitoring
Equivalent approved tooling

Detect:

Unexpected shell
Reverse shell
Package-manager execution
File execution from /tmp
Credential access
Runtime socket access
Privilege escalation
Cryptomining
Agent tampering

Task 12.4 — Validate SIEM Delivery

Confirm:

Alert generated
Parsed correctly
Enriched
Routed to SOC
Runbook attached
Delivery time measured

Phase 13 — Security Validation

Create a validation matrix.

Domain	Positive Test	Negative Test	Expected Result
RBAC	Developer lists Pods	Developer reads Secrets	Secret access denied
Pod security	Secure Pod deploys	Privileged Pod deploys	Privileged Pod denied
Network	Portal reaches API	Portal reaches database	Database access denied
Image security	Signed image deploys	Unsigned image deploys	Unsigned image denied
Secrets	App reads required secret	App reads unrelated secret	Access denied
Runtime	Normal process runs	Unexpected shell starts	Alert generated
Logging	Event generated	Pipeline interrupted	Health alert generated
Backup	Restore valid backup	Restore invalid backup	Invalid restore rejected

Phase 14 — Incident Simulation

Simulation Scenario

A vulnerable test application is simulated as compromised.

The scenario includes:

Authorised Test Request

↓

Shell Started in Test Pod

↓

Harmless File Created in /tmp

↓

Non-Sensitive Test Token Path Accessed

↓

Approved External Test Destination Contacted

↓

Runtime Alert Generated

Do not use:

Real malware
Real stolen credentials
Real exfiltration
Real reverse shell infrastructure
Production data

Expected Detections

Shell execution
File activity
Sensitive-path access
Network connection
Kubernetes Audit Log event
SIEM incident creation

Phase 15 — Investigation and Containment

Task 15.1 — Triage the Alert

Identify:

Cluster
Namespace
Pod
Container
Image
Process
Parent process
Service Account
Node
Source and destination
Detection rule

Task 15.2 — Collect Evidence

Preserve:

kubectl get pod <pod-name> \
  -n <namespace> \
  -o yaml \
  > pod.yaml

kubectl logs <pod-name> \
  -n <namespace> \
  --timestamps \
  > container.log

Collect:

Pod YAML
Pod JSON
Current logs
Previous logs
Process tree
Network connections
Audit events
Runtime alert
CloudTrail
VPC Flow Logs

Task 15.3 — Determine Scope

Assess:

Container

↓

Pod

↓

Namespace

↓

Node

↓

Cluster

↓

AWS Account

Task 15.4 — Contain the Test Workload

Approved actions may include:

Apply quarantine Network Policy
Remove Pod from Service traffic
Scale workload to zero
Revoke workload IAM association
Block test destination
Preserve evidence

Phase 16 — Recovery and Root Cause Analysis

Task 16.1 — Rebuild the Workload

Use:

Trusted source
Patched dependency
Clean pipeline
Scanned image
Signed image
Immutable digest
Secure manifest

Task 16.2 — Rotate Test Credentials

Rotate all credentials accessible to the simulated workload.

Task 16.3 — Validate Recovery

Confirm:

Correct image digest
No suspicious process
No unexpected network connection
Runtime monitoring active
Logs active
Business function healthy
Security policies enforced

Task 16.4 — Perform Root Cause Analysis

Document:

Initial access
Attack path
Control failures
Root causes
Contributing factors
Blast radius
Corrective actions
Preventive actions

Example Root Cause

The test application permitted command execution because a vulnerable dependency was not blocked by the CI/CD vulnerability gate.

The impact increased because the workload had unnecessary token access and unrestricted egress.

Phase 17 — Compliance and Production Readiness

Task 17.1 — Perform CIS Review

Assess applicable controls from:

CIS Kubernetes Benchmark
CIS Amazon EKS Benchmark
CIS AWS Foundations Benchmark

Task 17.2 — Map to NIST

Map controls to applicable:

NIST Cybersecurity Framework
NIST SP 800-53

Task 17.3 — Review Exceptions

Every exception must include:

Control
Justification
Risk
Compensating controls
Owner
Approver
Expiry
Remediation plan

Task 17.4 — Complete Production Checklist

Validate:

Identity
Nodes
Workloads
Network
Images
Secrets
Logging
Runtime
Backup
Incident response
Compliance
Governance

Production Security Scorecard

Domain	Weight	Status	Evidence
AWS governance	Critical	Pass/Fail	Account assessment
EKS architecture	Critical	Pass/Fail	Cluster configuration
Identity and RBAC	Critical	Pass/Fail	Access review
Worker nodes	Critical	Pass/Fail	Node validation
Workload security	Critical	Pass/Fail	Policy report
Network security	Critical	Pass/Fail	Connectivity tests
Supply chain	High	Pass/Fail	Image evidence
Secrets	Critical	Pass/Fail	Secret-access test
Logging	Critical	Pass/Fail	Log evidence
Runtime	High	Pass/Fail	Runtime alert
Incident response	High	Pass/Fail	Exercise evidence
Recovery	High	Pass/Fail	Restore test
Compliance	Medium	Pass/Fail	Compliance report

Critical Go-Live Blockers

Production approval should be blocked when:

Unknown cluster-admin access exists
EKS API exposure is unrestricted
Static credentials remain in workloads
Worker-node IAM permissions are excessive
Privileged production workloads are unresolved
Pod Security is not enforced
Network isolation is absent
Unapproved images can deploy
Secrets are stored in Git
Audit logging is missing
Runtime coverage is incomplete
Critical alerts do not reach the SIEM
Backup restoration fails
Critical vulnerabilities remain unresolved
Security exceptions are unapproved or expired
Incident-response procedures are unavailable

Phase 18 — Final Reporting and Presentation

Executive Report Structure

1. Executive Summary

2. Business Context

3. Project Scope

4. Current Security Posture

5. Critical Risks

6. Target Security Architecture

7. Implemented Controls

8. Validation Results

9. Incident Simulation Summary

10. Business Impact

11. Compliance Status

12. Remediation Roadmap

13. Residual Risk

14. Production Recommendation

Technical Report Structure

1. Introduction

2. Scope and Rules of Engagement

3. Environment Discovery

4. Current-State Assessment

5. Threat Model

6. Target Architecture

7. Identity and Access

8. Node and Workload Security

9. Network Security

10. Software Supply Chain

11. Secrets and Data Protection

12. Logging and Runtime Security

13. Security Validation

14. Incident Simulation

15. Investigation and Containment

16. Recovery and Root Cause Analysis

17. Compliance Mapping

18. Findings and Risk Register

19. Evidence Index

20. Production Readiness

21. Final Recommendation

Executive Summary Template

CloudNova Technologies commissioned an end-to-end Kubernetes security engagement for its production Amazon EKS platform.

The engagement included environment discovery, security assessment, threat modelling, architecture design, security implementation, control validation, incident simulation, investigation, recovery and compliance review.

The target architecture uses restricted cluster access, private worker nodes, federated identities, least-privilege RBAC, pod-level AWS identities, secure workload standards, default-deny Network Policies, signed container images, external secrets management, centralised audit logging, runtime threat detection and tested recovery procedures.

The project identified critical and high-risk findings that require remediation before full production approval. The final recommendation is based on verified controls, successful security testing, residual risk and business readiness.

Findings Template

Finding ID:

Title:

Severity:

Security Domain:

Affected Resource:

Description:

Evidence:

Technical Impact:

Business Impact:

Likelihood:

Recommendation:

Owner:

Target Date:

Status:

Example Finding

Finding ID:

CAP-SEC-001

Title:

Production Workload Uses Broad Node IAM Permissions

Severity:

Critical

Affected Resource:

payments/payment-api

Description:

The payment-api workload does not use a dedicated Pod Identity or IRSA role and can access permissions assigned to the worker-node IAM role.

Evidence:

CAP-IAM-04-node-role-review.json

Technical Impact:

A compromised Pod may obtain AWS permissions intended for node operations and access unrelated AWS resources.

Business Impact:

An attacker may access sensitive services, credentials or data beyond the application’s approved scope.

Recommendation:

Implement a dedicated workload IAM role, restrict the node IAM role and monitor CloudTrail for unexpected node-role activity.

Owner:

Platform Engineering

Status:

Open

Remediation Roadmap

Immediate — 0 to 30 Days

Restrict cluster endpoint exposure.
Remove unknown administrators.
Revoke excessive credentials.
Enable audit logging.
Remediate privileged workloads.
Apply emergency network restrictions.
Close runtime-monitoring gaps.
Validate backups.

Short Term — 31 to 90 Days

Implement namespace-scoped RBAC.
Deploy workload identities.
Enforce Pod Security Admission.
Implement default-deny Network Policies.
Deploy admission policies.
Implement external secrets.
Integrate alerts with the SIEM.
Test incident runbooks.

Medium Term — 91 to 180 Days

Implement image signing and verification.
Standardise secure EKS Terraform modules.
Automate compliance evidence.
Implement just-in-time administration.
Centralise egress inspection.
Mature threat hunting.
Conduct quarterly recovery tests.

Long Term — 6 to 12 Months

Build an enterprise EKS landing zone.
Implement multi-cluster security governance.
Automate drift remediation.
Mature Zero Trust enforcement.
Implement cross-Region recovery for critical services.
Establish continuous control validation.
Build executive risk analytics.

Final Production Recommendation Options

Approved

All critical controls pass.

Approved with Conditions

Minor non-critical issues remain with owners and deadlines.

Deferred

Production approval is paused pending remediation.

Rejected

Critical risk remains unresolved.

Project Success Criteria

The capstone is complete when:

Portfolio Presentation

When presenting this capstone, explain:

Business Problem

CloudNova Technologies needed a complete security validation before launching a critical enterprise Kubernetes platform.

Your Role

You acted as the lead Kubernetes Security Engineer and Cloud Security Architect.

What You Assessed

AWS account governance
Amazon EKS
IAM and RBAC
Worker nodes
Workloads
Networking
Images
Secrets
Logging
Runtime monitoring
Recovery

What You Designed

Target security architecture
Identity model
Network segmentation
Secure workload baseline
Software supply-chain controls
Runtime monitoring
Incident-response workflows
Backup and recovery

What You Implemented

Show selected examples of:

Terraform
RBAC
Service Accounts
Network Policies
Admission policies
Secure workload manifests
Runtime rules
Monitoring configuration
Backup configuration

What You Validated

Demonstrate:

Approved access succeeds.
Unapproved access fails.
Privileged Pods are denied.
Cross-namespace traffic is blocked.
Unsigned images are denied.
Secret access follows least privilege.
Runtime alerts reach the SIEM.
Incident runbooks operate.
Backups restore successfully.

Business Outcome

Explain how the project:

Reduced attack surface
Limited identity blast radius
Prevented insecure workloads
Restricted lateral movement
Improved software trust
Increased threat visibility
Improved incident response
Strengthened compliance
Increased recovery confidence
Supported production approval

Real-World Project Scenario

CloudNova Technologies completes the final pre-production security exercise.

A controlled test application simulates suspicious activity.

The Security Operations platform detects:

Unexpected shell execution
File creation under /tmp
Access to a protected test path
Outbound network communication
Kubernetes API activity

The response team:

Validates the alert.
Identifies the affected Pod and owner.
Collects Pod metadata and logs.
Correlates the event with Kubernetes Audit Logs.
Reviews the workload IAM role.
Reviews VPC Flow Logs.
Applies a quarantine Network Policy.
Removes the workload from Service traffic.
Rotates test credentials.
Rebuilds the application.
Scans and signs the new image.
Redeploys using an immutable digest.
Validates runtime monitoring.
Performs Root Cause Analysis.
Updates the CI/CD vulnerability gate.
Updates the runtime detection rule.
Documents the result in the production-readiness report.

The final review confirms that:

Preventive controls reduced the attack path.
Detective controls generated useful alerts.
Response procedures worked.
Recovery used trusted artefacts.
Evidence supported audit and management review.

Key Takeaways

Enterprise Kubernetes security requires end-to-end integration.
Discovery and inventory are required before security decisions.
Identity and access determine the possible blast radius.
Workload, node and network controls must work together.
Software supply-chain security protects the deployment path.
Secrets should be externalised and access controlled.
Runtime monitoring detects attacks after deployment.
Incident response must be tested before production.
Recovery must use trusted images, credentials and configuration.
Root Cause Analysis converts incidents into lasting improvements.
Compliance evidence should be generated by design.
Production approval should be based on validated controls.
Technical findings must be translated into business risk.
A mature security programme continuously reassesses and improves controls.

Knowledge Check

1. Why is this project considered end to end?

Answer: It covers the complete Kubernetes security lifecycle, including discovery, assessment, architecture, implementation, validation, detection, incident response, recovery, compliance and executive reporting.

2. Why should the project include both positive and negative security tests?

Answer: Positive tests confirm that approved activity still works, while negative tests confirm that prohibited activity is blocked or detected.

3. Why is a controlled incident simulation important?

Answer: It validates monitoring, investigation, containment and recovery procedures without introducing real malicious risk.

4. Why should Root Cause Analysis follow the incident simulation?

Answer: Root Cause Analysis identifies the underlying technical and process weaknesses that allowed the simulated attack path and defines corrective and preventive actions.

5. What determines the final production recommendation?

Answer: The recommendation is based on control effectiveness, unresolved findings, residual risk, recovery readiness, compliance status and business acceptance.

Final Capstone Completion

You have completed Project 10 — End-to-End Kubernetes Security Capstone when the complete project evidence demonstrates that you can:

Assess

↓

Design

↓

Implement

↓

Validate

↓

Detect

↓

Investigate

↓

Contain

↓

Recover

↓

Report

↓

Improve

Congratulations

You have completed the Enterprise Kubernetes Security Projects module.

You now have a complete portfolio of enterprise Kubernetes security projects covering:

Secure Amazon EKS architecture
Kubernetes security assessment
CIS Kubernetes auditing
Multi-tenant security
Production security architecture
Runtime threat detection
Kubernetes SOC integration
Zero Trust
Security Operations
End-to-end security delivery

You are now prepared to apply these skills in roles such as:

Kubernetes Security Engineer
Amazon EKS Security Engineer
Cloud Security Engineer
Platform Security Engineer
DevSecOps Engineer
Container Security Engineer
Cloud Security Architect
Kubernetes Security Consultant
Cloud SOC Engineer
Incident Response Engineer

➡️ Learning Path Complete — Kubernetes Security Engineer