Project 05 — Build a Production Kubernetes Security Architecture

Project Overview

Welcome to Project 05 — Build a Production Kubernetes Security Architecture.

In this project, you will act as a Kubernetes Security Architect for CloudNova Technologies.

Your responsibility is to create a complete production security architecture that can be used as the approved reference model for enterprise Kubernetes deployments.

This project is broader than securing one cluster.

You will define how Kubernetes should be governed, deployed, accessed, monitored, operated and recovered across multiple teams, environments and business services.

Business Requirements

↓

Security Principles

↓

Reference Architecture

↓

Security Standards

↓

Implementation Roadmap

↓

Validation and Governance

The final architecture should help CloudNova Technologies deploy Kubernetes platforms that are:

Secure by default
Consistent
Scalable
Auditable
Recoverable
Suitable for sensitive workloads
Operable by enterprise platform teams

Project Mission

CloudNova Technologies has adopted Kubernetes across several business units.

Different teams currently use different approaches for:

Cluster creation
Administrator access
RBAC
Worker-node configuration
Workload security
Network segmentation
Container-image validation
Secrets management
Logging
Runtime monitoring
Backup and disaster recovery

This inconsistency creates security gaps and operational complexity.

Your mission is to:

Understand the organisation’s business and security requirements.
Assess the current Kubernetes architecture.
Define enterprise security principles.
Design the target production security architecture.
Establish mandatory and recommended security controls.
Define ownership and governance.
Create architecture diagrams and standards.
Build an implementation roadmap.
Define validation and compliance requirements.
Produce a final enterprise architecture report.

Business Scenario

CloudNova Technologies operates:

Customer-facing digital platforms
Payment-processing services
Internal administration applications
Analytics workloads
Artificial intelligence services
Development and testing environments
Shared DevSecOps services
Security monitoring platforms

The organisation uses Kubernetes in:

Multiple AWS accounts
Multiple Amazon EKS clusters
Production and non-production environments
Different business units
Different regulatory scopes

The current environment includes:

Shared and dedicated clusters
Public and private cluster endpoints
Different authentication models
Inconsistent RBAC
Broad worker-node IAM permissions
Missing Network Policies
Privileged platform workloads
Mutable container-image tags
Secrets stored in Kubernetes
Inconsistent audit logging
Limited runtime monitoring
Untested recovery plans
Security exceptions without formal governance

Target Business Outcome

The completed project should provide:

An approved Kubernetes security reference architecture
A secure-by-default cluster pattern
Standard identity and access architecture
Standard network architecture
Standard workload-security baseline
Standard software supply-chain controls
Standard logging and runtime-detection architecture
Standard backup and recovery architecture
Defined ownership and governance
A phased enterprise implementation roadmap
Measurable production-readiness criteria

Project Objectives

By completing this project, you will be able to:

Translate business requirements into security architecture
Define Kubernetes trust boundaries
Design multi-account and multi-cluster security models
Design secure administrative access
Define human and workload identity patterns
Design private and segmented cluster networking
Establish worker-node hardening standards
Define secure namespace and workload baselines
Design admission and policy enforcement
Secure container build and deployment workflows
Design enterprise secrets management
Create centralised logging and monitoring architecture
Design runtime threat detection and incident response
Define compliance and evidence requirements
Design backup and disaster-recovery controls
Produce enterprise architecture documentation
Develop a practical implementation roadmap

Project Difficulty

Level: Advanced

Estimated Time

Recommended Duration: 14–22 hours

Project Type

Enterprise security architecture
Cloud-native platform design
Governance and standards project
Threat-modelling exercise
Production-readiness design
Portfolio project

Prerequisites

You should understand:

Kubernetes architecture
Amazon EKS
AWS IAM
Kubernetes RBAC
Service Accounts
EKS Pod Identity and IRSA
Amazon VPC
Network Policies
Pod Security Admission
Admission controllers
Container registries
Image scanning and signing
Secrets management
Kubernetes audit logging
Runtime security
Incident response
Backup and disaster recovery
Security governance

Recommended Tools

You may use:

Diagramming software
Markdown
Terraform
AWS Architecture Icons
Kubernetes manifests
AWS CLI
kubectl
Kyverno
OPA Gatekeeper
Amazon ECR
AWS Secrets Manager
Amazon GuardDuty
AWS Security Hub
Amazon CloudWatch
Prometheus
Grafana
Falco
Velero
AWS Backup

Architecture Principles

The architecture should follow these principles:

Defence in Depth

Use multiple preventive, detective, responsive and recovery controls.

Least Privilege

Grant only the minimum required access to users, workloads and automation.

Zero Trust

Authenticate, authorise and inspect every important request.

Secure by Default

New clusters and namespaces should inherit approved controls automatically.

Separation of Duties

Separate platform administration, security operations, application ownership and compliance responsibilities.

Automation First

Use Infrastructure as Code, Policy as Code and GitOps.

Immutable Infrastructure

Replace compromised or outdated infrastructure rather than manually repairing it.

Assume Breach

Design controls that limit the impact of a compromised Pod, identity or node.

Evidence by Design

Every critical security control should generate auditable evidence.

Required Deliverables

Architecture Deliverables

Enterprise Kubernetes high-level architecture
AWS account and organisational architecture
Cluster topology
Network architecture
Administrative access architecture
Workload identity architecture
Namespace and tenancy architecture
Container supply-chain architecture
Secrets-management architecture
Logging and monitoring architecture
Runtime security architecture
Incident-response architecture
Backup and disaster-recovery architecture
Trust-boundary diagram
Data-flow diagram

Security Deliverables

Security principles
Threat model
Security control catalogue
Mandatory production baseline
Identity standard
Network standard
Workload-security standard
Image-security standard
Secrets-management standard
Logging standard
Runtime-detection standard
Backup and recovery standard
Exception-management process
Compliance mapping

Operational Deliverables

RACI matrix
Cluster lifecycle process
Namespace onboarding process
Production-readiness review
Access-review process
Patch and upgrade process
Incident-response process
Backup and recovery process
Monitoring ownership model
Security validation plan

Management Deliverables

Executive summary
Current-state risk overview
Target-state architecture
Strategic security priorities
Investment considerations
Implementation roadmap
Residual-risk statement
Final architecture recommendation

Recommended Project Folder Structure

05-production-kubernetes-security-architecture/

├── README.md
│
├── 01-requirements/
│   ├── business-requirements.md
│   ├── security-requirements.md
│   ├── compliance-requirements.md
│   ├── assumptions.md
│   └── constraints.md
│
├── 02-current-state/
│   ├── environment-overview.md
│   ├── current-risks.md
│   ├── architecture-gaps.md
│   └── maturity-assessment.md
│
├── 03-architecture/
│   ├── high-level-architecture.md
│   ├── account-architecture.md
│   ├── cluster-architecture.md
│   ├── network-architecture.md
│   ├── identity-architecture.md
│   ├── workload-architecture.md
│   ├── supply-chain-architecture.md
│   ├── secrets-architecture.md
│   ├── logging-architecture.md
│   ├── runtime-architecture.md
│   ├── incident-response-architecture.md
│   └── backup-dr-architecture.md
│
├── 04-standards/
│   ├── cluster-security-standard.md
│   ├── identity-standard.md
│   ├── network-standard.md
│   ├── workload-standard.md
│   ├── image-security-standard.md
│   ├── secrets-standard.md
│   ├── logging-standard.md
│   ├── runtime-standard.md
│   ├── backup-standard.md
│   └── exception-standard.md
│
├── 05-governance/
│   ├── operating-model.md
│   ├── raci.md
│   ├── cluster-lifecycle.md
│   ├── namespace-onboarding.md
│   ├── access-review.md
│   ├── exception-management.md
│   └── production-readiness.md
│
├── 06-threat-model/
│   ├── assets.md
│   ├── threat-actors.md
│   ├── attack-paths.md
│   ├── trust-boundaries.md
│   └── control-mapping.md
│
├── 07-validation/
│   ├── architecture-validation.md
│   ├── control-validation-matrix.md
│   ├── security-test-plan.md
│   └── compliance-evidence.md
│
├── 08-roadmap/
│   ├── immediate-actions.md
│   ├── short-term-roadmap.md
│   ├── medium-term-roadmap.md
│   └── long-term-strategy.md
│
└── 09-report/
    ├── executive-summary.md
    ├── architecture-report.md
    ├── residual-risk.md
    └── final-recommendation.md

Project Phases

The project contains 15 phases.

Phase 1 — Requirements and Scope

↓

Phase 2 — Current-State Assessment

↓

Phase 3 — Threat Modelling

↓

Phase 4 — Governance and Account Architecture

↓

Phase 5 — Cluster Architecture

↓

Phase 6 — Identity and Access Architecture

↓

Phase 7 — Network Security Architecture

↓

Phase 8 — Worker-Node and Workload Architecture

↓

Phase 9 — Software Supply-Chain Architecture

↓

Phase 10 — Secrets and Data Protection

↓

Phase 11 — Logging, Monitoring and Runtime Security

↓

Phase 12 — Incident Response and Recovery

↓

Phase 13 — Compliance and Governance

↓

Phase 14 — Architecture Validation

↓

Phase 15 — Implementation Roadmap and Reporting

Phase 1 — Requirements and Scope

Task 1.1 — Define Project Scope

Document:

Organisation:

CloudNova Technologies

Project:

Production Kubernetes Security Architecture

Cloud Platform:

Amazon Web Services

Kubernetes Platform:

Amazon EKS

Environments:

Production
Pre-Production
Development
Testing

Business Units:

Digital Services
Payments
Finance
Human Resources
Data and Analytics
Artificial Intelligence

Architecture Owner:

Cloud Security Architecture

Platform Owner:

Platform Engineering

Operational Owner:

Site Reliability Engineering

Security Monitoring Owner:

Security Operations Centre

Task 1.2 — Define Business Requirements

Examples:

The platform must support multiple business units.
Production applications must operate across multiple Availability Zones.
Sensitive workloads must use stronger isolation controls.
Platform access must use enterprise federation.
Production infrastructure must be reproducible.
Security telemetry must be centralised.
Critical workloads must meet approved recovery objectives.
The architecture must support regulatory audits.
Platform standards must be reusable across clusters.

Task 1.3 — Define Security Requirements

ID	Requirement	Priority
ARCH-001	Production clusters must use approved AWS accounts	Critical
ARCH-002	Administrative access must use federation and MFA	Critical
ARCH-003	Production worker nodes must use private subnets	Critical
ARCH-004	Kubernetes API access must be private or restricted	Critical
ARCH-005	Permanent cluster-admin access must be minimised	Critical
ARCH-006	Workloads must use pod-level AWS identities	Critical
ARCH-007	Production namespaces must enforce Pod security	Critical
ARCH-008	Default-deny networking must be implemented	High
ARCH-009	Container images must be scanned and verified	High
ARCH-010	Secrets must use approved external management	Critical
ARCH-011	Audit logging must be centralised	Critical
ARCH-012	Runtime detection must cover production workloads	High
ARCH-013	Recovery procedures must be tested	High
ARCH-014	Security exceptions must expire	High

Task 1.4 — Identify Constraints

Possible constraints include:

Existing shared clusters
Legacy applications requiring root
Limited private connectivity
Existing CI/CD tools
Compliance requirements
Budget
Operational maturity
Third-party platform dependencies
Application migration timelines

Task 1.5 — Define Success Criteria

The project succeeds when:

Target architecture is approved.
Mandatory controls are documented.
Ownership is defined.
Threats are mapped to controls.
Validation criteria exist.
Implementation priorities are approved.
Residual risks are documented.

Phase 2 — Current-State Assessment

Task 2.1 — Inventory the Environment

Create an inventory of:

AWS accounts
EKS clusters
Kubernetes versions
Regions
VPCs
Node groups
Namespaces
Workloads
Public endpoints
IAM roles
Service Accounts
Security tools
Backup tools

Task 2.2 — Identify Current-State Risks

Examples include:

Production and development share accounts.
Public EKS endpoints allow broad CIDRs.
Cluster-admin is permanently assigned.
Workloads inherit node IAM permissions.
Namespaces have no default-deny policies.
Privileged Pods are permitted.
Images use latest.
Secrets are stored in manifests.
Runtime monitoring is incomplete.
Backups are not tested.

Task 2.3 — Perform Maturity Assessment

Domain	Current Maturity	Target Maturity
Governance	Initial	Managed
Identity	Developing	Optimised
Networking	Developing	Managed
Workloads	Initial	Managed
Supply chain	Developing	Managed
Runtime detection	Initial	Managed
Recovery	Developing	Managed
Compliance	Initial	Continuous

Task 2.4 — Create a Gap Register

Gap ID:

Domain:

Current State:

Target State:

Risk:

Priority:

Owner:

Dependency:

Phase 3 — Threat Modelling

Task 3.1 — Identify Critical Assets

Assets include:

Kubernetes API
Cluster administrator identities
Worker nodes
Application Pods
Customer data
Payment data
Container images
CI/CD credentials
Service Account tokens
IAM roles
Secrets
Audit logs
Backups
KMS keys

Task 3.2 — Identify Threat Actors

Consider:

External attackers
Malicious insiders
Compromised developers
Compromised workloads
Supply-chain attackers
Ransomware groups
Cloud credential thieves
Third-party compromise

Task 3.3 — Define Trust Boundaries

Internet

↓

AWS Account

↓

VPC

↓

EKS Cluster

↓

Namespace

↓

Pod

↓

Container Process

Also include:

CI/CD boundary
Registry boundary
Identity-provider boundary
Security-tooling boundary
Backup-account boundary

Task 3.4 — Create Attack Paths

Attack Path 1 — Application Compromise

Internet

↓

Vulnerable Application

↓

Pod Compromise

↓

Service Account Token Access

↓

Kubernetes API Abuse

↓

Secret Theft

↓

Lateral Movement

Attack Path 2 — Compromised Administrator

Stolen Enterprise Credentials

↓

Privileged AWS Role

↓

EKS Access Entry

↓

cluster-admin

↓

Malicious Workload Deployment

Attack Path 3 — Supply-Chain Compromise

Compromised Build Runner

↓

Malicious Container Image

↓

Trusted Registry

↓

Production Deployment

↓

Runtime Compromise

Attack Path 4 — Node Compromise

Privileged Pod

↓

Runtime Socket Access

↓

Worker Node Compromise

↓

Node IAM Credential Theft

↓

AWS Resource Access

Task 3.5 — Map Security Controls

Threat	Preventive Control	Detective Control	Response Control
Admin compromise	Federation, MFA, JIT access	Authenticator logs, CloudTrail	Revoke session and access entry
Pod compromise	Hardening, patching, Network Policies	Runtime monitoring	Quarantine Pod
Malicious image	Scanning, signing, admission verification	Registry and runtime monitoring	Block digest
Node compromise	Privileged-workload restrictions	Node and runtime telemetry	Cordon and replace node
Secret theft	External secrets and least privilege	Audit logs and CloudTrail	Rotate credentials

Phase 4 — Governance and Account Architecture

Task 4.1 — Define the AWS Account Model

AWS Organization

├── Security OU
│   ├── Security Tooling Account
│   └── Log Archive Account
│
├── Infrastructure OU
│   ├── Network Account
│   ├── Shared Services Account
│   └── CI/CD Account
│
├── Production OU
│   ├── Customer Platform Account
│   ├── Payments Account
│   └── Data Platform Account
│
└── Non-Production OU
    ├── Development Account
    ├── Testing Account
    └── Sandbox Account

Task 4.2 — Define Account Isolation Rules

Use separate accounts for:

Production and non-production
Regulated workloads
Security tooling
Central logging
Shared networking
CI/CD
Backup or recovery where required

Task 4.3 — Define Organisational Guardrails

Examples:

Prevent CloudTrail disablement
Prevent GuardDuty disablement
Restrict unsupported Regions
Prevent public storage
Protect backup vaults
Restrict unauthorised IAM changes
Require approved resource tags
Restrict internet-facing infrastructure

Task 4.4 — Define Cluster Strategy

Possible patterns:

Pattern	Use Case
Shared cluster	Standard internal applications
Dedicated cluster	Sensitive or regulated workloads
Cluster per environment	Strong production separation
Cluster per business unit	Independent ownership
Dedicated platform cluster	Shared platform tooling

Task 4.5 — Define Cluster Classification

Classification	Example	Security Requirement
Critical Production	Payments	Highest
Standard Production	Customer portal	High
Internal Production	HR application	High
Pre-Production	UAT	Moderate
Development	Developer workloads	Standard
Sandbox	Experimentation	Restricted connectivity

Phase 5 — Cluster Architecture

Task 5.1 — Define the Production Cluster Pattern

Production AWS Account

↓

Dedicated VPC

↓

Private Application Subnets

↓

Amazon EKS Control Plane

↓

Private Managed Node Groups

↓

Security and Platform Add-Ons

↓

Application Namespaces

Task 5.2 — Define Cluster API Access

Preferred model:

Administrator

↓

Enterprise Identity Provider

↓

IAM Identity Center

↓

MFA

↓

Temporary Role

↓

Approved Private Network

↓

Private EKS Endpoint

If public access is required:

Restrict CIDRs.
Require federation.
Require MFA.
Monitor access.
Document the exception.

Task 5.3 — Define Cluster Logging

Enable:

API Server
Audit
Authenticator
Controller Manager
Scheduler

Forward logs to:

CloudWatch Logs

↓

Central Logging Account

↓

Enterprise SIEM

Task 5.4 — Define Cluster Add-Ons

Approved add-ons may include:

Amazon VPC CNI
CoreDNS
kube-proxy
EBS CSI Driver
EFS CSI Driver
AWS Load Balancer Controller
EKS Pod Identity Agent
Policy engine
Runtime security agent
Logging agent
Metrics collector
GitOps controller
Backup agent

Task 5.5 — Define Add-On Governance

Each add-on must have:

Owner
Version
Approved source
IAM role
RBAC permissions
Availability requirement
Monitoring
Upgrade plan
Rollback procedure

Phase 6 — Identity and Access Architecture

Task 6.1 — Define Human Identity Flow

Enterprise Identity Provider

↓

AWS IAM Identity Center

↓

MFA

↓

Temporary AWS Role

↓

EKS Access Entry

↓

Kubernetes Group

↓

RBAC Role or ClusterRole

Task 6.2 — Define Enterprise Access Roles

Role	Scope
Platform Administrator	Controlled cluster-wide platform access
Cloud Security Engineer	Security review and incident response
Application Developer	Namespace-scoped application access
SOC Analyst	Read-only security visibility
Compliance Auditor	Read-only evidence access

Task 6.3 — Define Privileged Access

Privileged access must be:

Just in time
Time limited
Approved
Logged
Reviewed
Automatically removed

Task 6.4 — Define Break-Glass Access

Break-glass access must include:

Strong authentication
Restricted custodians
Immediate alerting
Session logging
Documented justification
Automatic expiry
Post-use review

Task 6.5 — Define Workload Identity

Application Pod

↓

Dedicated Service Account

↓

EKS Pod Identity or IRSA

↓

Dedicated IAM Role

↓

Approved AWS Resource

Task 6.6 — Define Identity Standards

Mandatory requirements:

No shared administrator identities
No permanent user access keys
No long-lived Kubernetes tokens
No application access through node roles
Dedicated Service Accounts
Resource-specific IAM permissions
Quarterly access reviews
CloudTrail monitoring

Phase 7 — Network Security Architecture

Task 7.1 — Define the VPC Pattern

Amazon VPC

├── Public Subnets
│   ├── Public Load Balancers
│   └── NAT Gateways
│
├── Private Application Subnets
│   ├── Worker Nodes
│   └── Application Pods
│
└── Restricted Data Subnets
    ├── Databases
    ├── Caches
    └── Internal Services

Task 7.2 — Define Node Placement

Worker nodes should:

Use private subnets
Avoid public IPs
Span multiple Availability Zones
Use controlled egress
Use least-privilege Security Groups
Use VPC endpoints where practical

Task 7.3 — Define Ingress Architecture

Internet

↓

CloudFront or Approved Entry Layer

↓

AWS WAF

↓

Application Load Balancer

↓

Approved Ingress Controller

↓

Application Service

↓

Application Pods

Ingress requirements:

TLS
Approved certificates
Access logging
WAF where required
Approved ports
Approved IngressClass
Restricted administrative endpoints
Authentication where required

Task 7.4 — Define Egress Architecture

Application Pod

↓

Network Policy

↓

Security Group or Egress Gateway

↓

Network Firewall or Proxy

↓

Approved Destination

Egress controls should:

Block arbitrary outbound traffic
Permit required AWS services
Use VPC endpoints
Log network flows
Filter DNS
Support incident blocking

Task 7.5 — Define Namespace Network Baseline

Every production namespace should have:

Default-Deny Ingress

+

Default-Deny Egress

+

Explicit DNS Access

+

Explicit Application Flows

+

Explicit Monitoring Flows

Task 7.6 — Define Network Telemetry

Enable:

VPC Flow Logs
Route 53 Resolver logs
Load balancer logs
WAF logs
Network Firewall logs
Service-mesh telemetry where used

Phase 8 — Worker-Node and Workload Architecture

Task 8.1 — Define Node Security Baseline

Production worker nodes should use:

Managed Node Groups where suitable
Approved EKS-optimised or hardened images
Private subnets
Encrypted EBS volumes
IMDSv2
Restricted metadata hop limits
Least-privilege node IAM
Restricted administration
Central logging
Runtime monitoring
Immutable replacement

Task 8.2 — Define Node-Group Segmentation

System Node Group

├── CoreDNS
├── Monitoring
└── Policy Controllers

Standard Application Node Group

Sensitive Workload Node Group

Approved Privileged Tooling Node Group

Use:

Taints
Tolerations
Node selectors
Affinity
Dedicated Security Groups
Dedicated IAM roles
Separate scaling policies

Task 8.3 — Define Namespace Baseline

Every namespace should include:

Owner label
Environment label
Data-classification label
Pod Security labels
ResourceQuota
LimitRange
Default-deny Network Policies
Dedicated Service Accounts
Namespace-scoped RBAC
Logging metadata

Task 8.4 — Define Secure Workload Standard

Production workloads should:

Run as non-root
Disable privilege escalation
Drop unnecessary capabilities
Use RuntimeDefault seccomp
Use read-only root filesystems where possible
Avoid privileged mode
Avoid host namespaces
Avoid HostPath
Avoid runtime-socket mounts
Define requests and limits
Use health probes
Use dedicated Service Accounts
Use approved images by digest

Task 8.5 — Define Pod Security Admission

Recommended rollout:

Warn

↓

Audit

↓

Remediate

↓

Enforce

Use the Restricted profile where workloads are compatible.

Task 8.6 — Define Admission Policies

Mandatory policies should validate:

Approved registries
Image digests
Image signatures
Non-root execution
No privileged containers
No HostPath
No host namespaces
No runtime sockets
Resource requests and limits
Required ownership labels
Approved Ingress classes
Restricted public Services

Phase 9 — Software Supply-Chain Architecture

Task 9.1 — Define Secure Delivery Flow

Developer

↓

Source Repository

↓

Pull Request Review

↓

Secret Scanning

↓

Dependency Scanning

↓

Static Analysis

↓

Infrastructure Scanning

↓

Container Build

↓

Image Scanning

↓

SBOM Generation

↓

Image Signing

↓

Amazon ECR

↓

Admission Verification

↓

Amazon EKS

Task 9.2 — Define Source-Control Requirements

Individual identities
MFA
Protected branches
Pull-request approval
CODEOWNERS
Secret scanning
Restricted automation tokens
Audit logging

Task 9.3 — Define CI/CD Identity

Pipelines should use:

Temporary credentials
Workload federation
Environment-specific roles
Least privilege
Protected production approvals

Task 9.4 — Define Registry Security

Amazon ECR requirements:

Private repositories
Encryption
Tag immutability
Image scanning
Lifecycle policies
Restricted repository access
CloudTrail monitoring
Replication where required

Task 9.5 — Define Image Verification

Production deployment requires:

Approved registry
Approved image digest
Passed vulnerability gate
SBOM
Signature
Provenance
Admission verification

Phase 10 — Secrets and Data Protection

Task 10.1 — Define Secrets Architecture

Application Pod

↓

Dedicated Service Account

↓

EKS Pod Identity or IRSA

↓

AWS Secrets Manager

↓

AWS KMS

↓

Read-Only Secret Delivery

Task 10.2 — Define Secret Standards

No plaintext Secrets in Git
No embedded credentials in images
No static AWS access keys
Workload-specific secret access
Rotation
CloudTrail logging
Emergency revocation
Least-privilege KMS access

Task 10.3 — Define Data-at-Rest Controls

Encrypt:

Worker-node volumes
Persistent volumes
Databases
Logs
Backups
Container registries
Evidence repositories

Task 10.4 — Define Data-in-Transit Controls

Use:

TLS ingress
Encrypted database connections
Service-to-service TLS where required
Secure AWS endpoints
Managed certificate lifecycles

Task 10.5 — Define KMS Governance

Document:

Key owner
Key administrators
Key users
Rotation
Deletion protection
Cross-account access
Monitoring
Separation of duties

Phase 11 — Logging, Monitoring and Runtime Security

Task 11.1 — Define Logging Sources

Collect:

EKS control-plane logs
Kubernetes Audit Logs
Application logs
Node logs
containerd logs
Admission-controller logs
Runtime alerts
CloudTrail
VPC Flow Logs
Load balancer logs
DNS logs
GuardDuty findings
Inspector findings

Task 11.2 — Define Central Logging Architecture

EKS Clusters

↓

CloudWatch Logs and Collectors

↓

Central Log Archive Account

↓

Security Data Platform

↓

Enterprise SIEM

↓

SOC

Task 11.3 — Define Monitoring Architecture

Use:

CloudWatch
Prometheus
Amazon Managed Service for Prometheus
Grafana
Amazon Managed Grafana
Application tracing where required

Task 11.4 — Define Runtime Security

Use:

Amazon GuardDuty Runtime Monitoring
Falco
eBPF-based tooling
Equivalent approved runtime platform

Detect:

Shell execution
Reverse shells
Malware
Cryptomining
Sensitive file access
Runtime socket access
Privilege escalation
Container escape indicators
Security-agent tampering

Task 11.5 — Define High-Value Alerts

Alerts should cover:

New cluster-admin binding
Secret access
pods/exec
Privileged Pod deployment
Public LoadBalancer creation
Network Policy deletion
Logging disablement
Runtime-agent deletion
Malicious image deployment
Suspicious AWS API activity

Task 11.6 — Define Monitoring Ownership

Every alert must have:

Owner
Severity
Threshold
Escalation
Runbook
Response target
Validation schedule

Phase 12 — Incident Response and Recovery

Task 12.1 — Define Incident-Response Architecture

Security Event

↓

SIEM Correlation

↓

SOC Triage

↓

Cloud Security Investigation

↓

Platform Containment

↓

Application Recovery

↓

Root Cause Analysis

Task 12.2 — Define Required Runbooks

Create runbooks for:

Compromised Pod
Privileged workload
Suspicious Secret access
Workload credential compromise
Container escape
Malicious image
Worker-node compromise
Cryptomining
Audit-log failure
Public exposure
Cluster-administrator compromise

Task 12.3 — Define Containment Capabilities

Containment options include:

Apply quarantine Network Policy
Remove workload from Service
Scale Deployment to zero
Revoke Pod Identity association
Remove public ingress
Block malicious digest
Cordon node
Replace worker node
Rotate credentials

Task 12.4 — Define Backup Architecture

Kubernetes Resources

↓

GitOps and Velero

Persistent Data

↓

EBS, EFS and Database Backups

↓

AWS Backup Vault

↓

Cross-Account Copy

↓

Cross-Region Copy

↓

Tested Recovery

Task 12.5 — Define Recovery Requirements

Each critical workload must have:

RPO
RTO
Recovery owner
Backup schedule
Retention
Clean recovery point
Recovery runbook
Restore-test schedule

Task 12.6 — Define Disaster-Recovery Strategy

Select by workload:

Strategy	Typical Use
Backup and restore	Standard workloads
Pilot light	Important workloads
Warm standby	Critical workloads
Active-active	Very high availability services

Phase 13 — Compliance and Governance

Task 13.1 — Define Control Frameworks

Map architecture to applicable frameworks such as:

CIS Kubernetes Benchmark
CIS Amazon EKS Benchmark
CIS AWS Foundations
NIST Cybersecurity Framework
NIST SP 800-53
ISO 27001
PCI DSS
SOC 2
Internal standards

Task 13.2 — Define Continuous Compliance

Security Standard

↓

Infrastructure as Code

↓

Policy as Code

↓

Automated Validation

↓

Central Findings

↓

Evidence Repository

↓

Compliance Reporting

Task 13.3 — Define Exception Management

Every exception should include:

Control:

Affected Resource:

Business Justification:

Security Risk:

Compensating Controls:

Owner:

Approver:

Expiry Date:

Remediation Plan:

Task 13.4 — Define Architecture Governance

Required review gates:

Cluster Request

↓

Business Classification

↓

Architecture Review

↓

Security Review

↓

Implementation

↓

Production Readiness Review

↓

Approval

↓

Continuous Monitoring

Task 13.5 — Define Security Metrics

Examples:

Metric	Target
Production clusters using approved baseline	100%
Clusters with audit logging	100%
Production namespaces with Restricted PSA	100%
Unapproved cluster-admin bindings	0
Production namespaces without Network Policies	0
Static AWS credentials in workloads	0
Critical image vulnerabilities beyond SLA	0
Production clusters connected to SIEM	100%
Tested production recovery plans	100%
Expired security exceptions	0

Phase 14 — Architecture Validation

Task 14.1 — Create a Control Validation Matrix

Control	Positive Test	Negative Test	Monitoring Test
RBAC	Developer lists own Pods	Developer cannot read Secrets	Audit event recorded
Pod security	Secure Pod deploys	Privileged Pod denied	Policy alert recorded
Networking	Approved service flow works	Cross-tenant flow denied	Flow logs available
Images	Signed image deploys	Unsigned image denied	Admission event sent
Runtime	Normal process runs	Unexpected shell alerts	SIEM receives finding
Secrets	App retrieves required secret	Other secret denied	CloudTrail records access
Backup	Approved restore succeeds	Invalid recovery point rejected	Backup failure alert sent

Task 14.2 — Validate Attack Paths

For each threat-modelled path, identify:

Preventive control
Detective control
Response control
Recovery control
Evidence source

Task 14.3 — Perform Failure-Mode Review

Assess what happens when:

Identity provider is unavailable
Admission controller fails
Runtime agent stops
Log pipeline fails
DNS fails
CNI fails
Region fails
KMS key is disabled
Backup job fails
SIEM integration stops

Task 14.4 — Validate Operational Feasibility

Confirm that the architecture:

Supports developer workflows
Supports emergency access
Supports upgrades
Supports scaling
Supports incident response
Supports recovery
Has clear ownership
Is financially sustainable

Phase 15 — Implementation Roadmap and Reporting

Task 15.1 — Define Immediate Priorities

0–30 Days

Restrict public EKS endpoints.
Remove unknown cluster-admin access.
Enable audit logging.
Remove static credentials.
Block privileged production workloads.
Rotate exposed Secrets.
Enable runtime monitoring for critical clusters.

Task 15.2 — Define Short-Term Priorities

31–90 Days

Implement EKS access entries.
Introduce pod-level AWS identity.
Apply namespace Pod Security labels.
Deploy default-deny Network Policies.
Standardise secure workload templates.
Integrate logs with SIEM.
Test backup restoration.
Implement formal exception management.

Task 15.3 — Define Medium-Term Priorities

91–180 Days

Implement image signing and verification.
Build secure EKS Terraform modules.
Implement central egress controls.
Establish just-in-time privileged access.
Automate compliance evidence.
Standardise node-group patterns.
Conduct incident-response exercises.

Task 15.4 — Define Long-Term Strategy

6–12 Months

Implement a reusable EKS landing zone.
Standardise multi-cluster governance.
Introduce continuous architecture validation.
Build enterprise Zero Trust controls.
Automate drift remediation.
Implement cross-Region recovery for critical services.
Mature security metrics and executive reporting.

Production Kubernetes Reference Architecture

AWS Organization

├── Security Tooling Account
├── Log Archive Account
├── Network Account
├── CI/CD Account
├── Production Workload Accounts
└── Non-Production Accounts

Enterprise Identity Provider

↓

IAM Identity Center

↓

MFA and Temporary Roles

↓

Approved Administrative Network

↓

Private Amazon EKS API

↓

EKS Access Entries

↓

Kubernetes RBAC

↓

Admission Controls

↓

Private Worker Nodes

├── System Node Groups
├── Application Node Groups
└── Sensitive Workload Node Groups

↓

Secure Namespaces

├── Pod Security Admission
├── ResourceQuota
├── LimitRange
├── Default-Deny Network Policies
├── Dedicated Service Accounts
└── Workload IAM Roles

↓

Secure Software Supply Chain

├── Scanning
├── SBOM
├── Signing
└── Verification

↓

External Secrets and Encryption

↓

Runtime Detection and Observability

↓

Central Security Account and SIEM

↓

Incident Response and Recovery

Mandatory Production Security Baseline

Governance

Approved AWS account
Approved Region
Documented owner
Infrastructure as Code
Security exception process

Identity

Federation and MFA
Temporary credentials
EKS access entries
Least-privilege RBAC
JIT privileged access
Pod-level identities

Network

Private nodes
Restricted API endpoint
Least-privilege Security Groups
Default-deny Network Policies
Controlled egress
Network telemetry

Workloads

Non-root
No privilege escalation
Capabilities dropped
Seccomp
No privileged mode
No HostPath
Resource controls
Dedicated Service Accounts

Supply Chain

Protected repositories
Security scanning
SBOM
Image signing
Private registry
Immutable digests
Admission verification

Secrets

External secret store
Resource-specific IAM
KMS encryption
Rotation
No plaintext credentials

Detection

Control-plane logging
Kubernetes Audit Logs
CloudTrail
Runtime monitoring
SIEM integration
Tested alerts

Recovery

Defined RPO and RTO
Protected backups
Cross-account copies
Recovery runbooks
Restore testing

Architecture Decision Record Template

Decision ID:

Title:

Status:

Context:

Decision:

Alternatives Considered:

Security Benefits:

Operational Impact:

Cost Impact:

Risks:

Compensating Controls:

Owner:

Review Date:

Security Standard Template

Standard Name:

Purpose:

Scope:

Mandatory Requirements:

Recommended Requirements:

Exceptions:

Validation Method:

Evidence Required:

Control Owner:

Review Frequency:

Architecture Review Checklist

Architecture Risks to Avoid

One Shared Production Account

Risk: Large blast radius and weak separation.

Architecture Response: Use workload and environment account boundaries.

Permanent cluster-admin Access

Risk: Compromised identity gains complete cluster control.

Architecture Response: Use JIT privileged access and limited break-glass roles.

Public Cluster Endpoint Without Restriction

Risk: Expanded attack surface.

Architecture Response: Use private endpoints or approved CIDRs.

Applications Using Node IAM Roles

Risk: Compromised Pods obtain broad AWS access.

Architecture Response: Use Pod Identity or IRSA.

Missing Default-Deny Networking

Risk: Unrestricted lateral movement.

Architecture Response: Apply namespace network baselines.

Mutable Image Tags

Risk: Unapproved image content can replace trusted workloads.

Architecture Response: Use signed digest-pinned images.

Secrets in Git

Risk: Credentials remain exposed in repository history.

Architecture Response: Use approved external secret stores.

Unmonitored Runtime Activity

Risk: Post-deployment attacks remain undetected.

Architecture Response: Use managed and open-source runtime detection.

Untested Backups

Risk: Recovery fails during an incident.

Architecture Response: Require scheduled restore testing.

Permanent Exceptions

Risk: Temporary weaknesses become standard practice.

Architecture Response: Require owner, expiry and remediation plan.

Executive Summary Template

CloudNova Technologies commissioned the development of a production Kubernetes security architecture to standardise and strengthen security across its Amazon EKS environments.

The architecture defines approved controls for AWS account isolation, administrative access, workload identity, network segmentation, worker-node hardening, Pod security, software supply-chain protection, secrets management, logging, runtime detection, incident response and disaster recovery.

The target design uses federated identities, restricted cluster endpoints, private worker nodes, pod-level AWS identities, default-deny Network Policies, secure workload standards, signed container images, external secret storage, centralised security telemetry and tested recovery procedures.

The implementation roadmap prioritises immediate remediation of critical access, logging, workload and network risks, followed by the development of a reusable secure EKS landing zone and continuous compliance programme.

Final Architecture Report Structure

1. Executive Summary

2. Project Scope

3. Business Requirements

4. Security Requirements

5. Assumptions and Constraints

6. Current-State Assessment

7. Threat Model

8. Security Principles

9. AWS Account Architecture

10. Cluster Architecture

11. Identity Architecture

12. Network Architecture

13. Node and Workload Architecture

14. Software Supply-Chain Architecture

15. Secrets and Data Protection

16. Logging and Runtime Security

17. Incident Response

18. Backup and Disaster Recovery

19. Compliance and Governance

20. Security Standards

21. Validation Plan

22. Implementation Roadmap

23. Residual Risk

24. Final Recommendation

Project Success Criteria

The project is complete when:

Portfolio Presentation

When presenting this project, explain:

Business Problem

CloudNova Technologies needed a consistent and scalable security architecture for Kubernetes environments operated by multiple teams.

Your Role

You acted as the Kubernetes Security Architect.

What You Designed

Multi-account AWS security model
Cluster strategy
Federated identity architecture
Private network architecture
Worker-node security standard
Namespace and workload baseline
Secure container supply chain
External secrets architecture
Runtime and SIEM architecture
Incident response and disaster recovery

How You Reduced Risk

Explain how the architecture:

Reduced administrative exposure
Limited workload permissions
Restricted lateral movement
Prevented unsafe deployments
Improved software provenance
Increased threat visibility
Improved response capability
Established recovery readiness

What You Delivered

Present:

Architecture diagrams
Threat model
Security standards
RACI matrix
Control catalogue
Validation matrix
Implementation roadmap
Executive report

Real-World Project Outcome

After completing this project, CloudNova Technologies has:

A standard production Kubernetes security architecture
A documented AWS account and cluster strategy
A secure administrative access pattern
A workload identity standard
A private and segmented network design
A worker-node hardening baseline
A secure namespace and workload standard
A controlled container software supply chain
An external secrets-management architecture
Centralised logging and runtime monitoring
Defined incident-response processes
Tested recovery requirements
Continuous compliance expectations
A formal exception-management process
A phased implementation roadmap

Key Takeaways

Production Kubernetes security begins with architecture and governance.
Security controls must be consistent across clusters and teams.
AWS accounts and clusters should be used as deliberate isolation boundaries.
Human and workload identities require different security patterns.
Private networking and default-deny controls reduce attack paths.
Worker nodes should be hardened, private and replaceable.
Pod Security Admission and policy engines enforce secure workloads.
Container-image trust requires scanning, signing and verification.
Secrets should be managed externally using least-privilege identities.
Runtime monitoring complements preventive controls.
Security telemetry must support enterprise SOC operations.
Incident response and recovery must be designed before an incident occurs.
Compliance should be continuously measured.
Exceptions must be controlled and time limited.
A practical roadmap is required to move from current state to target state.

Knowledge Check

1. Why is a reference architecture important for enterprise Kubernetes?

Answer: It provides a consistent, approved design that teams can reuse, reducing configuration drift, duplicated effort and security gaps.

2. Why should AWS accounts and Kubernetes clusters be used as isolation boundaries?

Answer: They limit blast radius, separate administrative control, support compliance scoping and prevent one environment from unnecessarily affecting another.

3. Why should human identity and workload identity be designed separately?

Answer: Human users require federation, MFA and temporary administrative access, while workloads require dedicated Service Accounts and least-privilege pod-level cloud identities.

4. Why are both preventive and detective controls required?

Answer: Preventive controls block known unsafe activity, while detective controls identify compromise, misuse or attacks that occur after deployment.

5. Why must the implementation roadmap be risk based?

Answer: A risk-based roadmap ensures that critical exposure, excessive access and missing detection controls are addressed before lower-priority improvements.

Project Completion

You have completed Project 05 — Build a Production Kubernetes Security Architecture when the target architecture, standards, threat model, validation plan, governance model, implementation roadmap and final report satisfy the project success criteria.

➡️ Next Project: Project 06 — Runtime Threat Detection Platform