Runbook 01 — Kubernetes Cluster Health Assessment

Runbook Information

Item	Details
Runbook ID	K8S-RB-01
Category	Operations
Difficulty	Intermediate
Estimated Time	45–60 Minutes
Environment	Kubernetes Cluster
Platform	Amazon EKS / Azure AKS / Google GKE / kind
Primary Role	Kubernetes Security Engineer
Team	Cloud Platform Operations
Escalation	Kubernetes Platform Team

Scenario

It is 08:30 AM.

You have just started your shift as a Kubernetes Security Engineer.

Several development teams are beginning work and production applications are expected to handle thousands of users throughout the day.

Before approving the platform for business operations, the Kubernetes Platform Team performs a daily health assessment.

Your responsibility is to verify that the cluster is healthy, stable and ready for production workloads.

Objectives

Perform a complete cluster health assessment covering:

Cluster connectivity
Control Plane
Worker Nodes
Namespaces
System Pods
Application Pods
Networking
DNS
Storage
Workloads
Events
Resource utilisation
Security observations

Success Criteria

The assessment is considered successful when:

All nodes are Ready
System Pods are healthy
Applications are Running
No critical events exist
Storage is healthy
DNS resolves correctly
Cluster networking functions correctly
No critical security observations remain unresolved

Assessment Workflow

Cluster Access

↓

Cluster Health

↓

Nodes

↓

Control Plane

↓

System Pods

↓

Applications

↓

Networking

↓

Storage

↓

Security

↓

Assessment Report

Phase 1 — Verify Cluster Connectivity

Step 1

Confirm Kubernetes context.

kubectl config current-context

Expected:

kind-cloudnova-security-lab

Step 2

Verify cluster access.

kubectl cluster-info

Expected:

Kubernetes control plane is running...

Step 3

Display cluster information.

kubectl version

Review:

Client Version
Server Version

Phase 2 — Assess Node Health

List Nodes

kubectl get nodes

Expected:

STATUS

Ready

Display Detailed Information

kubectl get nodes -o wide

Review

Internal IP
Kubernetes Version
Runtime
OS
Age

Inspect Every Node

kubectl describe node <NODE>

Review

Conditions
Memory Pressure
Disk Pressure
PID Pressure
Ready

Expected

Ready=True

MemoryPressure=False

DiskPressure=False

Health Checklist

Check	Status
Ready	✅
Memory Pressure	❌
Disk Pressure	❌
PID Pressure	❌

Phase 3 — Control Plane Assessment

List Control Plane Pods.

kubectl get pods -n kube-system

Confirm

kube-apiserver
etcd
kube-controller-manager
kube-scheduler

Status should be

Running

Inspect one component.

kubectl describe pod kube-apiserver...

Review

Restarts
Events
Node
Health Probes

Phase 4 — Worker Node Services

Review

kubectl get daemonsets -A

Verify

kube-proxy
CNI
CSI Driver

Running

Desired

Ready

Current

should match.

Phase 5 — Namespace Assessment

List

kubectl get namespaces

Review

Active
Terminating

Unexpected

Namespaces should be investigated.

Phase 6 — Pod Health

List every Pod.

kubectl get pods -A

Look for

CrashLoopBackOff

Pending

ImagePullBackOff

ContainerCreating

Evicted

Unknown

Investigate unhealthy Pods.

kubectl describe pod <POD>

Review Events.

Phase 7 — Deployment Health

kubectl get deployments -A

Review

READY

AVAILABLE

UP-TO-DATE

Unavailable Deployments require investigation.

Phase 8 — ReplicaSets

kubectl get rs -A

Review

Desired

Ready

Current

Phase 9 — StatefulSets

kubectl get statefulsets -A

Confirm

Ready

Replicas

Storage

Phase 10 — Services

kubectl get svc -A

Review

ClusterIP

NodePort

LoadBalancer

External IP

Unexpected public services should be reviewed.

Phase 11 — DNS

Confirm CoreDNS.

kubectl get pods -n kube-system -l k8s-app=kube-dns

Test DNS.

kubectl exec dns-test-client -- nslookup kubernetes.default

Expected

Successful resolution.

Phase 12 — Storage

List Storage Classes.

kubectl get storageclass

List PVs.

kubectl get pv

List PVCs.

kubectl get pvc -A

Review

Bound

Pending

Lost

Phase 13 — Events

Recent events.

kubectl get events -A --sort-by=.lastTimestamp

Review

Warnings

FailedScheduling

OOMKilled

FailedMount

ImagePullBackOff

Phase 14 — Resource Usage

Nodes

kubectl top nodes

Pods

kubectl top pods -A

High CPU

High Memory

Unexpected spikes

should be investigated.

Phase 15 — Cluster Networking

Review

kubectl get endpoints -A

Verify

Services have healthy endpoints.

Review

kubectl get networkpolicies -A

Unexpected missing policies

should be documented.

Phase 16 — Security Review

Review

kubectl get serviceaccounts -A

Review

kubectl get clusterrolebindings

Review

kubectl get secrets -A

Confirm

No unnecessary Secrets.

Review

kubectl auth can-i --list

Confirm

Administrator permissions are expected.

Phase 17 — Application Validation

Check

kubectl get ingress -A

Check

kubectl get endpointslices -A

Validate

Application

↓

Service

↓

Endpoints

↓

Pod

communication.

Health Assessment Matrix

Component	Status	Notes
API Server
etcd
Scheduler
Controller Manager
Nodes
kubelet
kube-proxy
CoreDNS
Storage
Applications
Services
Network
Security

Severity Matrix

Critical

Examples

API Server Down
etcd Down
All Nodes NotReady
CoreDNS Failure

Immediate escalation required.

High

Examples

Worker Node NotReady
Storage Failure
Multiple CrashLoopBackOff

Medium

Examples

One unhealthy application
High CPU
Pending Pods

Low

Examples

Warning Events
Restart Count Increased
Old ReplicaSets

Escalation Guide

Severity	Action
Critical	Platform Team immediately
High	Senior Kubernetes Engineer
Medium	Investigate within shift
Low	Record and monitor

Assessment Report

Cluster Name:

Assessment Date:

Engineer:

Cluster Status:

Control Plane:

Worker Nodes:

Applications:

Storage:

Networking:

Security:

Critical Findings:

Recommendations:

Overall Status:

PASS / FAIL

Best Practices

Always:

Verify cluster before deployments
Review Events first
Check Nodes before Applications
Review Storage health
Validate DNS
Monitor Resource usage
Document findings
Escalate Critical issues immediately

Troubleshooting Quick Reference

Symptom	Investigation
Node NotReady	describe node
Pod Pending	describe pod
CrashLoopBackOff	logs + describe
DNS Failure	CoreDNS
Storage Pending	PVC
API Issues	API Server
Scheduling Failure	Scheduler
Network Issue	Service + Endpoints

Evidence Collection

Capture:

Cluster Info
Node Status
System Pods
Resource Usage
Storage
Services
Events
Security Findings
Health Report

Runbook Summary

This runbook provides a repeatable operational procedure for validating the health of a Kubernetes cluster before business operations or maintenance activities.

Following this workflow ensures that engineers verify the Control Plane, Worker Nodes, workloads, networking, storage, DNS, and security posture in a consistent manner, enabling early detection of operational issues and reducing the risk of production incidents.