Runbook 03 — Kubernetes Traffic Investigation

Runbook Information

Item	Details
Runbook ID	K8S-NET-RB-03
Category	Kubernetes Network Security Investigation
Difficulty	Advanced
Estimated Time	3–6 Hours
Environment	Kubernetes Cluster
Platform	Amazon EKS / Azure AKS / Google GKE / OpenShift
Primary Role	Kubernetes Security Engineer
Supporting Teams	SOC, Platform Engineering, Cloud Operations, Incident Response
Escalation	Cloud Security Architect / Incident Response Manager

Purpose

This runbook provides a structured methodology for investigating Kubernetes network traffic during suspected security incidents, performance issues, compliance reviews, or abnormal application behaviour.

The investigation focuses on understanding:

East-west traffic
North-south traffic
Pod-to-Pod communication
Namespace communication
DNS activity
Service-to-Service communication
Service Mesh traffic
Network Policy enforcement
Suspicious network behaviour
Indicators of lateral movement

Enterprise Scenario

CloudNova Technologies operates a production Kubernetes environment supporting customer-facing applications, payment services, monitoring systems, and internal APIs.

The Security Operations Centre (SOC) receives an alert from the Network Detection and Response (NDR) platform indicating that a frontend application has initiated connections to multiple internal services that it has never accessed before.

Additional telemetry indicates:

Increased DNS requests
High outbound traffic
Multiple failed connections
Unexpected communication across namespaces

The Incident Response Team suspects that a workload has been compromised and may be attempting lateral movement within the Kubernetes cluster.

As the Kubernetes Security Engineer, your responsibility is to investigate network traffic, determine the scope of the incident, identify affected workloads, and recommend containment actions.

Investigation Objectives

Determine:

Source of suspicious traffic
Destination workloads
Communication paths
Policy violations
DNS anomalies
Service Mesh activity
Network Policy effectiveness
Lateral movement attempts
Data exfiltration indicators
Overall incident impact

Success Criteria

The investigation is successful when:

The originating workload is identified.
Communication paths are mapped.
Network Policy violations are detected.
Suspicious east-west traffic is analysed.
DNS anomalies are reviewed.
Service Mesh telemetry is validated.
Root cause is identified.
Appropriate containment recommendations are documented.

Enterprise Traffic Architecture

                    Internet
                        │
                  HTTPS Requests
                        │
                 Ingress Controller
                        │
          ┌─────────────┴─────────────┐
          │                           │
     Frontend Pods              API Services
          │                           │
          └─────────────┬─────────────┘
                        │
                 PostgreSQL Database
                        │
                Monitoring Platform

--------------------------------------------

Observed Suspicious Activity

Frontend

↓

Multiple Internal Services

↓

Cross-Namespace Communication

↓

Unexpected DNS Queries

↓

Potential Lateral Movement

Investigation Workflow

Receive Alert

      │

      ▼

Validate Cluster Health

      │

      ▼

Identify Source Workload

      │

      ▼

Analyse Network Traffic

      │

      ▼

Review Network Policies

      │

      ▼

Investigate DNS Activity

      │

      ▼

Review Service Mesh Telemetry

      │

      ▼

Containment Assessment

      │

      ▼

Incident Report

Phase 1 — Validate Cluster Health

Confirm the cluster is healthy.

kubectl cluster-info

kubectl get nodes

kubectl get pods -A

Review:

API Server status
Worker node health
Control Plane components
Networking components

Phase 2 — Identify the Source Workload

Locate the suspected Pod.

kubectl get pods -A -o wide

Collect:

Namespace
Pod name
Node
IP address
Labels
Service Account

Document the workload owner and business purpose.

Phase 3 — Review Network Policies

List all policies.

kubectl get networkpolicy -A

Inspect applicable policies.

kubectl describe networkpolicy

Determine:

Expected communication paths
Allowed ingress
Allowed egress
Any missing default deny policies

Phase 4 — Investigate Pod Communication

Determine which workloads the Pod communicates with.

Review:

Backend APIs
Databases
Monitoring services
External endpoints
Other namespaces

Expected communication matrix:

Source	Destination	Expected
Frontend	Backend	✅
Backend	Database	✅
Frontend	Database	❌
Frontend	Monitoring	❌
Development	Production	❌

Flag all unexpected communication.

Phase 5 — Review Service Inventory

List Services.

kubectl get svc -A

Review:

ClusterIP
NodePort
LoadBalancer
ExternalName

Determine whether suspicious traffic targets exposed Services or internal workloads.

Phase 6 — Investigate DNS Activity

Review CoreDNS.

kubectl get pods -n kube-system

Investigate:

Excessive DNS requests
Unknown domains
Repeated failed lookups
External name resolution
Unusual service discovery

Questions:

Is DNS enumeration occurring?
Are attackers discovering internal Services?

Phase 7 — Analyse Ingress Traffic

Review Ingress resources.

kubectl get ingress -A

Inspect:

kubectl describe ingress

Determine:

External entry point
Source IPs
Target Services
Host rules
Path routing

Look for:

Unexpected requests
High request rates
Unknown client IPs

Phase 8 — Investigate East-West Traffic

Review internal communication.

Determine:

Namespace-to-namespace traffic
Pod-to-Pod traffic
Unexpected application calls
Communication with sensitive workloads

Identify:

Lateral movement
Reconnaissance
Service discovery attempts

Phase 9 — Review Service Mesh Telemetry

If Istio or Linkerd is deployed:

Review:

istioctl proxy-status

Inspect:

mTLS status
AuthorizationPolicy
PeerAuthentication
DestinationRule
VirtualService

Determine:

Successful connections
Denied requests
Identity information
Encrypted traffic

Phase 10 — Review Logs

Collect logs from:

Application Pods
Ingress Controller
CoreDNS
Service Mesh
Kubernetes Events
Cloud logging platform

Example:

kubectl logs <pod-name>

kubectl get events --sort-by=.metadata.creationTimestamp

Review for:

Connection failures
Authentication failures
Unexpected outbound traffic
Policy denials

Phase 11 — Investigate Indicators of Compromise

Look for:

Port scanning
Service enumeration
DNS enumeration
Unexpected outbound connections
Data exfiltration attempts
Cross-namespace communication
New Pods
Suspicious images
Unknown Service Accounts

Phase 12 — Assess Network Segmentation

Validate:

Default deny policies
Namespace isolation
Least privilege networking
Zero Trust communication
Database protection

Determine whether segmentation successfully limited attacker movement.

Phase 13 — Root Cause Analysis

Identify:

Initial source
Compromised workload
Communication path
Policy gaps
Impacted applications
Business impact

Determine whether the incident resulted from:

Missing Network Policies
Misconfigured Services
Excessive permissions
Application vulnerability
Misconfigured Service Mesh

Phase 14 — Containment Recommendations

Immediate:

Isolate affected namespace
Block compromised Pod
Apply temporary Network Policies
Scale down malicious workloads
Revoke compromised credentials

Short-term:

Patch vulnerable applications
Harden Network Policies
Review Service Accounts
Enable additional monitoring

Long-term:

Improve Zero Trust controls
Automate policy validation
Conduct regular penetration testing
Expand network visibility

Investigation Checklist

Control	Status
Source Workload Identified	☐
Destination Workloads Mapped	☐
Network Policies Reviewed	☐
DNS Activity Investigated	☐
East-West Traffic Analysed	☐
Ingress Traffic Reviewed	☐
Service Mesh Validated	☐
Indicators of Compromise Identified	☐
Containment Recommendations Prepared	☐
Incident Report Completed	☐

Risk Classification

Critical

Unrestricted lateral movement
Public database access
Data exfiltration
Network Policy bypass
Compromised production workloads

High

Excessive namespace communication
Unexpected outbound traffic
Suspicious DNS activity
Weak Service Mesh policies

Medium

Missing monitoring
Broad Network Policies
Legacy Services
Incomplete logging

Low

Documentation updates
Naming inconsistencies
Monitoring improvements

Enterprise Traffic Investigation Report

Incident Number:

Investigation Date:

Assessor:

Cluster Name:

Affected Namespace:

Source Workload:

Destination Workloads:

Network Policies Reviewed:

Ingress Investigation:

DNS Investigation:

Service Mesh Review:

Indicators of Compromise:

Root Cause:

Containment Actions:

Recommendations:

Business Impact:

Overall Risk Rating:

Incident Status:

Open

Contained

Closed

Evidence Collection

Collect evidence for:

Cluster status
Pod inventory
Network Policies
Service inventory
Ingress configuration
DNS activity
Service Mesh telemetry
Application logs
Kubernetes Events
Connectivity tests
Incident timeline
Final investigation report

Operational Best Practices

Always:

Investigate unexpected communication immediately.
Review Network Policies before modifying workloads.
Validate east-west and north-south traffic during every investigation.
Use Service Mesh telemetry where available.
Correlate Kubernetes events with application and cloud logs.
Document all findings with timestamps.
Preserve evidence before remediation.
Follow incident response procedures for production environments.
Conduct a post-incident review to improve network security controls.

Troubleshooting Quick Reference

Issue	Investigation
Unexpected Pod communication	Review Network Policies, Services, and Pod labels
Cross-namespace access	Inspect namespace selectors and egress rules
High DNS activity	Review CoreDNS logs and external lookups
Service unreachable	Check Service, Endpoints, Network Policies, and Ingress
mTLS failures	Review PeerAuthentication, DestinationRules, and sidecar status
Unexpected outbound traffic	Analyse egress rules, cloud firewall logs, and Service Mesh telemetry
Network Policy not enforced	Verify CNI supports Network Policies and policies are correctly applied

Runbook Summary

This runbook provides an enterprise approach to investigating Kubernetes network traffic during security incidents. It guides security engineers through identifying suspicious workloads, analysing east-west and north-south communication, validating Network Policies, reviewing DNS activity, examining Service Mesh telemetry, and determining the root cause of abnormal behaviour.

By following this runbook, Security Operations, Incident Response, and Platform Engineering teams can rapidly detect lateral movement, contain compromised workloads, preserve forensic evidence, and strengthen Kubernetes network security through continuous improvement and Zero Trust networking principles.