Lesson 07 — Amazon EKS Logging & Monitoring

Learning Objectives

By the end of this lesson, you will be able to:

Explain the difference between logging, monitoring and observability
Identify the main Amazon EKS log and metric sources
Enable and understand EKS control plane logging
Explain the security value of Kubernetes audit logs
Collect application, container and worker-node logs
Monitor cluster and workload health with Amazon CloudWatch
Understand Container Insights and enhanced observability
Use Prometheus and Grafana for Kubernetes monitoring
Design alerts for operational and security events
Integrate Amazon EKS telemetry with an enterprise SIEM
Protect log confidentiality, integrity and availability
Define retention and evidence-management requirements
Build an enterprise Amazon EKS observability architecture
Develop a phased logging and monitoring implementation roadmap

Why This Matters

A secure Amazon EKS cluster must provide visibility into what is happening across the platform.

Without effective logging and monitoring, security and operations teams may not know:

Who accessed the cluster
Which Kubernetes resources were changed
Whether an administrator created a privileged workload
Why a Pod failed
Whether worker nodes are under resource pressure
Whether an application is communicating with an unusual destination
Whether an admission policy was bypassed
Whether an attacker accessed Kubernetes Secrets
Whether audit-log delivery has stopped
Whether a compromised container started a shell

No Visibility

↓

Delayed Detection

↓

Longer Attacker Dwell Time

↓

Greater Business Impact

Logging and monitoring support:

Security operations
Incident response
Troubleshooting
Capacity planning
Performance management
Compliance reporting
Forensic investigation
Service-level management

For a Cloud Security Engineer, collecting telemetry is only the beginning.

The organisation must also:

Protect the telemetry
Retain it for the required period
Analyse it
Create useful alerts
Assign alert ownership
Investigate abnormal activity
Test the complete monitoring pipeline

Logging, Monitoring and Observability

These terms are related but have different meanings.

Capability	Purpose
Logging	Records discrete events and messages
Monitoring	Measures system health and checks defined conditions
Observability	Uses logs, metrics and traces to understand internal system behaviour
Alerting	Notifies teams when a defined condition occurs
Security analytics	Correlates telemetry to identify suspicious activity

Logs

+

Metrics

+

Traces

+

Events

=

Observability

What Are Logs?

Logs are timestamped records of system, application or security activity.

Examples include:

Kubernetes API requests
Container standard output
Application authentication events
Node operating-system events
Admission-controller denials
Load balancer requests
AWS API activity
Runtime security alerts

A useful log should provide context such as:

When did the event occur?

Who initiated it?

Which resource was affected?

What action was attempted?

Was the action successful?

Where did the request originate?

What was the final result?

What Are Metrics?

Metrics are numerical measurements collected over time.

Examples include:

CPU usage
Memory usage
Pod restart count
API request latency
Failed scheduling attempts
Node filesystem utilisation
Network throughput
Admission webhook latency
Number of policy violations

Metrics are commonly stored as time-series data.

Metric Name

+

Value

+

Timestamp

+

Labels

=

Time-Series Observation

What Are Traces?

Distributed traces track a request as it moves through several services.

Customer Request

↓

Ingress

↓

Frontend Service

↓

Payment API

↓

Fraud Service

↓

Database

Tracing can reveal:

Slow services
Failed dependencies
Request paths
Latency between components
Application errors
Service relationships

Amazon EKS Observability Layers

Layer 1 — AWS Account

CloudTrail, AWS Config, GuardDuty and Security Hub

↓

Layer 2 — EKS Control Plane

API, Audit, Authenticator, Controller Manager and Scheduler Logs

↓

Layer 3 — Worker Nodes

Operating-System, kubelet, container-runtime and network logs

↓

Layer 4 — Kubernetes Platform

Events, add-on logs, policy reports and metrics

↓

Layer 5 — Applications

Application logs, business metrics and distributed traces

↓

Layer 6 — Network

VPC Flow Logs, DNS logs and load-balancer logs

A mature monitoring programme collects data from every relevant layer.

Amazon EKS Logging Sources

Amazon EKS Logging Sources

├── EKS Control Plane Logs
│   ├── API Server
│   ├── Audit
│   ├── Authenticator
│   ├── Controller Manager
│   └── Scheduler
│
├── Kubernetes Platform Logs
│   ├── Events
│   ├── CoreDNS
│   ├── VPC CNI
│   ├── kube-proxy
│   ├── CSI Drivers
│   ├── Admission Controllers
│   └── GitOps Controllers
│
├── Worker-Node Logs
│   ├── Operating System
│   ├── kubelet
│   ├── containerd
│   └── Security Agents
│
├── Workload Logs
│   ├── Application
│   ├── Sidecar
│   └── Container stdout and stderr
│
└── AWS Network and Security Logs
    ├── CloudTrail
    ├── VPC Flow Logs
    ├── Load Balancer Logs
    ├── GuardDuty Findings
    ├── Inspector Findings
    └── Security Hub Findings

Amazon EKS Control Plane Logging

Amazon EKS can export managed Kubernetes control plane logs to Amazon CloudWatch Logs.

The available control plane log types are:

API Server
Audit
Authenticator
Controller Manager
Scheduler

Amazon EKS Control Plane

↓

Selected Log Types

↓

CloudWatch Log Group

↓

Log Streams

↓

Security Analytics and Troubleshooting

Control plane logging should be explicitly enabled according to enterprise requirements.

Control Plane Log Types

Log Type	Primary Purpose
API Server	Records Kubernetes API Server activity and diagnostic information
Audit	Records Kubernetes API requests according to the managed audit policy
Authenticator	Records AWS IAM authentication activity
Controller Manager	Records controller operations and reconciliation activity
Scheduler	Records Pod scheduling decisions and failures

API Server Logs

API Server logs can help identify:

API errors
Request failures
Webhook problems
Connectivity issues
Resource-validation failures
API performance issues

These logs are valuable for both operations and security investigations.

Kubernetes Audit Logs

Audit logs provide a chronological record of requests made to the Kubernetes API.

Audit records may help answer:

Which identity made the request?
Which action was attempted?
Which resource was targeted?
Which namespace was involved?
Was the request allowed?
What source address initiated the request?
Which user agent was used?
When did the activity occur?

Identity

↓

Kubernetes API Request

↓

Audit Event

↓

CloudWatch Logs

↓

SIEM Detection

Security Value of Audit Logs

Audit logs can support detection of:

Unauthorised access attempts
Secret access
RBAC changes
cluster-admin assignments
Privileged Pod deployment
Pod execution
Admission webhook modification
Service Account token creation
Namespace deletion
Network Policy changes
Suspicious user agents
Repeated failed requests

Example High-Risk Kubernetes Activities

Activity	Security Concern
`get` or `list` Secrets	Credential discovery
Create ClusterRoleBinding	Privilege escalation
Create privileged Pod	Potential node compromise
Use `pods/exec`	Interactive container access
Delete namespace	Service disruption
Modify webhook configuration	Security-control bypass
Create Service Account token	Credential acquisition
Modify NetworkPolicy	Segmentation bypass
Create LoadBalancer Service	Unapproved external exposure

Authenticator Logs

Authenticator logs provide visibility into AWS IAM authentication activity associated with cluster access.

They may support investigation of:

IAM identity authentication
Authentication failures
Unexpected IAM roles
Access from unusual environments
Mapping or access-entry issues
Administrative access patterns

Authentication logs should be correlated with:

AWS CloudTrail
EKS access entries
Kubernetes audit logs
Enterprise identity-provider logs

Controller Manager Logs

Controller Manager logs can help troubleshoot:

Deployment reconciliation
Replica management
Node lifecycle events
Endpoint updates
Job operations
Resource-controller failures

These logs are primarily operational but may also reveal malicious or unexpected resource changes.

Scheduler Logs

Scheduler logs provide information about:

Pod placement
Unschedulable Pods
Resource shortages
Affinity conflicts
Taints and tolerations
Topology constraints
Scheduling failures

Scheduler issues may indicate:

Cluster capacity problems
Incorrect workload configuration
Resource exhaustion
Availability Zone imbalance
Abuse of resource requests
Misconfigured node selectors

Enable Control Plane Logs with AWS CLI

Example:

aws eks update-cluster-config \
  --name production-eks \
  --logging '{
    "clusterLogging": [
      {
        "types": [
          "api",
          "audit",
          "authenticator",
          "controllerManager",
          "scheduler"
        ],
        "enabled": true
      }
    ]
  }'

Verify the configuration:

aws eks describe-cluster \
  --name production-eks \
  --query 'cluster.logging'

Control Plane Log Group

Control plane logs are delivered to a cluster-specific CloudWatch Logs group.

Enterprise controls should define:

Log retention
Encryption
Access permissions
Subscription filters
Archival
Export requirements
Monitoring for delivery failure
Central-account forwarding

Control Plane Logging Considerations

Control plane logging introduces:

CloudWatch ingestion costs
Storage costs
Query costs
Increased telemetry volume
Potentially sensitive metadata

Cost should be managed through:

Defined retention periods
Central log architecture
Appropriate archive tiers
Targeted dashboards
Efficient queries
Duplicate-ingestion avoidance

Security visibility should not be disabled only to reduce cost.

Kubernetes Events

Kubernetes Events provide information about resource lifecycle and cluster activity.

View events:

kubectl get events -A

Sort events by time:

kubectl get events -A \
  --sort-by='.metadata.creationTimestamp'

Events may show:

Failed scheduling
Image pull failures
Container crashes
Volume mount failures
Probe failures
Node pressure
Scaling activity
Admission denials

Events Are Not a Long-Term Log Store

Kubernetes Events are temporary resources.

Do not rely on them as the only source of historical evidence.

Important events should be exported to a durable monitoring or logging platform.

Application Logging

Applications should normally write logs to:

stdout

and

stderr

The container runtime writes these streams to node-level log files.

A log agent can then collect and forward them.

Application

↓

stdout and stderr

↓

Container Runtime

↓

Node Log Files

↓

Log Collector

↓

CloudWatch Logs or SIEM

Application Logging Best Practices

Application logs should:

Use structured formats
Include timestamps
Include request or correlation IDs
Include service and environment names
Use consistent severity levels
Avoid sensitive values
Avoid passwords and tokens
Support central search
Include meaningful error context

Structured Logging

JSON is commonly used for structured application logs.

{
  "timestamp": "2026-07-31T08:30:00Z",
  "level": "ERROR",
  "service": "payment-api",
  "environment": "production",
  "requestId": "example-request-id",
  "message": "Payment authorisation failed",
  "errorCode": "PAYMENT-401"
}

Structured logs improve:

Searching
Filtering
Correlation
Dashboard creation
Automated detection
SIEM parsing

Log Severity Levels

Level	Typical Use
DEBUG	Detailed troubleshooting information
INFO	Normal application activity
WARN	Unexpected but recoverable condition
ERROR	Failed operation requiring attention
CRITICAL	Severe failure or security event

Production applications should not continuously use excessive debug logging.

Protect Sensitive Information

Applications should never log:

Passwords
API keys
Session tokens
Authorization headers
Private keys
Full payment-card data
Unnecessary personal information
Database connection strings
AWS credentials

Record the Event

Not the Secret

Log Redaction

Implement redaction for fields such as:

password

token

secret

authorization

cookie

privateKey

accessKey

Redaction should occur before the information enters the logging pipeline.

Container Log Collection

A common Kubernetes pattern uses a node-level DaemonSet.

Worker Node

├── Application Pod Logs
├── System Logs
└── Container Runtime Logs

↓

Log Collector DaemonSet

↓

Central Log Platform

A DaemonSet ensures that a collector runs on each eligible worker node.

Fluent Bit

Fluent Bit is commonly used as a lightweight log collector.

It can:

Read container logs
Parse structured data
Enrich records with Kubernetes metadata
Filter records
Forward logs to several destinations
Buffer during temporary network interruptions

A collector requires careful configuration to avoid:

Secret leakage
Excessive memory usage
Dropped records
Duplicate logs
Unbounded buffering
Excessive permissions

Log Metadata Enrichment

Useful metadata includes:

Cluster name
AWS account
AWS Region
Namespace
Pod name
Container name
Node name
Application
Environment
Workload owner
Data classification

Example:

cluster=payments-prod

namespace=payments

application=payment-api

environment=production

owner=payments-team

Worker-Node Logs

Worker-node logs may include:

Operating-system logs
kubelet logs
Container-runtime logs
Kernel events
Node bootstrap logs
Network-agent logs
Storage-driver logs
Security-agent logs

Node logs are important for:

Node failure analysis
Container-runtime investigation
Network troubleshooting
Privilege-escalation investigation
Forensic analysis

kubelet Monitoring

Monitor kubelet-related conditions such as:

Node registration failure
Pod startup failure
Probe failures
Volume mounting failure
Image pull errors
Resource pressure
Container-runtime communication errors

Container Runtime Monitoring

Amazon EKS EC2 worker nodes commonly use containerd.

Monitor for:

Runtime failures
Image-pull failures
Container crashes
Storage problems
Unexpected runtime configuration changes
Suspicious container execution

Platform Add-On Logs

Collect logs for critical components such as:

Amazon VPC CNI
CoreDNS
kube-proxy
EBS CSI Driver
EFS CSI Driver
AWS Load Balancer Controller
Karpenter
Cluster Autoscaler
Kyverno
OPA Gatekeeper
GitOps controllers
Secrets Store CSI Driver

CoreDNS Monitoring

Monitor:

DNS request failures
Resolution latency
Timeout rate
CPU and memory
Pod availability
Configuration changes
External DNS anomalies

CoreDNS failure may affect the entire cluster.

Amazon VPC CNI Monitoring

Monitor:

IP allocation failures
ENI allocation failures
Subnet capacity
CNI Pod health
Network-policy enforcement
API throttling
Pod-network setup failures

Admission Controller Monitoring

Monitor:

Admission denials
Webhook latency
Webhook timeouts
Evaluation errors
Certificate expiry
Controller availability
PolicyReport generation
Background-scan failures

Policy Engine Unavailable

↓

Possible Deployment Failure

or

↓

Possible Enforcement Gap

GitOps Monitoring

Monitor:

Reconciliation failures
Drift
Unauthorised direct changes
Failed deployments
Repository access
Controller authentication
Production rollback activity

Amazon CloudWatch

Amazon CloudWatch provides AWS-native capabilities for:

Logs
Metrics
Dashboards
Alarms
Events
Queries
Application observability
Container monitoring

Amazon EKS

↓

CloudWatch Logs and Metrics

↓

Dashboards and Alarms

↓

Operations and Security Teams

CloudWatch Logs

CloudWatch Logs can store:

EKS control plane logs
Application logs
Node logs
Add-on logs
Security-tool logs

CloudWatch Logs capabilities include:

Log groups
Log streams
Retention policies
Metric filters
Subscription filters
Logs Insights
Encryption
Access control

CloudWatch Logs Insights

CloudWatch Logs Insights allows teams to search and analyse logs.

Conceptual audit query:

fields @timestamp, user.username, verb, objectRef.resource,
       objectRef.namespace, sourceIPs

| filter verb in ["create", "update", "patch", "delete"]

| sort @timestamp desc

| limit 100

Queries should be tested against the actual audit-log field structure.

Detect Secret Access

Conceptual search:

filter objectRef.resource = "secrets"

| filter verb in ["get", "list", "watch"]

| sort @timestamp desc

The security team should maintain tested queries for high-risk Kubernetes actions.

CloudWatch Metric Filters

Metric filters can convert matching log patterns into CloudWatch metrics.

Example use cases:

Count failed authentications
Count Secret access events
Count cluster-admin changes
Count privileged Pod creation attempts
Count admission denials
Count critical application errors

Log Event

↓

Metric Filter

↓

CloudWatch Metric

↓

Alarm

↓

Notification or Automated Response

CloudWatch Alarms

Alarms may monitor:

Node CPU pressure
Node memory pressure
Pod restart rate
API error rate
Failed authentication
Log-delivery failure
Critical application errors
Subnet IP capacity
Load balancer health
Admission-controller availability

Every alarm should have:

Owner
Severity
Threshold
Evaluation window
Notification path
Runbook
Escalation procedure

CloudWatch Container Insights

Container Insights provides observability for containerised workloads.

It can provide cluster, node, namespace, workload and Pod-level information.

Typical views may include:

CPU usage
Memory usage
Network activity
Pod count
Node count
Container restart information
Workload performance
Cluster health

Enhanced Observability

Enhanced observability can provide more detailed infrastructure and container telemetry.

Enterprise teams should evaluate:

Required metrics
Collection architecture
Agent resource usage
Data volume
Cost
Retention
Multi-account visibility
Integration with existing tools

Amazon CloudWatch Observability Add-On

The Amazon CloudWatch Observability EKS add-on can simplify deployment of components used for:

Container Insights
Application Signals
Logs
Metrics
Traces
AWS X-Ray integration

Add-ons should still be governed through:

Approved versions
Least-privilege IAM
Configuration review
Monitoring
Upgrade testing
Ownership

Prometheus

Prometheus is a monitoring system and time-series database widely used with Kubernetes.

Prometheus normally:

Discovers metric endpoints
Scrapes metrics
Stores time-series data
Evaluates alerting rules
Supports PromQL queries

Kubernetes Components and Applications

↓

Prometheus Metrics Endpoints

↓

Prometheus

↓

Queries and Alerts

↓

Grafana

Prometheus Metric Format

Example:

http_requests_total{
  service="payment-api",
  namespace="payments",
  status="200"
} 15240

Labels support detailed filtering and aggregation.

Prometheus Monitoring Targets

Prometheus may monitor:

Kubernetes API
Nodes
kubelet
Pods
Deployments
Services
Ingress controllers
CoreDNS
VPC CNI
Admission controllers
Applications
Databases
Service-mesh components

Amazon Managed Service for Prometheus

Amazon Managed Service for Prometheus provides a managed Prometheus-compatible metrics environment.

Potential benefits include:

Managed scaling
Managed storage
AWS IAM integration
Reduced operational overhead
Multi-cluster aggregation
PromQL compatibility

Multiple EKS Clusters

↓

Prometheus-Compatible Collection

↓

Amazon Managed Service for Prometheus

↓

Central Grafana Dashboards

Self-Managed Versus Managed Prometheus

Self-Managed Prometheus	Managed Prometheus Service
Full operational control	Reduced infrastructure management
Cluster-local storage options	Managed scalable storage
Customer manages availability	AWS manages service availability
Customer manages upgrades	Managed-service lifecycle
May be simpler for small labs	Useful for enterprise aggregation
Requires backup planning	Separate service lifecycle and cost

Prometheus Cardinality

Cardinality refers to the number of unique metric label combinations.

High cardinality may be caused by labels such as:

Request IDs
User IDs
Session IDs
Full URLs
Dynamic container identifiers
Unbounded error messages

Too Many Unique Labels

↓

High Memory and Storage Usage

↓

Slow Queries

↓

Increased Cost

Do not use unbounded values as metric labels.

Grafana

Grafana visualises metrics from sources such as:

Prometheus
Amazon Managed Service for Prometheus
CloudWatch
Loki
Other enterprise data sources

Dashboards may show:

Cluster health
Node capacity
Namespace utilisation
Pod performance
Application latency
Error rates
Policy-engine health
Security findings

Amazon Managed Grafana

Amazon Managed Grafana can reduce the operational work required to host and maintain Grafana.

Enterprise requirements still include:

Identity integration
Dashboard access control
Data-source permissions
Workspace governance
Dashboard ownership
Change management

The Four Golden Signals

Application monitoring commonly focuses on:

Signal	Meaning
Latency	Time required to process requests
Traffic	Volume of system demand
Errors	Rate of failed operations
Saturation	How close resources are to capacity

Latency

Traffic

Errors

Saturation

These signals provide a strong foundation for service monitoring.

Kubernetes Platform Metrics

Important cluster metrics include:

Node readiness
CPU utilisation
Memory utilisation
Filesystem utilisation
Pod count
Pending Pods
Pod restarts
Failed Pods
Unschedulable Pods
API latency
API error rate
Deployment availability
DaemonSet coverage
StatefulSet health

Node Conditions

Review:

kubectl get nodes

kubectl describe node <node-name>

Important conditions may include:

Ready
MemoryPressure
DiskPressure
PIDPressure
NetworkUnavailable

Pod Health

Useful commands include:

kubectl get pods -A

kubectl get pods -A \
  --field-selector=status.phase!=Running

kubectl describe pod <pod-name> \
  -n <namespace>

kubectl logs <pod-name> \
  -n <namespace>

Previous Container Logs

For restarted containers:

kubectl logs <pod-name> \
  -n <namespace> \
  --previous

This can help investigate crash loops.

Multi-Container Pod Logs

kubectl logs <pod-name> \
  -n <namespace> \
  -c <container-name>

Follow Application Logs

kubectl logs <pod-name> \
  -n <namespace> \
  --follow

Production access to logs should follow approved access controls.

Application Health Probes

Kubernetes supports:

Startup probes
Readiness probes
Liveness probes

Startup Probe

Determines whether application startup has completed

Readiness Probe

Determines whether the Pod should receive traffic

Liveness Probe

Determines whether the container should be restarted

Probe Example

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080

  initialDelaySeconds: 20
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080

  initialDelaySeconds: 5
  periodSeconds: 5

Poorly configured probes may cause:

Restart loops
Service outages
Traffic to unhealthy Pods
False alerts
Excessive resource usage

Service-Level Indicators

A Service-Level Indicator is a measurable aspect of service performance.

Examples include:

Request success rate
Response latency
Availability
Processing time
Queue delay
Transaction completion rate

Service-Level Objectives

A Service-Level Objective defines a target.

Example:

99.9% of payment API requests

complete successfully

within the monthly measurement period

Error Budgets

An error budget represents the permitted level of unreliability.

SLO Target

↓

Allowed Failure

↓

Error Budget

Error budgets help balance:

Reliability
Development speed
Operational risk
Change frequency

Security Monitoring

Operational monitoring asks:

Is the service functioning?

Security monitoring asks:

Is suspicious or unauthorised activity occurring?

Both use overlapping telemetry.

High-Value EKS Security Detections

Create detections for:

New cluster-admin bindings
Unusual Secret access
Privileged Pod creation
Pod execution in production
Admission webhook changes
Audit logging disabled
EKS endpoint configuration changes
New public LoadBalancer Services
Service Account token creation
Network Policy deletion
Unapproved image deployment
Security-agent deletion
Unexpected namespace deletion
Repeated authentication failure

Detect cluster-admin Assignment

Investigation logic:

ClusterRoleBinding Created or Modified

↓

Role Reference Is cluster-admin

↓

Identify Subject

↓

Check Change Approval

↓

Investigate Source Identity

Detect Pod Execution

Use of pods/exec may be legitimate for troubleshooting but should be monitored in production.

Alert context should include:

User
Namespace
Pod
Container
Time
Source IP
User agent
Approved change or incident reference

Detect Privileged Workloads

Monitor creation or update of workloads containing:

securityContext:
  privileged: true

Also monitor:

HostPath
hostNetwork
hostPID
hostIPC
Added Linux capabilities
Privilege escalation
Unapproved Service Accounts

Detect Logging Disablement

Alert when:

EKS control plane logging is disabled
CloudWatch log groups are deleted
Retention is reduced unexpectedly
Subscription filters are removed
Log agents stop reporting
SIEM ingestion stops
KMS keys protecting logs are disabled

Monitoring the Monitoring System

Is a Security Requirement

AWS CloudTrail

AWS CloudTrail records AWS API activity.

For Amazon EKS, CloudTrail can help track:

Cluster creation
Cluster deletion
Cluster configuration changes
Logging changes
Endpoint-access changes
Access-entry changes
Node-group changes
Add-on changes
IAM and KMS operations

CloudTrail does not replace Kubernetes audit logging.

CloudTrail

Records AWS API Activity

Kubernetes Audit Logs

Record Kubernetes API Activity

Both are required for complete visibility.

Amazon GuardDuty

Amazon GuardDuty can provide managed threat-detection findings based on supported AWS and EKS data sources.

GuardDuty findings should be:

Centralised
Prioritised
Assigned
Investigated
Correlated with other telemetry
Retained according to policy

Amazon Security Hub

Security Hub can aggregate findings from AWS security services and supported integrations.

It can support:

Central finding visibility
Severity prioritisation
Multi-account aggregation
Compliance views
Workflow integration
SIEM forwarding

Amazon Inspector

Amazon Inspector findings may provide vulnerability visibility for supported workloads and resources.

Security teams should correlate vulnerabilities with:

Deployed image
Running workload
Internet exposure
Business criticality
Exploitability
Runtime behaviour

VPC Flow Logs

VPC Flow Logs provide network-flow metadata.

They may support detection of:

Unexpected outbound connections
Rejected traffic
Unusual cross-subnet activity
Communication with suspicious destinations
Network scanning
Database access anomalies

Load Balancer Logs

Application or network load-balancer telemetry may support:

Client-request analysis
Response-code monitoring
TLS analysis
Backend health analysis
Web-attack investigation
Traffic-volume monitoring

DNS Logging

DNS telemetry may help detect:

Command-and-control communication
Data exfiltration
Newly observed domains
Repeated failed lookups
Domain-generation algorithms
Access to malicious destinations

Runtime Security Logs

Runtime security tools can detect activity such as:

Shell execution
Unexpected process launch
Sensitive file access
Package manager execution
Container escape attempts
Privilege escalation
Cryptomining
Suspicious network connections

Runtime alerts should include enough context for investigation.

SIEM Integration

A Security Information and Event Management platform centralises and analyses security telemetry.

Amazon EKS Audit Logs

+

CloudTrail

+

GuardDuty

+

Runtime Alerts

+

Application Security Logs

+

Network Logs

↓

Enterprise SIEM

↓

Correlation Rules

↓

SOC Investigation

SIEM Data Requirements

Each event should include consistent fields such as:

Event time
Cluster
AWS account
Region
Namespace
Workload
Identity
Source address
Action
Resource
Result
Severity
Owner

SIEM Use Cases

Use Case	Data Sources
Privilege escalation	Audit logs, RBAC and CloudTrail
Secret access	Audit logs and application logs
Compromised Pod	Runtime alerts, network logs and audit logs
Public exposure	Audit logs, CloudTrail and load-balancer data
Malicious image	Registry, admission and runtime telemetry
Credential abuse	Authenticator, CloudTrail and identity-provider logs
Logging tampering	CloudTrail and platform monitoring

Cross-Account Logging Architecture

Workload AWS Accounts

├── Development EKS
├── Testing EKS
└── Production EKS

↓

Central Logging Account

├── CloudWatch Logs
├── S3 Archive
├── Security Analytics
└── SIEM Integration

Benefits include:

Separation of duties
Reduced tampering risk
Central retention
Enterprise correlation
Consistent access control

Multi-Cluster Observability

Large organisations may operate hundreds of clusters.

Standard metadata should identify:

AWS Account

Region

Cluster

Environment

Business Unit

Application

Owner

Criticality

Without consistent metadata, multi-cluster telemetry becomes difficult to analyse.

Log Routing Architecture

EKS Control Plane Logs

↓

CloudWatch Logs

Application and Node Logs

↓

Collector

↓

CloudWatch Logs

↓

Subscription or Export Pipeline

↓

Central Security Platform

↓

SIEM and Long-Term Archive

Log Retention

Retention should be based on:

Regulatory requirements
Incident-response needs
Threat-detection needs
Legal requirements
Data sensitivity
Storage cost
Evidence requirements

Example policy tiers:

Log Type	Example Retention Approach
High-value security logs	Long-term retention
Control plane audit logs	Compliance-defined retention
Application operational logs	Service-defined retention
Debug logs	Short retention
Archived evidence	Protected long-term storage

The exact periods must be approved by the organisation.

Log Integrity

Security logs must be protected from:

Deletion
Modification
Unauthorised access
Retention changes
KMS key deletion
Pipeline interruption

Controls may include:

Separate logging account
Restricted IAM roles
Encryption
Object versioning
Retention controls
Access logging
Integrity monitoring
Backup and replication

Log Confidentiality

Logs may contain:

User identities
Internal hostnames
Source IP addresses
Application paths
Error details
Security findings
Business data
Resource identifiers

Access should follow least privilege.

Logging Access Roles

Example roles:

Role	Access
Application Developer	Application logs for owned namespace
Platform Engineer	Platform and node telemetry
Security Engineer	Security and audit telemetry
SOC Analyst	Investigation and alert data
Compliance Auditor	Approved read-only evidence
Logging Administrator	Pipeline and retention management

Separation of Duties

Avoid allowing one administrator to:

Operate the production cluster
Disable logging
Delete the evidence
Modify the SIEM alert
Approve the same change

Separation of duties improves evidence integrity.

Monitoring Pipeline Health

Monitor the telemetry platform itself.

Important signals include:

Log-agent availability
Log-ingestion delay
Dropped records
Buffer saturation
CloudWatch delivery failure
Prometheus scrape failure
Dashboard-data delay
Alert delivery failure
SIEM connector failure
Storage-capacity failure

Data Completeness

A dashboard may appear healthy even when data is missing.

Implement controls such as:

Expected Clusters

Compared With

Reporting Clusters

Expected Log Sources

Compared With

Active Log Sources

Observability Coverage Register

Cluster	Audit Logs	App Logs	Metrics	Runtime Alerts	SIEM
payments-prod	Enabled	Enabled	Enabled	Enabled	Connected
customer-dev	Enabled	Enabled	Enabled	Partial	Connected
analytics-test	Enabled	Missing	Enabled	Missing	Partial

Coverage gaps should generate findings.

Alert Design

A useful alert should answer:

What happened?
Why does it matter?
Which resource is affected?
What is the severity?
Who owns the resource?
What should the responder do?
Which runbook applies?

Example Alert Record

Alert:

Privileged Pod Created

Severity:

Critical

Cluster:

payments-prod

Namespace:

payments

Workload:

payment-debug

Initiating Identity:

example-role

Source:

Kubernetes Audit Log

Required Action:

Validate approval and isolate workload if unauthorised

Runbook:

EKS Privileged Workload Investigation

Alert Severity

Severity	Meaning
Critical	Immediate high-impact threat or outage
High	Significant risk requiring urgent investigation
Medium	Suspicious or degraded condition
Low	Informational condition or minor deviation

Severity should consider:

Business impact
Exploitability
Resource sensitivity
Exposure
Confidence
Existing compensating controls

Avoid Alert Fatigue

Alert fatigue occurs when teams receive excessive low-value notifications.

Reduce alert fatigue by:

Tuning thresholds
Correlating related events
Suppressing known maintenance activity
Adding ownership metadata
Removing duplicate alerts
Measuring false-positive rates
Reviewing unused rules
Prioritising actionable conditions

More Alerts

Does Not Automatically Mean

Better Security

Alert Response Workflow

Alert Generated

↓

Notification Delivered

↓

Responder Acknowledges

↓

Context Enriched

↓

Activity Validated

↓

Incident Created if Required

↓

Containment and Investigation

↓

Alert Rule Improved

Dashboards

Create different dashboards for different audiences.

Platform Dashboard

Show:

Node health
Pod health
API latency
Scheduling failures
CNI health
DNS health
Storage health
Add-on health

Application Dashboard

Show:

Request rate
Error rate
Latency
Availability
Dependency health
Deployment version
Business transactions

Security Dashboard

Show:

Privileged access
Audit-log alerts
Secret access
Policy violations
Runtime findings
Public exposure
Vulnerabilities
Logging coverage

Executive Dashboard

Show:

Critical service health
Major security incidents
Compliance coverage
High-risk unresolved findings
Reliability trends
Business-service impact

Monitoring Cost Management

Observability can become expensive.

Major cost drivers include:

Log ingestion volume
Log retention
Metric cardinality
Query frequency
Duplicate collection
Trace sampling
Cross-region transfer
SIEM ingestion

Cost Optimisation Controls

Use:

Appropriate retention periods
Log-level management
Metric-cardinality controls
Trace sampling
Data filtering
Archive strategies
Dashboard query optimisation
Chargeback or showback
Regular usage reviews

Do not remove critical security telemetry solely for cost savings.

Troubleshooting Workflow

Service Alert

↓

Check Application Dashboard

↓

Review Pod and Deployment Status

↓

Review Kubernetes Events

↓

Review Application Logs

↓

Review Node and Platform Metrics

↓

Review Network and DNS Data

↓

Review Recent Changes

↓

Identify Root Cause

Common Monitoring Commands

kubectl get nodes

kubectl get pods -A

kubectl get deployments -A

kubectl get events -A \
  --sort-by='.metadata.creationTimestamp'

kubectl top nodes

kubectl top pods -A

The Metrics Server or a suitable metrics provider must be available for kubectl top.

Investigate a CrashLoopBackOff

Pod in CrashLoopBackOff

↓

Describe Pod

↓

Review Current Logs

↓

Review Previous Logs

↓

Check Configuration and Secrets

↓

Review Resource Limits

↓

Review Probe Failures

↓

Review Recent Deployment Changes

Commands:

kubectl describe pod <pod-name> \
  -n <namespace>

kubectl logs <pod-name> \
  -n <namespace>

kubectl logs <pod-name> \
  -n <namespace> \
  --previous

Investigate a Pending Pod

Check:

Resource requests
Node capacity
Taints
Tolerations
Node selectors
Affinity
Persistent volumes
Pod IP availability
Admission denials
Scheduling events

Investigate a Node NotReady

Check:

Node conditions
kubelet
Container runtime
Network connectivity
Disk pressure
Memory pressure
IAM permissions
CNI health
EC2 instance health

Common Logging and Monitoring Failures

Control Plane Logging Disabled

Risk: Administrative and security activity cannot be fully investigated.

Control: Enable required EKS control plane log types.

Logs Collected but Not Reviewed

Risk: Suspicious activity remains undetected.

Control: Create alerts, dashboards and SOC procedures.

Short Retention

Risk: Evidence is unavailable when an incident is discovered late.

Control: Align retention with risk and compliance requirements.

Sensitive Data in Logs

Risk: Logging infrastructure becomes a source of credential or data exposure.

Control: Apply redaction and least-privilege access.

Missing Cluster Metadata

Risk: Analysts cannot identify the affected owner or environment.

Control: Enrich logs with standard metadata.

Excessive Debug Logging

Risk: High cost, reduced performance and sensitive-data exposure.

Control: Define production logging levels.

Unbounded Metric Cardinality

Risk: Monitoring cost and performance become unmanageable.

Control: Restrict dynamic metric labels.

No Monitoring of the Log Pipeline

Risk: The organisation assumes visibility exists when ingestion has failed.

Control: Monitor collectors, delivery and data freshness.

Alerts Without Owners

Risk: No team investigates the condition.

Control: Assign every alert and resource to an accountable owner.

Duplicate Monitoring Tools

Risk: Increased cost, inconsistent data and alert duplication.

Control: Define an enterprise observability architecture.

Shared SIEM Credentials

Risk: Reduced accountability and broad access.

Control: Use workload identities and separate integration roles.

Enterprise EKS Observability Architecture

Amazon EKS Clusters

├── Control Plane Logs
├── Kubernetes Audit Logs
├── Application Logs
├── Node Logs
├── Kubernetes Events
├── Platform Add-On Logs
├── Prometheus Metrics
├── Distributed Traces
├── Runtime Security Alerts
└── Network Telemetry

↓

Collection Layer

├── CloudWatch Observability
├── Fluent Bit
├── Prometheus Collectors
└── OpenTelemetry Collectors

↓

AWS Observability and Security Services

├── CloudWatch Logs
├── CloudWatch Metrics
├── Amazon Managed Service for Prometheus
├── Amazon Managed Grafana
├── AWS X-Ray
├── GuardDuty
├── Inspector
└── Security Hub

↓

Central Logging and Security Account

↓

Enterprise SIEM

↓

Platform Operations, SRE, SOC and Compliance

Production Monitoring Baseline

Area	Minimum Requirement
Control plane	Required log types enabled
Kubernetes API	Audit logging enabled
Applications	Structured stdout and stderr logging
Nodes	Node and runtime telemetry collected
Metrics	Cluster, node, Pod and workload metrics
Security	Runtime and policy alerts
Networks	Flow, load-balancer and DNS visibility
Storage	Retention and encryption defined
Access	Least-privilege log access
Alerting	Owned and tested alert rules
SIEM	Critical security telemetry integrated
Pipeline health	Data completeness monitored
Governance	Evidence and retention documented

Enterprise Implementation Strategy

Phase 1 — Define Requirements

Identify critical business services.
Identify regulatory requirements.
Define operational use cases.
Define security detections.
Define retention periods.
Define recovery requirements.
Identify telemetry owners.

Phase 2 — Inventory Telemetry Sources

Inventory EKS clusters.
Identify control plane logging status.
Identify application logging.
Identify node telemetry.
Identify platform add-ons.
Identify network logs.
Identify security-service findings.
Document visibility gaps.

Phase 3 — Enable Control Plane Visibility

Enable API Server logs.
Enable Audit logs.
Enable Authenticator logs.
Enable Controller Manager logs.
Enable Scheduler logs.
Configure retention.
Protect log groups.
Monitor delivery.

Phase 4 — Collect Workload and Node Logs

Standardise application logging.
Deploy approved collectors.
Enrich logs with Kubernetes metadata.
Collect node and runtime logs.
Apply redaction.
Test buffering and failure handling.

Phase 5 — Implement Metrics and Dashboards

Collect cluster metrics.
Collect node and Pod metrics.
Monitor platform add-ons.
Define application SLIs.
Build platform dashboards.
Build application dashboards.
Establish SLO reporting.

Phase 6 — Implement Security Monitoring

Create audit-log detections.
Monitor privileged activity.
Monitor Secret access.
Monitor policy violations.
Integrate runtime alerts.
Monitor public exposure.
Monitor logging changes.

Phase 7 — Centralise Enterprise Telemetry

Establish a central logging account.
Forward required logs.
Aggregate multi-account findings.
Integrate with the SIEM.
Standardise event schemas.
Protect cross-account roles.

Phase 8 — Build Alert Operations

Assign alert owners.
Define severity.
Attach runbooks.
Define escalation paths.
Test notification delivery.
Tune false positives.
Measure response time.

Phase 9 — Protect and Govern Data

Encrypt logs.
Restrict access.
Define retention.
Implement archive requirements.
Monitor deletion and configuration changes.
Apply separation of duties.
Maintain compliance evidence.

Phase 10 — Validate Continuously

Reconcile expected and reporting clusters.
Test log-delivery failure alerts.
Test SIEM ingestion.
Test security detections.
Review dashboard usefulness.
Review monitoring costs.
Conduct incident-response exercises.
Improve coverage after incidents.

Enterprise Best Practices

As a Cloud Security Engineer:

Enable the required Amazon EKS control plane logs.
Treat Kubernetes audit logs as critical security evidence.
Combine CloudTrail with Kubernetes audit logs.
Standardise structured application logging.
Prevent credentials and sensitive data from entering logs.
Collect worker-node and platform-component logs.
Use CloudWatch for AWS-native logs, metrics and alarms.
Use Prometheus-compatible monitoring for Kubernetes and application metrics.
Control metric cardinality.
Create separate platform, application, security and executive dashboards.
Monitor CoreDNS, VPC CNI and admission controllers.
Integrate critical telemetry with the enterprise SIEM.
Centralise logs in a protected security or logging account.
Encrypt logs and restrict access.
Define retention by risk and compliance requirements.
Monitor collectors and telemetry-pipeline health.
Alert when logging is disabled or delivery stops.
Assign every alert to an owner.
Attach actionable runbooks to critical alerts.
Tune detections to reduce alert fatigue.
Reconcile the cluster inventory with telemetry coverage.
Test logging, alerting and incident-response workflows regularly.
Review observability cost without sacrificing essential security visibility.

Real-World Scenario

A multinational financial organisation operates more than 350 Amazon EKS clusters across development, testing and production AWS accounts.

The environment supports:

Customer banking
Payment processing
Fraud detection
Identity services
Internal analytics

A security review identifies:

Inconsistent EKS control plane logging
Kubernetes audit logs enabled only in selected clusters
Application logs stored locally on worker nodes
No central view of cluster health
Excessive debug logging
Secret values appearing in error messages
High Prometheus metric cardinality
Runtime alerts not integrated with the SIEM
No alerts when log collectors fail
Different retention periods across teams
Unowned CloudWatch alarms

The organisation launches an enterprise EKS observability programme.

The Platform, SRE and Cloud Security teams:

Create a central inventory of all EKS clusters.
Define a mandatory production telemetry baseline.
Enable all required control plane log types.
Centralise Kubernetes audit logs in a protected logging account.
Deploy standard log collectors to worker nodes.
Require applications to use structured JSON logging.
Implement automatic redaction for passwords and tokens.
Deploy CloudWatch Container Insights for AWS-native visibility.
Collect Prometheus-compatible application and platform metrics.
Use a managed metrics environment for multi-cluster aggregation.
Build Grafana dashboards for platform and application teams.
Create SIEM rules for Secret access and cluster-admin changes.
Integrate GuardDuty, Inspector and runtime findings.
Enable VPC Flow Logs and load-balancer access logs.
Create alerts for collector failure and telemetry delay.
Standardise retention and encryption policies.
Assign owners and runbooks to every critical alarm.
Implement monitoring-cost dashboards.
Conduct quarterly detection and incident-response exercises.
Review telemetry coverage against the cluster inventory.

The programme provides:

Faster security detection
Improved incident investigation
Better application troubleshooting
Consistent audit evidence
Reduced logging gaps
Stronger multi-cluster visibility
Controlled observability costs
Clear operational accountability

Key Takeaways

Logging records events, monitoring measures conditions and observability combines logs, metrics and traces.
Amazon EKS provides five managed control plane log types.
Kubernetes audit logs are essential for investigating API activity.
CloudTrail and Kubernetes audit logs provide different but complementary visibility.
Application logs should be structured and must not expose secrets.
Worker-node and platform add-on logs are important for troubleshooting and forensics.
CloudWatch provides AWS-native logs, metrics, dashboards and alarms.
Prometheus provides Kubernetes-focused time-series monitoring.
Grafana provides dashboard and visualisation capabilities.
Security monitoring should detect privileged access, Secret access and policy changes.
Critical telemetry should be centralised and integrated with the SIEM.
Logs must be encrypted, retained and protected against deletion.
Monitoring pipelines must themselves be monitored.
Alert ownership and runbooks are required for effective response.
Continuous coverage validation prevents false confidence.

Knowledge Check

1. What are the five Amazon EKS control plane log types?

Answer:

API Server
Audit
Authenticator
Controller Manager
Scheduler

2. What is the difference between CloudTrail and Kubernetes audit logs?

Answer: CloudTrail records AWS API activity affecting the EKS service and related AWS resources, while Kubernetes audit logs record requests made to the Kubernetes API inside the cluster.

3. Why should applications use structured logging?

Answer: Structured logging makes records easier to search, filter, correlate, parse and analyse across dashboards and security platforms.

4. Why must the logging and monitoring pipeline itself be monitored?

Answer: Collectors, integrations or delivery pipelines may fail. Without pipeline-health monitoring, teams may incorrectly believe they have visibility while important telemetry is missing.

5. What is metric cardinality, and why must it be controlled?

Answer: Metric cardinality is the number of unique combinations of metric labels. Unbounded labels can create excessive storage, memory use, query latency and monitoring cost.

What’s Next?

In the next lesson, we will explore Amazon EKS Runtime Security, including runtime threat detection, suspicious process activity, container escape indicators, Falco, GuardDuty, workload investigation and enterprise response patterns.

➡️ Next Lesson: Lesson 08 — Runtime Security