Lesson 06 — Runtime Detection

Learning Objectives

By the end of this lesson, you will be able to:

Understand Runtime Detection in Kubernetes
Learn why runtime security is essential
Understand common runtime attack techniques
Explore runtime detection architecture
Integrate Falco, Audit Logs and Prometheus
Learn enterprise runtime monitoring strategies
Apply runtime detection best practices for Amazon EKS

Why This Matters

Preventive security controls reduce risk before workloads reach production.

Examples include:

Secure Dockerfiles
Image Vulnerability Scanning
Image Signing
SBOM Generation
Admission Controllers
RBAC
Network Policies

However, no preventive control is perfect.

Attackers may still gain access through:

Zero-day vulnerabilities
Stolen credentials
Supply chain attacks
Insider threats
Misconfigurations
Application vulnerabilities

Runtime Detection provides the final layer of defence by monitoring workloads after deployment.

What is Runtime Detection?

Runtime Detection is the continuous monitoring of running Kubernetes workloads to identify malicious or abnormal behaviour.

Unlike vulnerability scanning, Runtime Detection answers questions such as:

Is a container spawning an unexpected shell?
Is malware executing?
Is someone modifying sensitive files?
Is a container attempting privilege escalation?
Is unusual network communication occurring?

Runtime Detection focuses on behaviour, not vulnerabilities.

Runtime Security Lifecycle

Build

↓

Scan

↓

Deploy

↓

Running Workload

↓

Runtime Monitoring

↓

Threat Detection

↓

Alert

↓

Incident Response

Security continues throughout the application’s lifecycle.

Runtime vs Build-Time Security

Build-Time Security	Runtime Detection
Prevents known risks	Detects active attacks
Image Scanning	Behaviour Monitoring
Dockerfile Analysis	System Call Monitoring
SBOM	Process Monitoring
Image Signing	Runtime Threat Detection
Admission Policies	Security Alerts

Both approaches are required for comprehensive Kubernetes security.

Runtime Threats

Cloud-native environments face numerous runtime threats.

Examples include:

Reverse shells
Cryptocurrency miners
Container escapes
Privilege escalation
Malware execution
Unexpected processes
File modification
Data exfiltration
Lateral movement
Secret theft

These attacks occur after the workload is already running.

Runtime Detection Workflow

Running Container

↓

Runtime Event

↓

Detection Engine

↓

Security Rule

↓

Alert

↓

SOC Investigation

Every significant runtime event is evaluated against security policies.

Runtime Telemetry Sources

Enterprise Runtime Detection combines multiple sources.

Containers

↓

System Calls

↓

Kubernetes Audit Logs

↓

Application Logs

↓

Prometheus Metrics

↓

Cloud Logs

↓

Detection Platform

Combining multiple telemetry sources provides greater accuracy.

System Call Monitoring

The Linux kernel records system calls generated by applications.

Examples include:

execve()
open()
connect()
mount()
chmod()
ptrace()

Falco monitors these events to detect suspicious behaviour.

Process Monitoring

Runtime Detection monitors process activity inside containers.

Application

↓

New Process

↓

Expected?

↓

Yes → Continue

No → Alert

Unexpected process execution often indicates compromise.

File Integrity Monitoring

Sensitive files should rarely change during runtime.

Examples include:

/etc/passwd
/etc/shadow
Kubernetes configuration files
Application binaries
Security certificates

Unexpected modifications may indicate malicious activity.

Network Behaviour Monitoring

Runtime Detection also analyses network activity.

Examples include:

Outbound connections
DNS requests
Communication between Pods
Internet access
External API communication

Unexpected network behaviour often indicates malware or data exfiltration.

Container Escape Detection

Container escapes are among the highest-risk runtime attacks.

Example:

Compromised Container

↓

Host Access Attempt

↓

Falco Rule Triggered

↓

Critical Alert

Immediate investigation is required when container escape attempts are detected.

Privilege Escalation Detection

Attackers often attempt to obtain elevated privileges.

Common indicators include:

Running privileged commands
Loading kernel modules
Mounting host filesystems
Accessing Docker socket
Changing Linux capabilities

Runtime Detection identifies these activities immediately.

Runtime Detection Architecture

Amazon EKS

↓

Running Containers

↓

Falco

↓

Kubernetes Audit Logs

↓

Prometheus

↓

Amazon CloudWatch

↓

AWS Security Hub

↓

Enterprise SIEM

↓

Security Operations Centre (SOC)

Multiple monitoring systems work together to detect threats.

Correlating Runtime Events

Enterprise security platforms correlate multiple signals.

Example:

Interactive Shell

+

CPU Spike

+

Outbound Connection

+

Audit Log

↓

High Confidence Security Incident

Correlation reduces false positives and improves detection accuracy.

Runtime Detection with Amazon EKS

A typical Amazon EKS deployment includes:

Falco DaemonSet
Kubernetes Audit Logs
Amazon CloudWatch
Amazon Managed Service for Prometheus
Amazon Managed Grafana
AWS Security Hub
Amazon GuardDuty
Enterprise SIEM

Together these services provide comprehensive runtime visibility.

Enterprise Runtime Monitoring Workflow

Workload

↓

Runtime Activity

↓

Falco Detection

↓

CloudWatch Logs

↓

Security Hub Finding

↓

SIEM Correlation

↓

SOC Investigation

↓

Incident Response

Automated workflows significantly reduce response times.

Enterprise Example

A multinational banking organization hosts payment services on Amazon EKS.

An attacker exploits a vulnerable application and uploads a cryptocurrency miner.

Within seconds:

Falco detects an unexpected process.
CPU utilization rapidly increases.
Prometheus records abnormal resource consumption.
Grafana dashboards highlight the affected node.
Kubernetes Audit Logs reveal the attacker executed kubectl exec.
AWS Security Hub creates a Critical finding.
The SIEM correlates all events.
The SOC isolates the affected Pod and begins forensic analysis.

The attack is contained before it impacts customer transactions.

Common Runtime Detection Challenges

Cloud Security Engineers frequently encounter:

Runtime monitoring disabled
Excessive false positives
Missing custom detection rules
No alert prioritisation
No telemetry correlation
Short log retention
Missing incident response playbooks
Unmonitored namespaces
Incomplete visibility across clusters
Delayed alerting

Proper tuning is essential for effective runtime detection.

Enterprise Monitoring

Security teams should continuously monitor:

Interactive shell sessions
New process execution
File modifications
Container escapes
Privilege escalation
Reverse shells
Unexpected network traffic
Secret access
High CPU usage
Kubernetes Audit events

These activities often indicate active attacks.

Enterprise Implementation Strategy

A recommended rollout:

Step 1

↓

Deploy Falco

↓

Step 2

↓

Enable Kubernetes Audit Logs

↓

Step 3

↓

Collect Prometheus Metrics

↓

Step 4

↓

Configure Grafana Dashboards

↓

Step 5

↓

Enable AWS Security Hub

↓

Step 6

↓

Integrate Enterprise SIEM

↓

Step 7

↓

Tune Detection Rules

↓

Step 8

↓

Perform Continuous Runtime Monitoring

This layered approach provides strong runtime protection.

Enterprise Runtime Security Architecture

Amazon EKS

↓

Running Pods

↓

Falco DaemonSet

↓

Kubernetes Audit Logs

↓

Amazon CloudWatch

↓

Prometheus

↓

Amazon Managed Grafana

↓

AWS Security Hub

↓

Amazon GuardDuty

↓

Enterprise SIEM

↓

Security Operations Centre (SOC)

This architecture combines behavioural detection, monitoring and centralized security operations.

Best Practices

As a Kubernetes Security Engineer:

Monitor every production workload at runtime.
Deploy Falco on every worker node.
Enable Kubernetes Audit Logging.
Correlate runtime events with logs and metrics.
Continuously tune Falco rules to reduce false positives.
Monitor privileged containers closely.
Alert on unexpected process execution.
Investigate unusual network behaviour immediately.
Integrate runtime alerts into enterprise SIEM platforms.
Regularly test runtime detection capabilities through security exercises.

Runtime Detection should be treated as a continuous security capability rather than a one-time deployment.

Real-World Scenario

A global healthcare provider operates electronic medical record systems on Amazon EKS.

An attacker gains access to a vulnerable application container and attempts to:

Launch a shell
Download malicious software
Access Kubernetes Secrets
Connect to an external command-and-control server

Runtime Detection immediately identifies:

Interactive shell execution
Suspicious network connections
Unexpected file modifications
Secret access attempts

AWS Security Hub generates a high-severity finding while the SIEM correlates supporting evidence from Prometheus, Kubernetes Audit Logs and CloudWatch.

The SOC quarantines the affected workload before patient data is exposed.

Key Takeaways

After completing this lesson, you should understand:

What Runtime Detection is
Runtime vs build-time security
Common runtime attack techniques
Runtime monitoring architecture
Behaviour-based threat detection
Falco integration
Prometheus and Grafana correlation
AWS Security Hub integration
Enterprise runtime detection best practices

Runtime Detection is a critical layer of Kubernetes security that identifies active threats after workloads have been deployed. By combining Falco, Kubernetes Audit Logs, Prometheus, Grafana and AWS security services, organizations gain real-time visibility into malicious behaviour and can respond quickly to security incidents within Amazon EKS environments.

Knowledge Check

Question 1

What is the primary purpose of Runtime Detection?

A. Build container images
B. Monitor running workloads for suspicious or malicious behaviour
C. Replace Kubernetes RBAC
D. Store application logs

Answer: B

Question 2

Which of the following is an example of a runtime threat?

A. Reverse shell inside a running container
B. Creating a Dockerfile
C. Deploying a Helm chart
D. Creating a Kubernetes Namespace

Answer: A

Question 3

Which open-source tool is commonly used for Kubernetes runtime threat detection?

A. Helm
B. Falco
C. Argo CD
D. Terraform

Answer: B

Question 4

Why is correlating runtime events with logs and metrics valuable?

A. It reduces storage costs.
B. It improves detection accuracy and reduces false positives.
C. It replaces vulnerability scanning.
D. It automatically patches containers.

Answer: B

Question 5

Which combination represents enterprise best practice?

A. Deploy Falco, enable Audit Logs, collect Prometheus metrics, visualize data with Grafana, integrate AWS Security Hub and an enterprise SIEM, and continuously tune runtime detection rules.
B. Monitor workloads only during maintenance windows.
C. Disable runtime alerts to reduce notifications.
D. Depend solely on vulnerability scanning for production security.

Answer: A

What’s Next?

In the next lesson, you will learn about Security Monitoring, where you’ll explore how enterprise organizations continuously monitor Kubernetes environments, correlate security telemetry, build monitoring strategies, and establish proactive detection capabilities for Amazon EKS.

➡️ Next Lesson: Lesson 07 — Security Monitoring