Lab 05 — Runtime Security Validation

Mission Information

Item	Details
Lab ID	K8S-WORKLOAD-LAB-05
Difficulty	Advanced
Estimated Time	4–5 Hours
Environment	Dedicated Kubernetes Training Cluster
Platform	Amazon EKS / Azure AKS / Google GKE / kind / minikube
Cost	Free for Local Cluster / Cloud Charges May Apply
Primary Role	Kubernetes Security Engineer
Supporting Roles	SOC Analyst, Incident Responder, Platform Engineer, DevSecOps Engineer
Module	Kubernetes Workload and Pod Security
Previous Lab	Lab 04 — Harden Production Pods
Next Item	Runbook 01 — Kubernetes Workload Security Assessment

Mission Scenario

CloudNova Technologies has hardened its production Kubernetes workloads using:

Non-root execution
Restricted Pod Security enforcement
Read-only root filesystems
Dropped Linux capabilities
RuntimeDefault seccomp profiles
Network Policies
Resource controls
Secure Service Account configuration

Although these preventive controls significantly reduce workload risk, the Security Operations Centre has identified an important limitation:

A workload can still become compromised through:

An application vulnerability
A malicious dependency
A compromised container image
Stolen credentials
Supply-chain attacks
Misused administrative access
Previously unknown vulnerabilities

Preventive controls reduce the likelihood and impact of compromise, but they do not provide complete visibility into what happens after a container starts.

The CISO has directed the Kubernetes Security team to implement runtime security validation capable of identifying:

Unexpected shell execution
Sensitive file access
Package installation attempts
Privilege-related activity
Container filesystem changes
Suspicious network connections
Kubernetes credential access
Container drift
Abnormal process execution

Your mission is to deploy a monitored workload, install or access a runtime detection platform, safely simulate suspicious activity, validate generated alerts, analyse the findings, and produce an enterprise runtime security report.

Learning Objectives

By completing this lab, you will learn how to:

Understand Kubernetes runtime security
Distinguish preventive and detective security controls
Establish expected workload behaviour
Inspect processes running inside containers
Identify unexpected shell execution
Detect sensitive file access
Detect package management activity
Detect container drift
Review network activity
Validate privilege-related protections
Use Falco for runtime threat detection
Review Kubernetes and container runtime events
Correlate runtime alerts with workload metadata
Classify runtime security findings
Produce an enterprise runtime validation report

Enterprise Runtime Security Architecture

                    Kubernetes Cluster
                            │
          ┌─────────────────┴─────────────────┐
          │                                   │
   Hardened Workloads                  Kubernetes Nodes
          │                                   │
   Application Processes               Container Runtime
          │                                   │
          └─────────────────┬─────────────────┘
                            │
                     Runtime Telemetry
                            │
             ┌──────────────┼──────────────┐
             │              │              │
          Falco         Audit Logs      Flow Logs
             │              │              │
             └──────────────┼──────────────┘
                            │
                     Security Platform
                            │
              SOC Investigation and Response

Runtime Security Model

Prevent

Pod Security Standards
Security Contexts
Network Policies
Image Scanning

        │

        ▼

Observe

Processes
Files
Network
System Calls
Kubernetes Events

        │

        ▼

Detect

Unexpected Shells
Sensitive File Access
Privilege Activity
Container Drift
Suspicious Connections

        │

        ▼

Respond

Investigate
Contain
Preserve Evidence
Remediate
Improve Controls

Lab Outcomes

By the end of this lab, you will have:

Created a runtime security namespace
Deployed a hardened application workload
Established a workload behaviour baseline
Installed or validated Falco
Reviewed active container processes
Simulated suspicious shell activity
Simulated sensitive file access
Tested blocked filesystem modification
Tested package installation behaviour
Tested Kubernetes credential discovery
Reviewed network connections and DNS activity
Generated and analysed runtime alerts
Correlated alerts with Kubernetes metadata
Classified runtime findings
Produced an enterprise runtime security validation report

Important Safety Notice

Perform this lab only in an authorised training environment.

Do not:

Execute malware
Download unauthorised offensive tools
Modify Kubernetes worker nodes
Access real production credentials
Exfiltrate data
Disable security controls in production
Perform destructive runtime tests

The simulations in this lab use harmless commands to demonstrate detection behaviour.

Prerequisites

Before starting, ensure that you have:

Completed Labs 01–04
A running Kubernetes cluster
kubectl installed and configured
Permission to create namespaces, Deployments, Pods, Services, and Network Policies
Permission to install Falco, or access to an environment where Falco is already installed
Helm installed if using the Helm deployment method
A CNI plugin that supports Network Policies
Visual Studio Code or another YAML editor
Git Bash or PowerShell
Basic knowledge of Linux processes, filesystems, and networking

Tools Used

Tool	Purpose
kubectl	Deploy and inspect Kubernetes workloads
Falco	Detect suspicious runtime behaviour
Helm	Install Falco
BusyBox	Perform controlled runtime tests
nginx-unprivileged	Run the hardened application
Kubernetes Events	Review workload lifecycle activity
Container Logs	Review application and security events
Visual Studio Code	Create manifests and reports
Git Bash / PowerShell	Execute commands

Recommended Lab File Structure

lab-05-runtime-security-validation/
├── 01-namespace.yaml
├── 02-runtime-deployment.yaml
├── 03-runtime-service.yaml
├── 04-network-policies.yaml
├── 05-authorised-test-pod.yaml
├── 06-suspicious-test-pod.yaml
├── 07-falco-custom-rules.yaml
├── evidence/
└── runtime-security-report.md

Task 01 — Verify Cluster Health

Verify cluster connectivity.

kubectl cluster-info

Review the nodes.

kubectl get nodes -o wide

Review system components.

kubectl get pods -n kube-system

Confirm:

Kubernetes API is reachable.
Worker nodes report Ready.
Container runtime and networking components are healthy.
You are connected to the correct training cluster.

Task 02 — Create the Runtime Security Namespace

Create 01-namespace.yaml.

apiVersion: v1
kind: Namespace
metadata:
  name: runtime-security
  labels:
    environment: training
    owner: cloud-security
    security-monitoring: enabled
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest

Apply the namespace.

kubectl apply -f 01-namespace.yaml

Verify the labels.

kubectl get namespace runtime-security --show-labels

Task 03 — Deploy the Hardened Monitored Workload

Create 02-runtime-deployment.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: monitored-web
  namespace: runtime-security
  labels:
    app: monitored-web
    environment: training
    security-monitoring: enabled
spec:
  replicas: 2
  selector:
    matchLabels:
      app: monitored-web
  template:
    metadata:
      labels:
        app: monitored-web
        environment: training
        security-monitoring: enabled
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 101
        runAsGroup: 101
        fsGroup: 101
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: web
          image: nginxinc/nginx-unprivileged:1.27-alpine
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
          securityContext:
            privileged: false
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 128Mi
          readinessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 15
            periodSeconds: 20
          volumeMounts:
            - name: nginx-cache
              mountPath: /var/cache/nginx
            - name: nginx-run
              mountPath: /var/run
            - name: temporary-files
              mountPath: /tmp
      volumes:
        - name: nginx-cache
          emptyDir:
            sizeLimit: 50Mi
        - name: nginx-run
          emptyDir:
            sizeLimit: 10Mi
        - name: temporary-files
          emptyDir:
            sizeLimit: 50Mi

Apply the Deployment.

kubectl apply -f 02-runtime-deployment.yaml

Verify the rollout.

kubectl rollout status deployment/monitored-web \
  -n runtime-security

List the Pods.

kubectl get pods \
  -n runtime-security \
  -l app=monitored-web \
  -o wide

Task 04 — Create the Internal Service

Create 03-runtime-service.yaml.

apiVersion: v1
kind: Service
metadata:
  name: monitored-web
  namespace: runtime-security
  labels:
    app: monitored-web
spec:
  type: ClusterIP
  selector:
    app: monitored-web
  ports:
    - name: http
      port: 80
      targetPort: 8080

Apply the Service.

kubectl apply -f 03-runtime-service.yaml

Verify it.

kubectl get service monitored-web \
  -n runtime-security

Task 05 — Apply Runtime Network Isolation

Create 04-network-policies.yaml.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: runtime-security
spec:
  podSelector: {}
  policyTypes:
    - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: runtime-security
spec:
  podSelector: {}
  policyTypes:
    - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-approved-web-access
  namespace: runtime-security
spec:
  podSelector:
    matchLabels:
      app: monitored-web
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              runtime-access: approved
      ports:
        - protocol: TCP
          port: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: runtime-security
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Apply the policies.

kubectl apply -f 04-network-policies.yaml

Verify them.

kubectl get networkpolicy -n runtime-security

Task 06 — Establish the Expected Behaviour Baseline

Before testing suspicious actions, document normal workload behaviour.

Store one application Pod name.

Git Bash

POD_NAME=$(kubectl get pod \
  -n runtime-security \
  -l app=monitored-web \
  -o jsonpath='{.items[0].metadata.name}')

PowerShell

$POD_NAME = kubectl get pod `
  -n runtime-security `
  -l app=monitored-web `
  -o jsonpath='{.items[0].metadata.name}'

Review the runtime identity.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- id

Review the main process.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- ps

Review listening ports.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- sh -c "netstat -lnt 2>/dev/null || ss -lnt 2>/dev/null || true"

Review mounted filesystems.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- mount

Review environment variables without exposing secrets.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- env

Record expected behaviour.

Behaviour Category	Expected Activity
Primary process	NGINX
Runtime user	UID 101
Listening port	TCP 8080
Writable paths	`/tmp`, `/var/run`, `/var/cache/nginx`
Kubernetes token	Not mounted
Shell usage	Administrative testing only
Package installation	Not expected
External connections	Not expected
System file modification	Not expected

Task 07 — Install Falco with Helm

Add the Falco Helm repository.

helm repo add falcosecurity \
  https://falcosecurity.github.io/charts

Update the repositories.

helm repo update

Create the Falco namespace.

kubectl create namespace falco

Install Falco.

helm install falco falcosecurity/falco \
  --namespace falco \
  --set tty=true

Verify the installation.

kubectl get pods -n falco

Falco is commonly deployed as a DaemonSet so that runtime activity can be observed across Kubernetes worker nodes.

Check the DaemonSet.

kubectl get daemonset -n falco

Environment Note

Falco driver support depends on the Kubernetes platform, host kernel, container runtime, and installation method.

In some managed or local environments, additional configuration may be required for:

Modern eBPF
Kernel modules
Driver Loader
Host access
Managed-node restrictions

Use the Falco installation method approved for your environment.

Task 08 — Validate Falco Health

Review the Falco Pods.

kubectl get pods -n falco -o wide

Review recent logs.

kubectl logs \
  -n falco \
  -l app.kubernetes.io/name=falco \
  --tail=50

Look for:

Falco engine started
Rules loaded
Driver loaded
Event source available
No repeating fatal errors

Describe a Falco Pod if troubleshooting is required.

kubectl describe pod \
  -n falco \
  -l app.kubernetes.io/name=falco

Task 09 — Review Falco’s Runtime Role

Falco observes runtime activity and evaluates events against behavioural rules.

Examples of activity that may generate alerts include:

Shell opened inside a container
Sensitive file read
Package manager executed
Binary executed from an unusual directory
Privileged container started
Container modifying sensitive directories
Unexpected outbound connection
Kubernetes credential file access
System administration tools executed

Falco does not automatically replace:

Image scanning
Pod Security admission
Kubernetes RBAC
Network Policies
Incident response
SIEM correlation

It provides a detective security layer.

Task 10 — Monitor Falco Alerts

Open a separate terminal and stream Falco logs.

kubectl logs \
  -n falco \
  -l app.kubernetes.io/name=falco \
  --follow

Keep this terminal open while completing the runtime simulations.

In environments with multiple Falco Pods, select the Pod running on the same node as the monitored application.

Find the application node.

kubectl get pod "$POD_NAME" \
  -n runtime-security \
  -o wide

Find the Falco Pod on that node.

kubectl get pods -n falco -o wide

Stream logs from the matching Falco Pod.

kubectl logs -n falco <falco-pod-name> --follow

Task 11 — Simulate Interactive Shell Execution

Open a shell inside the monitored application.

kubectl exec -it "$POD_NAME" \
  -n runtime-security \
  -- sh

Inside the container, run:

id

pwd

ps

Exit.

exit

Review the Falco terminal.

Depending on the active ruleset, you may observe an alert related to:

Shell spawned in a container
Terminal shell in a container
Interactive process execution

Record:

Timestamp
Rule name
Priority
Pod name
Namespace
Container name
Command
User identity

Task 12 — Simulate Sensitive File Access

Read a sensitive system file.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- cat /etc/shadow

Expected application-level result may be:

Permission denied

Security Analysis

Even when an operating-system permission blocks access, the attempted action can still be security-relevant.

Detection of failed actions helps identify reconnaissance and attacker intent.

Task 13 — Simulate Root Filesystem Modification

Attempt to create a file under /etc.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- touch /etc/runtime-test

Expected:

Read-only file system

Attempt to modify application content.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- sh -c "echo modified > /usr/share/nginx/html/runtime-test.html"

Expected:

Read-only file system

Review whether the runtime platform records:

Failed write activity
File-open events
Write attempts to sensitive paths

Validation Result

Preventive control:

Read-Only Root Filesystem

Detective control:

Runtime Event Monitoring

Both controls contribute to defence in depth.

Task 14 — Validate Approved Writable Paths

Write to the approved temporary directory.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- sh -c "echo approved-runtime-data > /tmp/runtime-test.txt"

Verify it.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- cat /tmp/runtime-test.txt

Review whether this produces an alert.

Expected:

Normal writes to approved application directories should not automatically be treated as malicious.
Detection rules should focus on unexpected or high-risk behaviour.

Task 15 — Simulate Package Manager Execution

Check whether the package manager exists.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- sh -c "command -v apk || command -v apt || command -v yum || true"

For the Alpine-based container, test:

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- apk --version

Do not install packages.

Review Falco for an alert related to:

Package management process launched
Package manager executed inside a container

Security Analysis

Production containers should generally not install packages at runtime.

Required software should be:

Added during image build.
Scanned.
Tested.
Approved.
Redeployed as a new immutable image.

Task 16 — Simulate Credential Discovery

Attempt to list the Kubernetes Service Account token directory.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- sh -c "ls -la /var/run/secrets/kubernetes.io/serviceaccount"

Expected:

No such file or directory

Search for token-like files without reading any real secrets.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- sh -c "find /var/run/secrets -maxdepth 4 -type f 2>/dev/null"

Expected:

No automatically mounted Kubernetes Service Account token.

Review runtime alerts for credential-file discovery attempts.

Task 17 — Validate Privilege Escalation Controls

Check the process status.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- sh -c "grep '^NoNewPrivs' /proc/1/status"

Expected:

NoNewPrivs: 1

Review effective capabilities.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- sh -c "grep '^CapEff' /proc/1/status"

Expected:

CapEff: 0000000000000000

Attempt to create a device node.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- mknod /tmp/test-device c 1 3

Expected:

Operation not permitted

Record whether the failed operation is visible in runtime telemetry.

Task 18 — Review Active Processes

List processes inside the monitored container.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- ps

Review process information from Kubernetes.

kubectl top pod "$POD_NAME" \
  -n runtime-security

The kubectl top command requires Metrics Server.

If unavailable, continue using Pod status, logs, and runtime security telemetry.

Build a process baseline.

Process	Expected	Action
nginx master process	Yes	Allow
nginx worker process	Yes	Allow
sh started through approved test	Temporary	Investigate or suppress for lab
Package manager	No	Alert
Network scanner	No	Alert
Compiler	No	Alert
Unknown binary from `/tmp`	No	High-priority alert

Task 19 — Inspect Container Drift

Container drift occurs when the runtime state differs from the approved image or expected workload behaviour.

Examples include:

New executable files
Modified configuration files
Downloaded scripts
Unexpected processes
Runtime package installation
New scheduled tasks
Unapproved certificates or keys

Review writable directories.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- find /tmp -maxdepth 2 -type f -ls

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- find /var/run -maxdepth 2 -type f -ls

Document all files that are expected during normal application operation.

Task 20 — Simulate Harmless Container Drift

Create a harmless script inside the approved temporary volume.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- sh -c "printf '#!/bin/sh\necho runtime-test\n' > /tmp/runtime-test.sh"

Make it executable.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- chmod +x /tmp/runtime-test.sh

Execute it.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- /tmp/runtime-test.sh

Expected:

runtime-test

Review Falco for rules related to:

Execution from /tmp
Newly created executable
Unexpected binary execution
Shell activity

Remove the test script.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- rm -f /tmp/runtime-test.sh

Security Finding

Execution from temporary directories is uncommon for many production web applications and may indicate:

Downloaded malware
Living-off-the-land activity
Staged exploit payloads
Container drift
Persistence attempts

Task 21 — Create an Authorised Test Pod

Create 05-authorised-test-pod.yaml.

apiVersion: v1
kind: Pod
metadata:
  name: authorised-runtime-test
  namespace: runtime-security
  labels:
    runtime-access: approved
spec:
  automountServiceAccountToken: false
  restartPolicy: Never
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    runAsGroup: 10001
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: tester
      image: busybox:1.36
      command:
        - sh
        - -c
        - sleep 3600
      securityContext:
        privileged: false
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop:
            - ALL
      resources:
        requests:
          cpu: 10m
          memory: 16Mi
        limits:
          cpu: 50m
          memory: 32Mi
      volumeMounts:
        - name: tmp
          mountPath: /tmp
  volumes:
    - name: tmp
      emptyDir:
        sizeLimit: 10Mi

Apply it.

kubectl apply -f 05-authorised-test-pod.yaml

Wait for readiness.

kubectl wait \
  --for=condition=Ready \
  pod/authorised-runtime-test \
  -n runtime-security \
  --timeout=120s

Task 22 — Validate Approved Network Activity

Access the application Service.

kubectl exec authorised-runtime-test \
  -n runtime-security \
  -- wget -qO- http://monitored-web

Expected:

The NGINX page is returned.
The traffic is permitted by the Network Policy.

Review runtime or network telemetry.

Record:

Source Pod
Destination Service
Destination port
Result
Expected or unexpected classification

Task 23 — Deploy an Unauthorised Test Pod

Create 06-suspicious-test-pod.yaml.

apiVersion: v1
kind: Pod
metadata:
  name: suspicious-runtime-test
  namespace: runtime-security
  labels:
    runtime-access: denied
spec:
  automountServiceAccountToken: false
  restartPolicy: Never
  securityContext:
    runAsNonRoot: true
    runAsUser: 10002
    runAsGroup: 10002
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: tester
      image: busybox:1.36
      command:
        - sh
        - -c
        - sleep 3600
      securityContext:
        privileged: false
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop:
            - ALL
      resources:
        requests:
          cpu: 10m
          memory: 16Mi
        limits:
          cpu: 50m
          memory: 32Mi
      volumeMounts:
        - name: tmp
          mountPath: /tmp
  volumes:
    - name: tmp
      emptyDir:
        sizeLimit: 10Mi

Apply it.

kubectl apply -f 06-suspicious-test-pod.yaml

Wait for readiness.

kubectl wait \
  --for=condition=Ready \
  pod/suspicious-runtime-test \
  -n runtime-security \
  --timeout=120s

Task 24 — Simulate Unauthorised Network Access

Attempt to reach the monitored Service.

kubectl exec suspicious-runtime-test \
  -n runtime-security \
  -- wget -T 5 -qO- http://monitored-web

Expected:

The connection times out or is denied.
The Network Policy prevents unauthorised ingress.

Document whether the CNI, flow-log, or runtime platform records the denied connection.

Task 25 — Review DNS Activity

Resolve the application Service from the authorised Pod.

kubectl exec authorised-runtime-test \
  -n runtime-security \
  -- nslookup monitored-web

Review CoreDNS logs where logging is enabled.

kubectl logs \
  -n kube-system \
  -l k8s-app=kube-dns \
  --tail=100

Your environment may use a different CoreDNS label.

Review:

kubectl get pods -n kube-system --show-labels

Identify:

Expected internal DNS requests
Repeated failed resolutions
Unknown external domains
Service enumeration patterns

Task 26 — Review Kubernetes Events

Review namespace events.

kubectl get events \
  -n runtime-security \
  --sort-by=.metadata.creationTimestamp

Look for:

Pod creation
Container restarts
Probe failures
Admission denials
Scheduling problems
Image pull failures
Resource pressure
Volume errors

Kubernetes Events are operational evidence but should not be treated as a complete security log.

Task 27 — Review Application Logs

Review the monitored application logs.

kubectl logs "$POD_NAME" \
  -n runtime-security

Review logs from all replicas.

kubectl logs \
  -n runtime-security \
  -l app=monitored-web \
  --prefix=true

Look for:

Unusual request paths
Unexpected user agents
Repeated failures
Administrative endpoint access
High request volumes
Application errors

Task 28 — Correlate a Runtime Alert

Select one Falco alert generated during the simulations.

Record:

Alert Timestamp:

Falco Rule:

Priority:

Namespace:

Pod:

Container:

Node:

User:

Process:

Parent Process:

Command Line:

File or Network Target:

Expected Behaviour:

Security Interpretation:

Correlate the alert with Kubernetes metadata.

kubectl get pod <pod-name> \
  -n <namespace> \
  -o wide

kubectl get pod <pod-name> \
  -n <namespace> \
  --show-labels

kubectl describe pod <pod-name> \
  -n <namespace>

Determine:

Workload owner
Deployment controller
Image
Node
Service Account
Start time
Restart count
Relevant events

Task 29 — Create a Custom Falco Rule

Create 07-falco-custom-rules.yaml.

The following example rule detects executable activity from /tmp inside a container.

customRules:
  runtime-security-rules.yaml: |-
    - rule: Execute Program from Temporary Directory
      desc: Detect execution of a program located under /tmp inside a container
      condition: >
        spawned_process
        and container
        and proc.exepath startswith /tmp
      output: >
        Program executed from temporary directory
        (user=%user.name
        command=%proc.cmdline
        executable=%proc.exepath
        container=%container.name
        image=%container.image.repository
        pod=%k8s.pod.name
        namespace=%k8s.ns.name)
      priority: WARNING
      tags:
        - container
        - runtime
        - execution

Upgrade the Falco release using the custom rule.

helm upgrade falco falcosecurity/falco \
  --namespace falco \
  --reuse-values \
  -f 07-falco-custom-rules.yaml

Verify the rollout.

kubectl rollout status daemonset/falco \
  -n falco

Review logs for rule-loading errors.

kubectl logs \
  -n falco \
  -l app.kubernetes.io/name=falco \
  --tail=100

Task 30 — Test the Custom Rule

Recreate the harmless script.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- sh -c "printf '#!/bin/sh\necho custom-rule-test\n' > /tmp/custom-rule-test.sh"

Make it executable.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- chmod +x /tmp/custom-rule-test.sh

Execute it.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- /tmp/custom-rule-test.sh

Review Falco logs.

Expected custom alert:

Execute Program from Temporary Directory

Remove the script.

kubectl exec "$POD_NAME" \
  -n runtime-security \
  -- rm -f /tmp/custom-rule-test.sh

Task 31 — Classify Runtime Alerts

Use the following model.

Classification	Meaning
True Positive	Malicious or unauthorised activity correctly detected
Benign True Positive	Detected behaviour occurred but was authorised
False Positive	Alert triggered on behaviour that should not be considered suspicious
False Negative	Suspicious behaviour occurred but no alert was generated
Informational	Useful context without immediate security impact

Classify each lab event.

Event	Detection Result	Classification
Interactive shell
`/etc/shadow` access attempt
Root filesystem write attempt
Package manager execution
Service Account token discovery
Device node creation attempt
Execution from `/tmp`
Approved web connection
Blocked unauthorised connection

Task 32 — Assess Detection Coverage

Evaluate the environment.

Runtime Security Domain	Control Available	Validation Result
Process execution	Falco
Interactive shell detection	Falco
Sensitive file access	Falco and OS permissions
Filesystem modification	Read-only root filesystem and Falco
Package installation	Falco
Credential access	Token control and Falco
Privilege activity	Security Context and Falco
Network access	Network Policies and flow telemetry
DNS activity	CoreDNS and network telemetry
Container drift	Runtime monitoring
Kubernetes events	Kubernetes API
Application activity	Container logs
Alert centralisation	SIEM or log platform

Task 33 — Identify Detection Gaps

Review whether the environment can detect:

Reverse shell behaviour
Unexpected outbound connections
Cryptocurrency mining
Container escape attempts
Kernel module activity
New executable files
Sensitive file reads
Shell execution
Package installation
Kubernetes credential access
DNS tunnelling
Data exfiltration
Privileged Pod creation
Changes to Network Policies
Changes to security contexts

Document gaps such as:

Falco not integrated with SIEM
No long-term alert retention
No CNI flow visibility
CoreDNS logging disabled
No application-level audit logging
No automated containment
No 24×7 alert ownership
Incomplete custom-rule coverage

Task 34 — Simulate a Runtime Incident

Scenario:

A shell is opened inside the monitored application. Shortly afterward, the workload:

Reads /etc/passwd
Checks for package-management tools
Searches for Kubernetes tokens
Creates and executes a file under /tmp
Attempts to connect to an unauthorised Service

Using the evidence generated during the lab, build the incident timeline.

Time 1:
Interactive shell opened

Time 2:
System account file accessed

Time 3:
Package manager checked

Time 4:
Kubernetes credentials searched

Time 5:
Executable created under /tmp

Time 6:
Temporary executable launched

Time 7:
Unauthorised network connection attempted

Determine whether the activity represents:

Approved administration
Security testing
Suspicious behaviour
Confirmed compromise

Task 35 — Define Containment Actions

For a genuine runtime compromise, potential immediate actions include:

Isolate the affected namespace
Apply an emergency deny-all Network Policy
Scale the compromised Deployment to zero
Delete the affected Pod
Preserve logs and runtime evidence
Revoke affected credentials
Block the compromised image digest
Quarantine the node if host compromise is suspected
Notify the SOC and Incident Response team

Example emergency isolation policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: emergency-isolation
  namespace: runtime-security
spec:
  podSelector:
    matchLabels:
      app: monitored-web
  policyTypes:
    - Ingress
    - Egress

Do not apply containment actions to production without following the approved incident response process.

Task 36 — Preserve Runtime Evidence

Collect evidence before deleting or restarting the affected workload.

Recommended commands:

kubectl get pod "$POD_NAME" \
  -n runtime-security \
  -o yaml

kubectl describe pod "$POD_NAME" \
  -n runtime-security

kubectl logs "$POD_NAME" \
  -n runtime-security \
  --timestamps

kubectl get events \
  -n runtime-security \
  --sort-by=.metadata.creationTimestamp

kubectl get networkpolicy \
  -n runtime-security \
  -o yaml

kubectl logs \
  -n falco \
  <falco-pod-name> \
  --timestamps

Record the image reference.

kubectl get pod "$POD_NAME" \
  -n runtime-security \
  -o jsonpath='{.spec.containers[0].image}{"\n"}'

Record the image ID.

kubectl get pod "$POD_NAME" \
  -n runtime-security \
  -o jsonpath='{.status.containerStatuses[0].imageID}{"\n"}'

Task 37 — Perform the Enterprise Runtime Security Assessment

Complete the assessment.

Control Domain	Expected Control	Status
Runtime monitoring	Falco or approved equivalent deployed
Process visibility	Container process execution visible
Shell detection	Interactive shell detected
Sensitive file monitoring	Access attempts observable
Filesystem integrity	Root filesystem protected
Container drift	Temporary executable detected
Privilege monitoring	Capability and escalation attempts reviewed
Credential protection	Service Account token absent
Network isolation	Unauthorised access blocked
DNS visibility	DNS activity review available
Kubernetes event visibility	Events collected
Application logging	Workload logs available
Alert enrichment	Pod and namespace metadata included
Alert centralisation	Alerts forwarded to central platform
Evidence preservation	Procedures documented
Incident ownership	SOC and IR roles defined

Use one of the following classifications:

Effective
Partially Effective
Ineffective

Task 38 — Produce the Runtime Security Report

Create runtime-security-report.md.

Assessment Title:
Kubernetes Runtime Security Validation

Cluster Name:

Namespace:
runtime-security

Assessment Date:

Assessor:

Runtime Security Tool:
Falco

Falco Version:

Monitored Workload:
monitored-web

Container Image:

Baseline Processes:

Expected Network Behaviour:

Runtime Tests Performed:

Interactive Shell Detection:

Sensitive File Access Detection:

Filesystem Modification Validation:

Package Manager Detection:

Credential Discovery Validation:

Privilege Activity Validation:

Container Drift Detection:

Network Policy Validation:

DNS Review:

Kubernetes Event Review:

Application Log Review:

Custom Rule Validation:

True Positive Alerts:

Benign True Positives:

False Positives:

False Negatives:

Detection Gaps:

Containment Recommendations:

Evidence Preserved:

Overall Runtime Security Rating:

Production Recommendation:

Approved

Conditionally Approved

Rejected

Task 39 — Evidence Collection

Collect evidence for:

Cluster and node health
Runtime namespace labels
Hardened Deployment manifest
Service configuration
Network Policies
Workload process baseline
Runtime user and capabilities
Falco installation status
Falco health logs
Interactive shell alert
Sensitive file alert
Root filesystem write denial
Package-manager alert
Credential-discovery result
Temporary executable alert
Custom Falco rule
Authorised network test
Unauthorised network test
CoreDNS review
Kubernetes Events
Application logs
Alert correlation worksheet
Detection coverage assessment
Incident timeline
Final runtime security report

Suggested filenames:

01-cluster-health.txt
02-runtime-namespace-labels.txt
03-monitored-deployment.yaml
04-runtime-network-policies.yaml
05-process-baseline.txt
06-runtime-identity.txt
07-falco-status.txt
08-falco-health-logs.txt
09-shell-alert.txt
10-sensitive-file-alert.txt
11-readonly-validation.txt
12-package-manager-alert.txt
13-credential-discovery.txt
14-container-drift-alert.txt
15-custom-falco-rule.yaml
16-authorised-network-test.txt
17-blocked-network-test.txt
18-dns-review.txt
19-kubernetes-events.txt
20-application-logs.txt
21-alert-correlation.md
22-runtime-security-report.md

Task 40 — Clean Up

Delete the test Pods.

kubectl delete pod authorised-runtime-test \
  suspicious-runtime-test \
  -n runtime-security \
  --ignore-not-found

Delete the monitored application.

kubectl delete deployment monitored-web \
  -n runtime-security

Delete the Service.

kubectl delete service monitored-web \
  -n runtime-security

Delete the Network Policies.

kubectl delete networkpolicy --all \
  -n runtime-security

Delete the runtime namespace.

kubectl delete namespace runtime-security

If Falco was installed only for this lab, uninstall it.

helm uninstall falco \
  --namespace falco

Delete the Falco namespace.

kubectl delete namespace falco

Verify cleanup.

kubectl get namespace runtime-security falco

Expected:

NotFound

Enterprise Runtime Security Checklist

Control	Status
Runtime monitoring platform deployed	☐
Runtime monitoring health validated	☐
Workload process baseline documented	☐
Interactive shell activity detectable	☐
Sensitive file access detectable	☐
Package manager execution detectable	☐
Execution from temporary paths detectable	☐
Container drift monitored	☐
Privilege-related activity monitored	☐
Kubernetes credential access monitored	☐
Root filesystem protected	☐
Runtime user is non-root	☐
Linux capabilities removed	☐
Seccomp enabled	☐
Network isolation enforced	☐
DNS activity review available	☐
Kubernetes Events collected	☐
Application logs available	☐
Alerts include Kubernetes metadata	☐
Custom rules tested	☐
Alerts centralised in SIEM	☐
Evidence preservation procedure documented	☐
Containment workflow documented	☐
Incident ownership assigned	☐

Risk Classification

Critical

Examples:

Confirmed container escape
Runtime activity indicating node compromise
Execution of known malware
Unrestricted access to Kubernetes credentials
Data exfiltration from production workloads
Privileged workload performing unexpected host operations

High

Examples:

Reverse shell execution
Unknown binary executed from /tmp
Package installation in production
Unexpected root shell
Sensitive credential file access
Unauthorised connection to production databases
Runtime security monitoring disabled

Medium

Examples:

Administrative shell without a change record
Excessive DNS lookup activity
Incomplete runtime alert enrichment
Repeated blocked filesystem writes
Missing central alert retention
Broad but unsuccessful network scanning

Low

Examples:

Approved troubleshooting activity
Missing ownership labels
Rule tuning requirements
Documentation gaps
Non-security operational process execution

Recommended Runtime Security Improvements

Immediate

Investigate high-priority runtime alerts.
Isolate confirmed compromised workloads.
Revoke exposed credentials.
Block suspicious images and digests.
Preserve runtime evidence before remediation.
Escalate suspected node compromise immediately.

Short-Term

Forward Falco alerts to the SIEM.
Enable CNI flow logs.
Enable appropriate CoreDNS logging.
Tune runtime rules for production workloads.
Create application-specific process baselines.
Add alerts for executable activity from writable directories.

Long-Term

Implement automated runtime containment with approval controls.
Integrate runtime security findings with incident-management platforms.
Deploy eBPF-based workload visibility where appropriate.
Continuously test runtime detection coverage.
Maintain workload-specific detection-as-code repositories.
Correlate runtime events with cloud, identity, application, and audit telemetry.
Perform recurring Purple Team validation.

Skills Developed

By completing this lab, you will be able to:

Establish Kubernetes workload behaviour baselines
Deploy and validate Falco
Monitor runtime process execution
Detect interactive container shells
Identify sensitive file-access attempts
Detect package-management activity
Detect execution from temporary directories
Analyse container drift
Validate runtime privilege controls
Review Kubernetes and application logs
Analyse DNS and network behaviour
Correlate alerts with Kubernetes metadata
Classify runtime alerts
Identify detection gaps
Preserve runtime evidence
Recommend containment actions
Produce enterprise runtime security reports

Knowledge Check

Question 1

What is the primary purpose of Kubernetes runtime security?

A. Create additional worker nodes
B. Detect and investigate suspicious behaviour after workloads begin running
C. Replace container image scanning
D. Configure Kubernetes Services

Answer: B

Question 2

Which activity is commonly suspicious inside a production web container?

A. NGINX serving HTTP requests
B. A new executable launched from /tmp
C. A readiness probe accessing /
D. A worker process reading approved configuration

Answer: B

Question 3

What does Falco primarily analyse?

A. Only container image vulnerabilities
B. Runtime events and system behaviour evaluated against security rules
C. Kubernetes billing data
D. DNS zone configuration only

Answer: B

Question 4

Why should interactive shell execution inside a production container generate an alert?

A. Shell access may indicate troubleshooting, misuse, or attacker activity and should be investigated.
B. Every container requires an interactive shell.
C. Shells automatically increase memory limits.
D. Shell activity creates Network Policies.

Answer: A

Question 5

What is container drift?

A. A Pod moving to another node
B. Runtime changes that cause a container to differ from its approved image or expected behaviour
C. A Service changing its ClusterIP
D. A Deployment increasing its replica count

Answer: B

Question 6

Which control helps prevent modification of system files inside a container?

A. readOnlyRootFilesystem: true
B. hostNetwork: true
C. privileged: true
D. runAsUser: 0

Answer: A

Question 7

What is a benign true positive?

A. Malicious behaviour that was not detected
B. An alert that correctly detected activity, but the activity was authorised
C. An alert that contains no timestamp
D. A failed container image pull

Answer: B

Question 8

Why should runtime alerts include Kubernetes metadata?

A. To increase Pod memory
B. To identify the namespace, Pod, container, image, node, and workload owner involved
C. To replace application logs
D. To create persistent storage

Answer: B

Question 9

Which action should occur before deleting a suspected compromised Pod?

A. Remove all Network Policies
B. Preserve relevant logs, Pod metadata, events, runtime alerts, and image identifiers
C. Expose the Pod through a LoadBalancer
D. Increase its CPU limit

Answer: B

Question 10

Why are preventive and detective controls both required?

A. Preventive controls reduce risk, while detective controls identify suspicious activity that still occurs.
B. They perform exactly the same function.
C. Detective controls create Kubernetes namespaces.
D. Preventive controls eliminate every possible vulnerability.

Answer: A

Lab Summary

In this lab, you validated the runtime security posture of a hardened Kubernetes workload.

You implemented and reviewed:

Restricted Pod Security enforcement
Non-root execution
Dropped Linux capabilities
RuntimeDefault seccomp
Read-only root filesystem protection
Service Account token protection
Default-deny Network Policies
Process and filesystem baselining
Falco runtime monitoring
Interactive shell detection
Sensitive file-access monitoring
Package-manager detection
Container drift detection
Execution from temporary directories
Network and DNS validation
Kubernetes Event review
Alert correlation
Evidence preservation
Runtime incident assessment

You also created and tested a custom Falco rule to detect executable activity from /tmp.

Runtime security provides visibility into workload behaviour after deployment. It complements secure images, admission control, security contexts, Network Policies, RBAC, and vulnerability management by detecting activity that preventive controls cannot completely eliminate.

A mature Kubernetes security programme combines prevention, detection, investigation, containment, and continuous improvement.

What’s Next?

Next Runbook: Runbook 01 — Kubernetes Workload Security Assessment

In the next runbook, you will perform a structured enterprise assessment of Kubernetes workload security by reviewing Pod Security Standards, security contexts, privileged workloads, runtime controls, resource governance, image configuration, Service Accounts, volumes, health checks, and production readiness.