Lesson 04 — Prometheus

Learning Objectives

By the end of this lesson, you will be able to:

Understand what Prometheus is
Learn how Prometheus collects metrics
Understand Kubernetes metrics and exporters
Explore Prometheus architecture
Learn Prometheus Alertmanager
Integrate Prometheus with Amazon EKS
Apply enterprise monitoring best practices

Why This Matters

Logging tells us what happened.

Metrics tell us how the system is performing.

For example:

Is CPU usage increasing?
Is memory almost exhausted?
Are Pods restarting frequently?
Are API requests failing?
Is latency increasing?
Is the Kubernetes cluster healthy?

Without metrics, organizations cannot proactively detect performance issues before they become outages.

Prometheus is the most widely adopted monitoring platform in the Kubernetes ecosystem.

What is Prometheus?

Prometheus is an open-source monitoring and alerting system originally developed by SoundCloud and now maintained by the Cloud Native Computing Foundation (CNCF).

Prometheus continuously collects metrics from:

Kubernetes clusters
Applications
Containers
Nodes
Cloud services
Databases
Operating systems

These metrics provide real-time visibility into system health.

What are Metrics?

Metrics are numerical measurements collected over time.

Examples include:

CPU utilization
Memory usage
Disk utilization
Network traffic
HTTP request rate
API response time
Pod restarts
Node availability

Unlike logs, metrics are optimized for trend analysis and alerting.

Metrics vs Logs

Metrics	Logs
Numerical values	Text records
Time-series data	Event records
Performance monitoring	Troubleshooting
Alerting	Investigation
Trend analysis	Root cause analysis

Enterprise monitoring platforms use both metrics and logs together.

Prometheus Architecture

Applications

↓

Exporters

↓

Prometheus Server

↓

Time-Series Database

↓

Alertmanager

↓

Grafana

↓

Operations Team

Prometheus periodically collects metrics from configured targets.

Pull-Based Monitoring

Unlike many monitoring systems, Prometheus uses a pull model.

Prometheus

↓

Requests Metrics

↓

Application

↓

Metrics Returned

↓

Stored

Prometheus scrapes metrics at regular intervals.

Prometheus Components

A standard deployment includes:

Component	Purpose
Prometheus Server	Collects and stores metrics
Exporters	Expose application metrics
Alertmanager	Sends alerts
Grafana	Visualizes metrics
PromQL	Query language for metrics

Each component plays an important role in enterprise observability.

Exporters

Applications expose metrics using exporters.

Common exporters include:

Node Exporter
kube-state-metrics
cAdvisor
Blackbox Exporter
MySQL Exporter
PostgreSQL Exporter
NGINX Exporter
Redis Exporter

Exporters convert application data into a format Prometheus understands.

Node Exporter

Node Exporter collects operating system metrics.

Examples include:

CPU usage
Memory usage
Disk utilization
Filesystem usage
Network interfaces
Load average

These metrics help monitor worker node health.

kube-state-metrics

kube-state-metrics provides Kubernetes object information.

Examples include:

Deployments
Pods
ReplicaSets
DaemonSets
StatefulSets
Nodes
Persistent Volumes
Namespaces

It enables visibility into Kubernetes resource status.

cAdvisor

cAdvisor collects container-level metrics.

Examples include:

Container CPU usage
Container memory usage
Network statistics
Filesystem utilization
Container lifecycle events

Most Kubernetes environments use cAdvisor for container monitoring.

Prometheus Data Flow

Application

↓

Metrics Endpoint

↓

Prometheus

↓

Time-Series Database

↓

Alertmanager

↓

Grafana Dashboard

Metrics are collected continuously and stored for historical analysis.

Prometheus Time-Series Database

Prometheus stores metrics as time-series data.

Example:

CPU Usage

↓

08:00 → 25%

↓

09:00 → 42%

↓

10:00 → 67%

↓

11:00 → 81%

This enables trend analysis and capacity planning.

PromQL

PromQL (Prometheus Query Language) is used to retrieve and analyse metrics.

Examples include:

CPU utilization
Memory consumption
Request rate
Error rate
Pod restart count

PromQL powers dashboards, reports and alerts.

Alertmanager

Alertmanager manages notifications generated by Prometheus.

Prometheus

↓

Alert Rule

↓

Alertmanager

↓

Email

↓

Slack

↓

Microsoft Teams

↓

PagerDuty

↓

SOC Team

Alertmanager prevents alert storms by grouping and suppressing duplicate alerts.

Common Kubernetes Metrics

Security and Operations teams commonly monitor:

CPU utilization
Memory usage
Disk usage
Pod restart count
Node readiness
API Server latency
Network traffic
Container restarts
Failed Pods
Failed Deployments

These metrics provide early warning of operational issues.

Prometheus on Amazon EKS

Prometheus is commonly deployed using:

Helm Charts
Amazon Managed Service for Prometheus (AMP)
Kubernetes Operators
kube-prometheus-stack

Amazon Managed Service for Prometheus simplifies large-scale metric collection and management.

Amazon EKS Monitoring Architecture

Amazon EKS

↓

Applications

↓

Exporters

↓

Prometheus

↓

Amazon Managed Service for Prometheus

↓

Grafana

↓

AWS CloudWatch

↓

Security Operations Centre (SOC)

This architecture provides scalable enterprise monitoring.

Enterprise Example

A multinational e-commerce company operates more than 400 Amazon EKS clusters across multiple AWS Regions.

Prometheus continuously monitors:

Node health
Application latency
Pod restart rates
CPU utilization
Memory consumption
Kubernetes API availability

One afternoon, Prometheus detects rapidly increasing memory utilization on several worker nodes.

Alertmanager immediately sends notifications to the operations team.

Engineers identify a memory leak in a recently deployed application and roll back the deployment before customer-facing services experience downtime.

Because Prometheus detected the issue early, the incident is resolved without impacting online sales.

Common Monitoring Challenges

Cloud Security Engineers frequently encounter:

Missing exporters
Poor metric retention
Excessive metric collection
Alert fatigue
Missing alert thresholds
Inconsistent dashboards
No high availability
Large Prometheus servers
Missing service discovery
Lack of metric security

These challenges reduce monitoring effectiveness.

Enterprise Monitoring

Operations teams should continuously monitor:

Cluster health
Node health
Pod health
CPU utilization
Memory utilization
Storage capacity
API Server availability
Network performance
Container restart rates
Kubernetes component availability

Continuous monitoring enables proactive issue detection.

Enterprise Implementation Strategy

A recommended rollout:

Step 1

↓

Deploy Prometheus

↓

Step 2

↓

Deploy Exporters

↓

Step 3

↓

Configure Service Discovery

↓

Step 4

↓

Create Alert Rules

↓

Step 5

↓

Deploy Alertmanager

↓

Step 6

↓

Integrate Grafana

↓

Step 7

↓

Configure Dashboards

↓

Step 8

↓

Continuously Tune Alerts

This phased approach provides scalable and reliable monitoring.

Enterprise Prometheus Architecture

Amazon EKS

↓

Applications

↓

Node Exporter

↓

kube-state-metrics

↓

cAdvisor

↓

Prometheus Server

↓

Amazon Managed Service for Prometheus

↓

Alertmanager

↓

Amazon Managed Grafana

↓

AWS CloudWatch

↓

Enterprise SIEM

↓

Security Operations Centre (SOC)

This architecture combines metrics, visualization and alerting into a centralized observability platform.

Best Practices

As a Kubernetes Security Engineer:

Deploy Prometheus across every production cluster.
Monitor both infrastructure and application metrics.
Use exporters appropriate for your workloads.
Configure meaningful alert thresholds.
Avoid excessive metric collection.
Secure Prometheus using RBAC and network policies.
Integrate Alertmanager with enterprise notification platforms.
Use Amazon Managed Service for Prometheus for large-scale deployments.
Regularly review dashboards and alert rules.
Combine Prometheus metrics with logs and runtime security events.

Prometheus is most effective when integrated into a complete observability platform.

Real-World Scenario

A global financial institution hosts payment processing services on Amazon EKS.

Prometheus detects:

Increasing API latency
Rising CPU utilization
High Pod restart rates
Declining node availability

Alertmanager immediately notifies the Site Reliability Engineering (SRE) team.

Engineers discover a recently deployed application consuming excessive resources.

The deployment is rolled back before customers experience payment failures.

Prometheus provides the early warning necessary to prevent a production outage.

Key Takeaways

After completing this lesson, you should understand:

What Prometheus is
How Prometheus collects metrics
Exporters and service discovery
Prometheus architecture
Time-series databases
PromQL
Alertmanager
Amazon Managed Service for Prometheus
Enterprise monitoring best practices

Prometheus is the industry-standard monitoring platform for Kubernetes. By continuously collecting metrics from applications, containers and infrastructure, it enables organizations to proactively detect performance issues, improve reliability and support secure, observable Amazon EKS environments.

Knowledge Check

Question 1

What is the primary purpose of Prometheus?

A. Store container images
B. Collect, store and analyse time-series metrics
C. Replace Kubernetes Audit Logs
D. Encrypt Kubernetes Secrets

Answer: B

Question 2

Which monitoring model does Prometheus primarily use?

A. Push model
B. Pull model
C. Peer-to-peer model
D. Manual collection

Answer: B

Question 3

Which component is responsible for sending notifications based on Prometheus alerts?

A. kube-state-metrics
B. Alertmanager
C. Node Exporter
D. cAdvisor

Answer: B

Question 4

Which exporter provides Kubernetes object information such as Pods and Deployments?

A. Node Exporter
B. kube-state-metrics
C. Blackbox Exporter
D. Redis Exporter

Answer: B

Question 5

Which combination represents enterprise best practice?

A. Deploy Prometheus with exporters, configure Alertmanager, integrate with Grafana, secure access using RBAC, tune alert thresholds and combine metrics with logs and runtime security events.
B. Monitor only CPU utilization.
C. Disable alerts to reduce notifications.
D. Collect every available metric without reviewing storage requirements.

Answer: A

What’s Next?

In the next lesson, you will learn about Grafana, exploring how enterprise organizations build dashboards, visualize Kubernetes metrics, correlate logs and metrics, and create operational and security dashboards for Amazon EKS.

➡️ Next Lesson: Lesson 05 — Grafana