Lesson 05 — Kubernetes Secrets

Learning Objectives

By the end of this lesson, you will be able to:

Understand what Kubernetes Secrets are
Learn why Secrets are critical for Kubernetes security
Differentiate Secrets from ConfigMaps
Understand how Secrets are stored and accessed
Learn different Secret types
Identify common Secret security risks
Explore enterprise secret management
Apply security best practices for protecting sensitive information

Why This Matters

Every enterprise application requires sensitive information to function.

Examples include:

Database passwords
API keys
OAuth tokens
TLS certificates
SSH private keys
Cloud credentials
Encryption keys

If attackers obtain these secrets, they may gain complete access to your infrastructure.

Hardcoding credentials into applications is one of the most common cloud security mistakes.

Kubernetes provides Secrets to securely manage sensitive information separately from application code.

What is a Kubernetes Secret?

A Kubernetes Secret is an object that stores confidential information used by applications.

Unlike ConfigMaps, Secrets are specifically designed to hold sensitive data.

Examples include:

Database credentials
TLS certificates
Docker registry credentials
OAuth tokens
API keys
AWS credentials
Encryption keys

Applications retrieve Secrets securely at runtime instead of embedding them directly into container images.

Secrets Architecture

Application

↓

Pod

↓

Kubernetes Secret

↓

Sensitive Data

↓

Database / AWS Service / API

Secrets provide applications with the information they need while keeping sensitive values separate from application code.

Why Not Store Secrets in Code?

Bad example:

Application

↓

Database Password

↓

Source Code

↓

Git Repository

Problems include:

Credentials exposed in Git
Difficult credential rotation
Shared secrets across environments
High risk of compromise

Instead:

Application

↓

Kubernetes Secret

↓

Password

↓

Database

This approach is significantly more secure.

Secrets vs ConfigMaps

ConfigMap	Secret
Non-sensitive configuration	Sensitive information
Environment variables	Passwords
Application settings	API Keys
Feature flags	Certificates
Logging configuration	Tokens

Rule of thumb:

If exposure could create a security incident, store it as a Secret.

Common Secret Types

Kubernetes supports multiple Secret types.

Secret Type	Purpose
Opaque	Generic application secrets
kubernetes.io/tls	TLS certificates
kubernetes.io/dockerconfigjson	Container registry credentials
kubernetes.io/basic-auth	Username and password
kubernetes.io/ssh-auth	SSH private keys
kubernetes.io/service-account-token	Service Account tokens

Each Secret type supports a specific authentication or configuration scenario.

Secret Storage

Secrets are stored inside etcd, Kubernetes’ distributed key-value database.

Application

↓

API Server

↓

etcd

↓

Secret

Because etcd contains sensitive information, it must be properly secured.

Base64 is NOT Encryption

Many beginners believe Kubernetes encrypts Secrets because the values appear unreadable.

Example:

Password123

↓

UGFzc3dvcmQxMjM=

This is Base64 encoding, not encryption.

Base64 simply converts data into another format.

Anyone can decode it.

Proper protection requires encryption.

Encryption at Rest

Enterprise Kubernetes clusters encrypt Secrets before storing them in etcd.

Secret

↓

AWS KMS

↓

Encrypted Secret

↓

etcd

Even if etcd is compromised, encrypted Secrets remain protected.

Amazon EKS supports encryption using AWS Key Management Service (AWS KMS).

How Applications Consume Secrets

Applications can access Secrets in several ways.

Environment Variables

Pod

↓

Secret

↓

Environment Variable

↓

Application

Mounted Volume

Pod

↓

Secret Volume

↓

Application Reads File

Mounted files are often preferred for certificates and cryptographic material.

Enterprise Secret Lifecycle

Secrets should follow a managed lifecycle.

Create

↓

Store

↓

Use

↓

Rotate

↓

Expire

↓

Delete

Security teams should never create secrets that remain unchanged indefinitely.

Secret Rotation

Enterprise organisations regularly rotate:

Database passwords
API tokens
TLS certificates
OAuth secrets
Cloud credentials

Example:

Old Secret

↓

Rotate

↓

New Secret

↓

Application Continues Running

Frequent rotation limits the impact of credential compromise.

External Secret Management

Large organisations rarely store production credentials directly inside Kubernetes.

Instead they use dedicated secret management platforms.

Common solutions include:

AWS Secrets Manager
AWS Systems Manager Parameter Store
HashiCorp Vault
Azure Key Vault
Google Secret Manager

These systems provide:

Centralised secret storage
Automatic rotation
Fine-grained access control
Auditing
Versioning
High availability

Kubernetes External Secrets Architecture

Application

↓

Kubernetes Secret

↓

External Secrets Operator

↓

AWS Secrets Manager

↓

AWS KMS

Applications continue using Kubernetes Secrets while the actual secret values are securely managed outside the cluster.

Secret Access Control

Secrets should only be accessible to authorised workloads.

Payment Service

↓

Payment Secret

✓ Allowed

-------------------------

Inventory Service

↓

Payment Secret

✗ Denied

RBAC controls which users and Service Accounts can access specific Secrets.

Enterprise Example

A financial services company deploys a payment application.

The application requires:

Database password
Payment gateway API key
TLS certificate
AWS KMS key

Architecture:

Payment Application

↓

Service Account

↓

RBAC

↓

External Secrets Operator

↓

AWS Secrets Manager

↓

AWS KMS

Only the payment application can retrieve these credentials.

Common Secret Security Risks

Cloud Security Engineers frequently discover:

Passwords stored in ConfigMaps
Secrets committed to Git repositories
Hardcoded credentials
Shared Secrets across applications
Default Service Accounts with Secret access
Unencrypted etcd
Long-lived credentials
Excessive RBAC permissions
Secrets exposed through CI/CD logs
Unused Secrets remaining in clusters

These issues often lead to credential theft and privilege escalation.

Enterprise Monitoring

Security teams should monitor:

Secret creation
Secret deletion
Secret modifications
Failed Secret access
RBAC changes
Secret rotation events
Access to AWS Secrets Manager
Unusual API activity
Certificate expiration
Secret age

Monitoring helps detect suspicious behaviour before credentials are abused.

Real-World Scenario

A developer accidentally commits a Kubernetes Secret containing production database credentials to a public Git repository.

Within hours, automated scanners identify the exposed credentials.

An attacker:

Connects to the production database
Downloads customer records
Creates administrator accounts
Deletes audit logs

If the organisation had:

Stored credentials in AWS Secrets Manager
Used External Secrets Operator
Rotated credentials regularly
Enabled Git secret scanning

the exposure would have been significantly less severe.

Best Practices

As a Kubernetes Security Engineer:

Never store credentials inside source code.
Store sensitive information only in Kubernetes Secrets.
Enable Encryption at Rest using AWS KMS.
Use external secret management for production workloads.
Rotate secrets regularly.
Restrict Secret access using RBAC.
Monitor Secret usage.
Remove unused Secrets.
Audit Secret access.
Protect etcd backups.

Secrets should always be treated as high-value assets.

Key Takeaways

After completing this lesson, you should understand:

What Kubernetes Secrets are
Why Secrets are required
How Secrets differ from ConfigMaps
How Secrets are stored in etcd
Why Base64 is not encryption
How Encryption at Rest protects Secrets
Enterprise secret management strategies
Common security risks
Best practices for securing Secrets

Protecting Secrets is one of the most important responsibilities of a Kubernetes Security Engineer because compromised credentials often lead to full cluster compromise.

Knowledge Check

Question 1

What is the primary purpose of a Kubernetes Secret?

A. Store application logs
B. Store sensitive information securely
C. Schedule Pods
D. Create Deployments

Answer: B

Question 2

Where are Kubernetes Secrets stored?

A. Amazon S3
B. etcd
C. Route 53
D. kubelet

Answer: B

Question 3

Why is Base64 encoding insufficient for protecting Secrets?

A. It increases storage usage.
B. It is an encoding mechanism, not encryption.
C. Kubernetes cannot decode it.
D. It only works on Linux.

Answer: B

Question 4

Which AWS service is commonly used to encrypt Kubernetes Secrets in Amazon EKS?

A. Amazon CloudWatch
B. AWS KMS
C. Amazon SNS
D. AWS Lambda

Answer: B

Question 5

Which of the following is considered an enterprise best practice?

A. Store passwords inside ConfigMaps.
B. Commit Secrets to Git repositories.
C. Use AWS Secrets Manager with regular secret rotation.
D. Share one Secret across every application.

Answer: C

What’s Next?

In the next lesson, you will learn about Identity Federation, exploring how Amazon EKS integrates with enterprise identity providers using OpenID Connect (OIDC), AWS IAM Identity Center, Microsoft Entra ID, and Okta to provide secure, centralized authentication for Kubernetes users.

➡️ Next Lesson: Lesson 06 — Identity Federation