Lesson 06 — Amazon EKS Secrets Management

Learning Objectives

By the end of this lesson, you will be able to:

Explain how Kubernetes Secrets work
Identify the risks associated with poorly managed secrets
Understand the difference between encoding and encryption
Protect Kubernetes Secrets using AWS KMS
Compare Kubernetes Secrets, AWS Secrets Manager and Parameter Store
Use workload identity to retrieve secrets securely
Understand the Secrets Store CSI Driver and AWS Secrets and Configuration Provider
Mount external secrets into Amazon EKS Pods
Design secret-rotation strategies
Restrict secret access using IAM and Kubernetes RBAC
Monitor secret access and configuration changes
Build an enterprise secrets-management architecture for Amazon EKS

Why This Matters

Applications require sensitive information to operate.

Examples include:

Database passwords
API keys
OAuth client secrets
TLS certificates
Private keys
Service credentials
Encryption keys
Third-party access tokens

If these secrets are exposed, attackers may gain access to:

Databases
AWS services
Internal applications
Customer information
Payment systems
External platforms
Encryption infrastructure

Exposed Secret

↓

Unauthorised Access

↓

Privilege Escalation

↓

Data Theft or Service Compromise

Secrets management is therefore not simply the process of storing passwords.

It includes:

Secure creation
Controlled distribution
Encryption
Access governance
Rotation
Monitoring
Revocation
Secure deletion

What is a Secret?

A secret is sensitive information used by a user, application or system to authenticate, authorise or establish trust.

Examples include:

Secret Type	Example
Database credential	Username and password
API credential	API key or token
Cloud credential	Temporary AWS credential
Encryption material	Private key
Certificate	TLS certificate and private key
Application secret	OAuth client secret
Integration credential	Third-party service token

Secrets should be treated differently from ordinary application configuration.

Secrets Versus Configuration

Configuration	Secret
Application port	Database password
Feature flag	API key
Log level	Private key
Service URL	Authentication token
Timeout setting	OAuth client secret

Configuration may often be stored in a ConfigMap.

Sensitive information should not be stored in a ConfigMap.

Amazon EKS Secrets Architecture

Application Pod

↓

Kubernetes Service Account

↓

Workload Identity

↓

Secrets Management Service

↓

Authorised Secret

↓

Application

A secure architecture ensures that:

The workload receives only the secrets it requires.
The secret is not stored in the container image.
The secret is not committed to Git.
Access is logged.
The secret can be rotated.
Access can be revoked.

Secret Lifecycle

Create

↓

Store

↓

Authorise

↓

Retrieve

↓

Use

↓

Rotate

↓

Revoke

↓

Delete

Security controls should exist at every stage.

Common Secret Locations

Secrets may appear in:

Kubernetes Secret objects
Environment variables
Mounted files
Application configuration files
CI/CD systems
Container images
Git repositories
Helm values
Terraform state
CloudFormation parameters
Application logs
Developer workstations

Each location introduces different risks.

Kubernetes Secrets

Kubernetes provides the Secret resource for storing sensitive data.

Example:

apiVersion: v1
kind: Secret
metadata:
  name: payment-database
  namespace: payments

type: Opaque

data:
  username: cGF5bWVudC11c2Vy
  password: RXhhbXBsZVBhc3N3b3Jk

The values shown under data are Base64 encoded.

Base64 is Not Encryption

Base64 converts binary data into a text representation.

It does not protect confidentiality.

Plaintext Secret

↓

Base64 Encoding

↓

Encoded Secret

Not

↓

Encrypted Secret

Anyone who can read the Secret object can decode it.

Example:

echo "RXhhbXBsZVBhc3N3b3Jk" | base64 --decode

Creating a Kubernetes Secret

A generic Secret can be created using kubectl.

kubectl create secret generic payment-database \
  --namespace payments \
  --from-literal=username=payment-user \
  --from-literal=password='ExamplePassword'

This method may expose sensitive values through:

Terminal history
Process inspection
Shell logs
Screen recording
Automation logs

For production, use approved secret-management workflows rather than entering sensitive values directly into command lines.

Create a Secret from Files

kubectl create secret generic payment-certificate \
  --namespace payments \
  --from-file=tls.crt=./tls.crt \
  --from-file=tls.key=./tls.key

Local files must be protected and securely deleted when no longer required.

Kubernetes Secret Types

Common Secret types include:

Secret Type	Purpose
`Opaque`	General sensitive information
`kubernetes.io/tls`	TLS certificate and private key
`kubernetes.io/dockerconfigjson`	Private registry credentials
`kubernetes.io/service-account-token`	Service Account token
`bootstrap.kubernetes.io/token`	Cluster bootstrap data

Use the appropriate Secret type for the intended purpose.

Consuming Secrets as Environment Variables

A Pod can reference a Secret through environment variables.

apiVersion: v1
kind: Pod
metadata:
  name: payment-api
  namespace: payments

spec:
  containers:
    - name: payment-api
      image: example/payment-api:1.0

      env:
        - name: DATABASE_USERNAME

          valueFrom:
            secretKeyRef:
              name: payment-database
              key: username

        - name: DATABASE_PASSWORD

          valueFrom:
            secretKeyRef:
              name: payment-database
              key: password

Risks of Environment Variables

Environment variables may be exposed through:

Process inspection
Application debug endpoints
Crash reports
Diagnostic tools
Accidental logging
Pod specifications
Child processes

For highly sensitive secrets, mounted files or direct API retrieval may provide better control.

AWS recommends considering file-based secret mounting to reduce the risk of secrets leaking through environment variables. :contentReference[oaicite:0]{index="0"}

Consuming Secrets as Files

apiVersion: v1
kind: Pod
metadata:
  name: payment-api
  namespace: payments

spec:
  containers:
    - name: payment-api
      image: example/payment-api:1.0

      volumeMounts:
        - name: database-secret
          mountPath: /var/run/secrets/database
          readOnly: true

  volumes:
    - name: database-secret

      secret:
        secretName: payment-database

The Pod receives files such as:

/var/run/secrets/database/username

/var/run/secrets/database/password

Secret Volume Permissions

File permissions should be restricted.

volumes:
  - name: database-secret

    secret:
      secretName: payment-database
      defaultMode: 0400

Applications should run using an identity that can read only the required files.

Kubernetes Secret Storage

Kubernetes stores Secret objects in the cluster data store.

In Amazon EKS, AWS manages the Kubernetes control plane and its backing data store.

The customer must still control:

Who can create Secrets
Who can read Secrets
Which Pods can mount Secrets
How Secrets are encrypted
How Secrets are rotated
How access is audited

Envelope Encryption

Envelope encryption protects sensitive Kubernetes API data using a data-encryption key that is protected by a key-encryption key.

Kubernetes Secret

↓

Data Encryption Key

↓

AWS KMS Key

↓

Encrypted Storage

AWS KMS provides central key management and auditing.

AWS KMS

AWS Key Management Service provides managed cryptographic keys.

KMS supports:

Customer-managed keys
Key policies
IAM integration
CloudTrail logging
Key rotation
Grants
Central governance

KMS Key Responsibilities

The organisation should define:

Key owner
Key administrators
Key users
Rotation requirements
Deletion controls
Recovery procedures
Logging and monitoring
Separation of duties

KMS Key Policy

A key policy determines who can administer and use a KMS key.

Avoid policies that grant broad access.

Conceptual example:

{
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::123456789012:role/eks-security-admin"
  },
  "Action": [
    "kms:DescribeKey",
    "kms:CreateGrant"
  ],
  "Resource": "*"
}

Production policies should be designed and reviewed according to enterprise standards.

Encryption Does Not Replace Access Control

Encryption at rest protects stored data.

It does not prevent an authorised Kubernetes identity from requesting and reading a Secret through the API.

Encryption at Rest

Protects Stored Data

+

RBAC

Controls API Access

+

Workload Identity

Controls External Secret Retrieval

All layers are required.

Kubernetes RBAC for Secrets

Secret access should be tightly restricted.

Example namespace-scoped Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: payment-secret-reader
  namespace: payments

rules:
  - apiGroups:
      - ""

    resources:
      - secrets

    resourceNames:
      - payment-database

    verbs:
      - get

This is safer than granting access to every Secret in the namespace.

Avoid Broad Secret Access

Avoid:

resources:
  - secrets

verbs:
  - "*"

Also review permissions such as:

list
watch
create
update
patch
delete

Each permission introduces different risks.

Indirect Secret Access

A user may access Secrets indirectly without having the get secrets permission.

Examples include identities that can:

Create Pods that mount Secrets
Modify Deployments
Execute commands inside Pods
Read application environment variables
Create debug containers
Modify Service Accounts
Create RoleBindings
Access node filesystems

Permission to Create Pod

↓

Pod Mounts Secret

↓

User Executes into Pod

↓

Secret Exposed

RBAC reviews must therefore consider privilege-escalation paths.

Default Service Account Risk

Every namespace normally contains a default Service Account.

Applications should not automatically use it.

Use a dedicated Service Account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: payment-api
  namespace: payments

Then reference it:

spec:
  serviceAccountName: payment-api

Disable Unnecessary API Tokens

If the application does not need the Kubernetes API:

spec:
  automountServiceAccountToken: false

This reduces credential exposure after Pod compromise.

External Secret Management

Instead of storing long-lived secret values directly in Kubernetes, applications can retrieve secrets from an external secret-management service.

AWS options include:

AWS Secrets Manager
AWS Systems Manager Parameter Store
AWS Private Certificate Authority for suitable certificate workflows
AWS KMS for cryptographic operations

AWS Secrets Manager

AWS Secrets Manager is designed to store, protect and manage secrets.

It supports:

Encryption
Fine-grained IAM access
Versioning
Rotation
CloudTrail logging
Cross-account access patterns
Integration with AWS services

Typical use cases include:

Database credentials
API keys
Application passwords
Third-party tokens

Systems Manager Parameter Store

Parameter Store manages configuration data and secure strings.

It supports:

Hierarchical parameter names
Standard and advanced parameter tiers
SecureString parameters
IAM access control
AWS KMS encryption
Version history
Integration with automation

Typical use cases include:

Application configuration
Environment-specific settings
Lower-complexity secure parameters

Secrets Manager Versus Parameter Store

Capability	Secrets Manager	Parameter Store
Primary purpose	Secrets lifecycle management	Configuration and parameter management
Secret rotation	Native rotation capabilities	Rotation usually requires custom automation
Versioning	Yes	Yes
KMS encryption	Yes	SecureString parameters
Hierarchical paths	Supported through naming	Strong hierarchical naming model
Typical use	Database passwords and API keys	Configuration and secure parameters
Cost model	Per-secret and API usage charges	Standard and advanced tier considerations

The organisation should select the service based on lifecycle, rotation, scale and governance requirements.

External Secret Retrieval Flow

Application Pod

↓

Kubernetes Service Account

↓

EKS Pod Identity or IRSA

↓

Temporary AWS Credentials

↓

AWS Secrets Manager

↓

Authorised Secret

The Pod should never contain permanent AWS access keys.

EKS Pod Identity

EKS Pod Identity associates an IAM role with a Kubernetes Service Account.

Pod

↓

Service Account

↓

EKS Pod Identity Association

↓

IAM Role

↓

Secrets Manager

Benefits include:

Temporary credentials
Simplified workload authentication
Least-privilege IAM roles
Reduced node-role dependency
Central AWS visibility

EKS Pod Identity allows applications to receive credentials without embedding or distributing static AWS credentials inside containers. :contentReference[oaicite:1]{index="1"}

IAM Roles for Service Accounts

IRSA uses a cluster OpenID Connect provider and a Kubernetes Service Account token.

Pod

↓

Service Account Token

↓

OIDC Provider

↓

AWS STS

↓

IAM Role

↓

Secrets Manager

IRSA remains a widely used workload-identity method.

EKS Pod Identity Versus IRSA

EKS Pod Identity	IRSA
Uses Pod Identity associations	Uses IAM OIDC trust
Simplifies IAM integration	Requires OIDC provider configuration
Uses Pod Identity Agent	Uses projected Service Account tokens
Suitable for supported EKS workloads	Widely adopted across existing clusters
Central association management	IAM trust policy includes Service Account identity

Both approaches support least-privilege, temporary AWS credentials.

IAM Policy for Secret Access

Example policy allowing retrieval of one secret:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadPaymentDatabaseSecret",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:example-region:123456789012:secret:payments/database-*"
    }
  ]
}

Avoid:

{
  "Effect": "Allow",
  "Action": "secretsmanager:*",
  "Resource": "*"
}

KMS Permissions for External Secrets

If a secret uses a customer-managed KMS key, the workload may also require permission such as:

{
  "Effect": "Allow",
  "Action": [
    "kms:Decrypt"
  ],
  "Resource": "arn:aws:kms:example-region:123456789012:key/example-key-id"
}

Use encryption context and key policies where appropriate to reduce unintended key use.

Secrets Store CSI Driver

The Secrets Store CSI Driver allows external secrets to appear as files mounted inside Kubernetes Pods.

Application Pod

↓

CSI Volume

↓

Secrets Store CSI Driver

↓

AWS Provider

↓

Secrets Manager or Parameter Store

The application reads the mounted files without requiring the secret value to be included in the Pod manifest.

AWS Secrets and Configuration Provider

The AWS Secrets and Configuration Provider works with the Secrets Store CSI Driver.

It can retrieve:

AWS Secrets Manager secrets
Systems Manager Parameter Store parameters

and mount them as files inside Amazon EKS Pods. :contentReference[oaicite:2]{index="2"}

CSI-Based Retrieval Architecture

Pod Starts

↓

CSI Driver Receives Mount Request

↓

AWS Provider Uses Pod Identity

↓

IAM Authorisation Evaluated

↓

Secret Retrieved

↓

Secret Mounted as In-Memory File

↓

Application Reads Secret

SecretProviderClass

A SecretProviderClass describes which external objects should be mounted.

Example:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: payment-secrets
  namespace: payments

spec:
  provider: aws

  parameters:
    usePodIdentity: "true"

    objects: |
      - objectName: "payments/production/database"
        objectType: "secretsmanager"
        objectAlias: "database.json"

AWS provides examples of SecretProviderClass resources for both Secrets Manager and Parameter Store integrations. :contentReference[oaicite:3]{index="3"}

Pod Using the CSI Secret

apiVersion: v1
kind: Pod
metadata:
  name: payment-api
  namespace: payments

spec:
  serviceAccountName: payment-api
  automountServiceAccountToken: false

  containers:
    - name: payment-api
      image: example/payment-api:1.0

      volumeMounts:
        - name: payment-secrets
          mountPath: /mnt/secrets
          readOnly: true

  volumes:
    - name: payment-secrets

      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true

        volumeAttributes:
          secretProviderClass: payment-secrets

The application can read:

/mnt/secrets/database.json

Parameter Store Example

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: payment-parameters
  namespace: payments

spec:
  provider: aws

  parameters:
    usePodIdentity: "true"

    objects: |
      - objectName: "/payments/production/api-url"
        objectType: "ssmparameter"
        objectAlias: "api-url"

Secret Synchronisation

Some CSI configurations can synchronise mounted external data into a native Kubernetes Secret.

External Secret Store

↓

CSI Driver

↓

Kubernetes Secret

↓

Environment Variable or Secret Volume

This may improve application compatibility, but it also reintroduces the secret into Kubernetes storage.

Before enabling synchronisation, assess:

Encryption
RBAC
Secret lifecycle
Cleanup behaviour
Rotation
Exposure through the Kubernetes API

Mounting Versus Synchronising

Direct CSI Mount	Kubernetes Secret Synchronisation
Secret appears as mounted file	Secret is created in Kubernetes
Reduces Kubernetes API exposure	Supports applications expecting native Secrets
Application must read a file	Supports environment variables
Rotation behaviour must be understood	Kubernetes Secret must be refreshed
External service remains source of truth	Duplicates secret into cluster storage

Direct mounting is generally preferable where the application supports file-based secrets.

Secret Rotation

Secret rotation replaces an existing credential with a new value.

Current Secret

↓

New Secret Generated

↓

Dependent Systems Updated

↓

Applications Reload Secret

↓

Old Secret Revoked

Rotation limits the period during which a compromised credential remains useful.

Rotation Requirements

Define:

Rotation frequency
Secret owner
Rotation method
Application-reload method
Failure handling
Rollback method
Revocation process
Monitoring
Evidence retention

Automatic Rotation

AWS Secrets Manager can support automated rotation workflows for suitable secrets.

A rotation workflow may:

Create a new credential.
Update the target service.
Test the new credential.
Promote the new version.
Retire the previous credential.

Rotation should be tested in non-production environments.

Application Reload Strategies

Applications may load secrets:

At startup
At regular intervals
When a file changes
Through an application reload endpoint
Through a controlled restart

If an application reads a secret only at startup, rotation may require a Pod restart.

Rotation Architecture

Secrets Manager

↓

Secret Rotated

↓

CSI Mount Updated

↓

Application Detects Change

or

↓

Controlled Pod Restart

↓

New Secret Used

The exact behaviour depends on the CSI configuration and application design.

Dual-Credential Rotation

For systems that support two active credentials:

Credential A Active

↓

Create Credential B

↓

Applications Move to B

↓

Validate

↓

Disable Credential A

This reduces downtime during rotation.

Emergency Rotation

Emergency rotation may be required after:

Suspected compromise
Accidental disclosure
Employee departure
Repository exposure
Logging exposure
Security incident
Third-party breach

Emergency procedures should identify:

Which secrets are affected
Which applications use them
Who can rotate them
How applications will be restarted
How old credentials will be revoked
How access will be investigated

Secret Inventory

Maintain an inventory containing:

Attribute	Example
Secret name	Payment database credential
Secret owner	Payments Team
Application	Payment API
Environment	Production
Storage location	AWS Secrets Manager
Rotation frequency	30 days
Last rotated	Recorded date
IAM role	Payment API role
KMS key	Approved customer-managed key
Criticality	High

Unknown secrets cannot be governed effectively.

Secret Naming Convention

Example:

/<business-unit>/<environment>/<application>/<purpose>

Examples:

/payments/production/payment-api/database

/digital/development/customer-portal/oauth

/security/production/scanner/api-key

Consistent naming improves access control and automation.

Secret Ownership

Every secret should have:

Business owner
Technical owner
Rotation owner
Recovery contact
Defined consumers
Approved storage service
Classification
Expiry or rotation requirement

Secrets in Git

Never commit plaintext secrets to:

Application repositories
Infrastructure repositories
Helm charts
Kubernetes manifests
.env files
Documentation
Example files

Git Secret Exposure

Secret Committed

↓

Repository History Retains Secret

↓

Secret Removed from Latest File

But

↓

Old Commit Still Contains Secret

Removing a secret from the latest commit does not automatically remove it from repository history.

The secret should be treated as compromised and rotated.

Secret Scanning

Use automated secret scanning in:

Developer workstations
Pre-commit checks
Pull requests
CI/CD pipelines
Container image scans
Repository monitoring

Detect patterns such as:

AWS access keys
Private keys
API tokens
Database connection strings
Passwords
OAuth credentials

Secrets in Container Images

Do not use:

ENV DATABASE_PASSWORD=ExamplePassword

Do not use:

COPY production-secrets.json /app/secrets.json

Image layers may retain secret data even if later Dockerfile steps delete the file.

Build-Time Secrets

When build-time secrets are unavoidable:

Use secure build-secret mechanisms.
Do not persist secrets in image layers.
Use short-lived credentials.
Restrict build-system access.
Review build logs.
Rotate exposed credentials.

Secrets in CI/CD

CI/CD systems may require:

Registry credentials
Deployment roles
Signing credentials
API tokens

Controls should include:

Short-lived credentials
Workload federation
Protected variables
Restricted log output
Approval controls
Secret masking
Rotation
Access reviews

Prefer identity federation over stored AWS access keys.

Secrets in Helm

Avoid plaintext values:

databasePassword: ExamplePassword

Use:

External secret references
Encrypted values workflows
Approved secret-management plugins
CSI-mounted secrets
Deployment-time secret retrieval

Secrets in Terraform

Terraform state may contain secret values.

Protect state through:

Encrypted remote storage
Restricted IAM access
State locking
Versioning
Audit logging
Backup
No public access
Limited output of sensitive values

Marking a Terraform value as sensitive limits display but does not necessarily remove it from the state file.

Secret Exposure Through Logs

Applications should never log:

Passwords
Tokens
API keys
Authorization headers
Private keys
Full connection strings

Apply:

Log redaction
Structured logging
Sensitive-field filtering
Secure error handling
Log-access controls

Secret Access Monitoring

Monitor access through:

AWS CloudTrail
Kubernetes Audit Logs
AWS KMS logs
Secrets Manager events
IAM policy changes
Kubernetes RBAC changes
Pod creation events
Service Account changes

High-Risk Secret Events

Alert on:

Unusual GetSecretValue activity
Access from an unexpected IAM role
Large numbers of secret retrievals
Access outside normal hours
Secret deletion
Rotation failure
KMS key disabling
Secret-policy changes
Cross-account access changes
Kubernetes Secret listing
New RoleBindings granting Secret access

Monitoring Architecture

Secrets Manager Events

+

KMS Events

+

CloudTrail

+

Kubernetes Audit Logs

+

Runtime Alerts

↓

Central SIEM

↓

SOC Investigation

CloudTrail Monitoring

CloudTrail can record activities such as:

Secret creation
Secret retrieval
Secret update
Rotation configuration
Secret deletion
Resource-policy changes
KMS operations
IAM role changes

Event data must be centralised and protected.

Kubernetes Audit Monitoring

Audit events should detect:

Secret get
Secret list
Secret watch
Secret creation
Secret update
Secret deletion
Pods mounting sensitive Secrets
Pod execution
RBAC changes

Audit policies should balance visibility, cost and sensitive-data protection.

Avoid Logging Secret Contents

Audit and application logs should capture access metadata without unnecessarily recording secret values.

Record:

Who accessed the secret

When

From where

Which secret

Result

Do Not Record:

Secret value

Cross-Account Secret Access

Enterprises may centralise secrets or share selected secrets across AWS accounts.

Architecture:

Application Account

↓

Pod IAM Role

↓

Cross-Account Authorisation

↓

Central Secrets Account

↓

Approved Secret

Controls should include:

Secret resource policies
IAM role policies
KMS key policies
Explicit account IDs
Least privilege
CloudTrail monitoring
Formal ownership

Cross-account designs add complexity and should be carefully reviewed.

Multi-Region Secrets

Applications operating in multiple Regions may require regional secret availability.

Consider:

Replication
KMS keys
Rotation consistency
Failover timing
Data residency
Recovery procedures
Regional IAM policies
Application configuration

High Availability

The secret-management architecture should not become a single point of failure.

Plan for:

AWS service availability
Regional failure
Network connectivity
VPC endpoints
CSI driver availability
Provider availability
IAM and STS access
Application caching
Controlled fallback behaviour

VPC Endpoints

Private EKS environments may use VPC endpoints for services such as:

AWS Secrets Manager
Systems Manager
AWS STS
AWS KMS

Private Pod

↓

VPC Endpoint

↓

AWS Secrets Manager

Benefits include:

Reduced internet dependency
Private service access
Endpoint-policy controls
Reduced NAT usage

Failure Behaviour

Applications should define behaviour when a secret cannot be retrieved.

Options may include:

Fail securely
Retry with backoff
Continue using a cached secret for a limited period
Stop accepting traffic
Trigger an operational alert

Avoid insecure fallback credentials.

Secret Caching

Caching may improve availability and reduce API calls.

Risks include:

Stale secrets
Delayed rotation
Memory exposure
Longer credential lifetime
Inconsistent replicas

Cache duration should be limited and documented.

Secret Revocation

Revocation removes the ability to use a credential.

Examples include:

Disable a database account
Delete or deactivate an API key
Remove an IAM role association
Revoke a certificate
Change a password
Remove a secret version
Update a KMS key policy

Deleting the stored secret without revoking the underlying credential may not prevent its use.

Kubernetes Secret Deletion

Deleting a Kubernetes Secret does not immediately erase secret values already loaded into:

Application memory
Environment variables
Mounted volumes
Logs
Backups
Running processes

Incident response may require:

Credential rotation
Pod restart
Node investigation
Log review
Image review

Backup and Recovery

Secret recovery must be carefully designed.

Questions include:

Can deleted secrets be recovered?
Are previous versions retained?
Who can restore them?
Are backups encrypted?
Are recovery actions logged?
Can recovery restore an already compromised credential?

Restoring old credentials may reintroduce security risk.

Certificate Management

TLS certificates contain sensitive private keys.

Manage:

Certificate issuance
Renewal
Private key storage
Rotation
Expiry monitoring
Revocation
Trust chains

Certificate automation may involve:

AWS Certificate Manager
AWS Private Certificate Authority
Kubernetes certificate controllers
External secret systems

Certificate Expiry Monitoring

Alert before certificates expire.

Example thresholds:

60 days
30 days
14 days
7 days

The exact thresholds should match organisational policy.

Admission Policies for Secrets

Admission controls can enforce:

No plaintext passwords in ConfigMaps
No suspicious environment-variable values
Approved secret-provider usage
Dedicated Service Accounts
No default Service Account for production
No unapproved Secret types
Required ownership labels
No hostPath mounting of credential locations
Restricted synchronisation into Kubernetes Secrets

Admission policy cannot detect every encoded secret reliably, so secret scanning is also required.

Namespace Isolation

Secrets should be stored in the same namespace as the workloads that use them unless a controlled alternative architecture is required.

Payments Namespace

├── Payment Workloads
├── Payment Service Account
└── Payment Secrets

Avoid unnecessarily sharing secrets across namespaces.

Secret Replication Risks

Replicating one Secret across many namespaces increases:

Exposure
Rotation complexity
Inconsistent versions
Cleanup difficulty
Audit volume

Prefer each workload retrieving the required secret from the authoritative external store.

Production Secrets Baseline

Control Area	Required Baseline
Storage	Approved external secret store
Encryption	AWS KMS
Workload access	EKS Pod Identity or IRSA
Kubernetes access	Least-privilege RBAC
Delivery	CSI-mounted file where supported
Rotation	Defined and tested
Monitoring	CloudTrail and Kubernetes audit
Repository protection	Secret scanning
Ownership	Named business and technical owners
Recovery	Tested secure procedure
Exceptions	Approved and time-limited

Common Secrets-Management Failures

Base64 Treated as Encryption

Risk: Anyone with API access can decode the secret.

Control: Use encryption at rest and restrict RBAC.

Secrets Stored in Git

Risk: Repository history preserves exposed credentials.

Control: Remove, rotate and investigate the credential.

Broad Secret Permissions

Risk: One compromised identity can read every application secret.

Control: Use resource-specific IAM and RBAC policies.

Secrets Stored in Images

Risk: Anyone who pulls the image may retrieve the secret.

Control: Inject secrets only at runtime.

Long-Lived AWS Access Keys

Risk: Stolen credentials remain valid for long periods.

Control: Use EKS Pod Identity or IRSA.

Secrets Exposed as Environment Variables

Risk: Values may appear in process or diagnostic data.

Control: Prefer mounted files where appropriate.

No Rotation

Risk: Compromised credentials remain valid indefinitely.

Control: Define automated or scheduled rotation.

Rotation Without Application Reload

Risk: Applications continue using an old credential.

Control: Test refresh and restart behaviour.

Secret Values in Logs

Risk: Sensitive information enters central logging systems.

Control: Implement redaction and secure error handling.

Shared Application Credentials

Risk: Accountability and revocation become difficult.

Control: Use application-specific identities and credentials.

Excessive Node IAM Permissions

Risk: Compromised Pods access unrelated secrets.

Control: Use pod-level identity and narrow IAM roles.

Unmonitored Secret Access

Risk: Credential theft remains undetected.

Control: Centralise CloudTrail and audit events.

Enterprise Secrets Architecture

Enterprise Identity and Governance

↓

AWS Secrets Manager or Parameter Store

↓

Customer-Managed AWS KMS Key

↓

Resource-Specific IAM Policy

↓

EKS Pod Identity or IRSA

↓

Kubernetes Service Account

↓

AWS Secrets and Configuration Provider

↓

Secrets Store CSI Driver

↓

Read-Only File Mount

↓

Application Pod

↓

CloudTrail, Audit Logs and SIEM

Enterprise Implementation Strategy

Phase 1 — Discover Existing Secrets

Scan Git repositories.
Review Kubernetes Secrets.
Review CI/CD variables.
Review container images.
Review Terraform state.
Identify application credentials.
Identify shared secrets.
Assign owners.

Phase 2 — Classify Secrets

Classify by:

Business criticality
Data sensitivity
Environment
Consumer
Rotation requirement
Compliance scope
Recovery requirement

Phase 3 — Select Authoritative Secret Stores

Use Secrets Manager for managed secret lifecycles.
Use Parameter Store for suitable secure parameters.
Define approved Kubernetes Secret use cases.
Define KMS key requirements.
Define regional and cross-account models.

Phase 4 — Implement Workload Identity

Create dedicated Service Accounts.
Use EKS Pod Identity or IRSA.
Create resource-specific IAM roles.
Remove long-lived access keys.
Reduce worker-node permissions.

Phase 5 — Implement Secure Delivery

Install and govern the Secrets Store CSI Driver.
Install the AWS provider.
Define SecretProviderClass resources.
Mount secrets as read-only files.
Avoid synchronisation unless required.
Use private VPC endpoints where appropriate.

Phase 6 — Apply Access Controls

Restrict Kubernetes RBAC.
Restrict IAM roles.
Restrict KMS access.
Protect Service Account associations.
Review indirect secret-access permissions.
Apply separation of duties.

Phase 7 — Implement Rotation

Define rotation schedules.
Automate suitable credential rotation.
Test application reload.
Define emergency rotation.
Revoke old credentials.
Monitor rotation failures.

Phase 8 — Protect Development and CI/CD

Enable repository secret scanning.
Use pre-commit checks.
Protect pipeline variables.
Use workload federation.
Scan images and build logs.
Protect Terraform state.

Phase 9 — Monitor and Respond

Centralise CloudTrail events.
Enable Kubernetes Audit Logs.
Alert on unusual access.
Monitor KMS changes.
Monitor Secret deletion.
Create credential-exposure runbooks.
Test emergency rotation.

Phase 10 — Validate Continuously

Review secret inventory.
Review owners.
Review rotation status.
Review IAM and RBAC.
Review unused secrets.
Review failed retrievals.
Review expired certificates.
Review exceptions.

Enterprise Best Practices

As a Cloud Security Engineer:

Never treat Base64 encoding as encryption.
Avoid storing long-lived secret values directly in Kubernetes where possible.
Use AWS Secrets Manager or Parameter Store as the authoritative source.
Encrypt secrets using approved AWS KMS keys.
Use dedicated Kubernetes Service Accounts.
Use EKS Pod Identity or IRSA for AWS access.
Grant workloads access only to specific secrets.
Prefer read-only file mounts where applications support them.
Avoid passing sensitive values through environment variables.
Never commit secrets to Git.
Never embed secrets in container images.
Protect CI/CD variables and Terraform state.
Define and test secret rotation.
Revoke old credentials after rotation.
Monitor CloudTrail and Kubernetes audit events.
Alert on unusual secret retrieval.
Disable unnecessary Service Account token mounting.
Review indirect secret-access paths.
Use VPC endpoints for private AWS service access where required.
Maintain an accurate secret inventory.
Assign ownership and rotation responsibility.
Treat exposed secrets as compromised and rotate them immediately.
Test secret-management failure and recovery scenarios.

Real-World Scenario

A financial organisation runs a payment platform on Amazon EKS.

A security review identifies:

Database passwords stored in Kubernetes manifests
Base64 values incorrectly described as encrypted
Shared credentials used by several applications
Broad RBAC access to all namespace Secrets
Worker-node roles with permission to retrieve every secret
Secrets passed through environment variables
No rotation process
No monitoring of secret retrieval
API keys accidentally stored in CI/CD logs

The Cloud Security and Platform teams introduce an enterprise secrets-management programme.

They:

Inventory every application secret.
Assign business and technical owners.
Move production credentials into AWS Secrets Manager.
Encrypt secrets using approved AWS KMS keys.
Create a dedicated Service Account for each application.
Implement EKS Pod Identity with one IAM role per workload.
Restrict each role to its required Secrets Manager resource.
Deploy the Secrets Store CSI Driver and AWS provider.
Mount secrets as read-only files.
Remove passwords from manifests, Helm values and environment variables.
Reduce Kubernetes RBAC access to Secret objects.
Remove secret permissions from worker-node roles.
Implement database credential rotation.
Update applications to reload rotated credentials.
Enable repository and CI/CD secret scanning.
Centralise CloudTrail and Kubernetes audit logs.
Create alerts for unusual secret access.
Test emergency credential rotation.
Establish expiry and certificate monitoring.
Review unused and unowned secrets every quarter.

The organisation achieves:

Reduced credential exposure
Stronger least privilege
Centralised secret ownership
Automated rotation
Improved auditability
Faster incident response
Reduced dependency on static credentials

Key Takeaways

Kubernetes Secrets are Base64 encoded and require additional protection.
Encryption at rest does not replace RBAC or workload identity.
AWS Secrets Manager provides managed storage, auditing and rotation capabilities.
Parameter Store supports configuration and secure parameter management.
EKS Pod Identity and IRSA provide temporary workload credentials.
The Secrets Store CSI Driver can mount external secrets as files inside Pods.
File-based delivery may reduce exposure compared with environment variables.
IAM roles should grant access only to specific secrets and KMS keys.
Secret rotation must include application reload and old-credential revocation.
Secrets must never be committed to Git or embedded in container images.
CloudTrail and Kubernetes Audit Logs provide important access visibility.
Secret inventory, ownership and continuous review are essential enterprise controls.

Knowledge Check

1. Why is Base64 encoding not sufficient protection for Kubernetes Secrets?

Answer: Base64 only changes the representation of the data. Anyone with access to the encoded value can easily decode it, so encryption and access controls are still required.

2. What is the benefit of using EKS Pod Identity or IRSA?

Answer: They provide temporary, least-privilege AWS credentials to individual Kubernetes workloads without embedding static credentials or relying on broad worker-node IAM roles.

3. What is the role of the Secrets Store CSI Driver?

Answer: It allows Kubernetes Pods to retrieve secrets from external providers and mount them as files through a CSI volume.

4. Why may file-mounted secrets be preferable to environment variables?

Answer: Environment variables may be exposed through process inspection, debugging, crash reports or accidental logging. File mounts can provide more controlled access and easier refresh behaviour.

5. What should happen when a secret is accidentally committed to Git?

Answer: The secret should be treated as compromised, immediately rotated or revoked, removed from active files and repository history where appropriate, and investigated for unauthorised use.

What’s Next?

In the next lesson, we will explore Amazon EKS Logging and Monitoring, including EKS control plane logs, Kubernetes Audit Logs, application and node logs, CloudWatch, Prometheus, Grafana, security alerts and enterprise SIEM integration.

➡️ Next Lesson: Lesson 07 — Logging & Monitoring