Lab 03 GCP Incident Response

Cloud security incidents rarely affect only one resource.

A compromised identity may create infrastructure, modify IAM, access data, change network rules, or attempt to hide activity.

In this lab, you will combine the skills from:

Cloud Logging
Cloud Audit Logs
GCP IAM Security

and apply them to a structured Google Cloud incident response investigation.

Mission Goal: Investigate a simulated Google Cloud security incident, determine what happened, identify the affected identities and resources, contain the issue, remediate the root cause, and document the complete incident timeline.

Mission Information

Difficulty: Intermediate
Estimated Time: 90–120 minutes
Primary Skill: Google Cloud Incident Response
Platform: Google Cloud
Career Alignment: Cloud Security Engineer, SOC Analyst, Incident Responder, Security Consultant

What You Will Learn

By completing this lab, you will learn how to:

Triage a cloud security alert
Define an investigation scope
Identify suspicious principals
Review Cloud Audit Logs
Investigate IAM changes
Trace affected resources
Build an incident timeline
Assess potential blast radius
Contain compromised access
Preserve relevant evidence
Remediate security weaknesses
Validate recovery
Create a professional incident report

Why This Lab Matters

A cloud security engineer must be able to move from:

Something Suspicious Happened

to:

What Happened?
        ↓
Who Did It?
        ↓
Which Resources Were Affected?
        ↓
How Far Did It Spread?
        ↓
How Do We Contain It?
        ↓
How Do We Prevent It Again?

That is the core purpose of incident response.

Incident Response Lifecycle

A practical cloud incident response lifecycle looks like:

Preparation
    ↓
Detection
    ↓
Triage
    ↓
Investigation
    ↓
Containment
    ↓
Eradication
    ↓
Recovery
    ↓
Lessons Learned

In this lab, your focus will be primarily on:

Detection
    ↓
Triage
    ↓
Investigation
    ↓
Containment
    ↓
Remediation
    ↓
Recovery

Lab Scenario

You are working as a Cloud Security Engineer.

Your SOC team reports suspicious activity in a Google Cloud project.

The alert indicates that an unexpected administrative change occurred, followed by activity involving additional cloud resources.

The initial information provided is:

Project:
Training GCP Project

Alert:
Unexpected administrative activity

Approximate Time:
During the current lab window

Possible Indicators:
IAM change
Resource creation
Network modification

Your task is to determine what happened.

Simulated Attack Story

For this lab, imagine the following authorized scenario:

Compromised / Test Identity
        ↓
Unexpected IAM Activity
        ↓
Cloud Resource Created
        ↓
Network Configuration Changed
        ↓
Additional Cloud Activity

You will investigate the sequence from the defensive side.

Important Lab Rule

Use only:

Your own Google Cloud lab
An authorized training environment
Disposable resources
Test identities

Do not perform the scenario against production or another organization’s environment.

Lab Architecture

Suspicious Identity
        ↓
IAM Change
        ↓
GCP Project
        │
        ├── Compute Resource
        ├── Network Configuration
        ├── Storage / Service
        │
        └── Cloud Audit Logs
                 ↓
           Cloud Logging
                 ↓
         Security Investigation
                 ↓
            Containment

Part 1 — Preparation

Before investigating, understand the environment.

Document:

Project ID:
[Record]

Expected Administrators:
[Record]

Known Service Accounts:
[Record]

Normal Workloads:
[Record]

Expected Network Configuration:
[Record]

This creates a basic baseline.

Why Baselines Matter

You cannot easily identify abnormal behavior unless you understand what normal looks like.

For example:

New VM

may be suspicious in one project and completely normal in another.

Context matters.

Part 2 — Receive the Security Alert

Your simulated SOC alert is:

Unexpected privileged activity has been observed inside the GCP project.

Do not immediately assume full compromise.

Begin with triage.

Triage Questions

Ask:

What triggered the alert?

When did it occur?

Which project was affected?

Which identity was involved?

Which resource was involved?

Was the operation successful?

Record the initial information.

Incident Record

Create:

Incident ID:
GCP-IR-LAB-001

Severity:
Under Investigation

Detected:
[Timestamp]

Affected Project:
[Project]

Initial Indicator:
Unexpected Administrative Activity

Analyst:
[Your Name]

Part 3 — Define the Investigation Time Window

Start with a manageable time range.

For example:

Alert Time
    ↓
Review activity before alert
    ↓
Review activity after alert

You want enough context to find:

Initial activity
Follow-on actions
Related changes

Do not immediately search weeks of logs if the suspicious event occurred minutes ago.

Part 4 — Open Cloud Logging

Navigate to:

Logging
   ↓
Log Explorer

Set your investigation time window.

Begin with administrative activity.

Look for:

Resource creation
IAM changes
Firewall changes
Service account activity
Resource deletion
Unexpected administrative operations

Investigation Principle

Start broad.

Then narrow.

Time Window
    ↓
Suspicious Event
    ↓
Principal
    ↓
Resources
    ↓
Related Activity

Part 5 — Identify the First Suspicious Event

Locate an event that appears related to the alert.

For the event, record:

Timestamp:
[Record]

Principal:
[Record]

Service:
[Record]

Method:
[Record]

Resource:
[Record]

Result:
[Record]

Key Question

Ask:

Was this principal expected to perform this action?

If not, mark the event for deeper investigation.

Part 6 — Identify the Principal

Determine which identity initiated the suspicious action.

It may be:

Human user
Service account
Workload identity
Automation identity

Document:

Principal:
[Identity]

Identity Type:
[User / Service Account / Other]

Expected Role:
[Record]

Observed Activity:
[Record]

Principal Analysis

Now ask:

Is this identity legitimate?
        ↓
Is the activity expected?
        ↓
Does the identity normally have this privilege?
        ↓
Has the identity performed other unusual actions?

Part 7 — Expand the Investigation Around the Principal

Once you identify the suspicious principal, review other activity generated by it.

Your objective becomes:

Known Suspicious Identity
        ↓
Everything It Did

Look for:

IAM changes
Resource creation
Resource deletion
Firewall changes
Service account actions
Storage access
Logging changes
Key-management actions

Build a Principal Activity Table

Time	Action	Resource	Result	Assessment
Record	IAM Change	Project	Success	Suspicious
Record	VM Creation	VM	Success	Suspicious
Record	Firewall Change	Rule	Success	Suspicious

Part 8 — Investigate IAM Changes

Identity changes can increase attacker capability.

Look for events involving:

Role assignments
Policy changes
Service account permissions
Impersonation permissions
Privileged access

Ask:

Did the identity gain privilege?
        ↓
Did it grant privilege to another identity?
        ↓
Did it modify access to sensitive resources?

IAM Escalation Example

A possible sequence might look like:

Low-Privilege Identity
        ↓
IAM Change
        ↓
Higher Privilege
        ↓
Administrative Operations

This is a high-value investigation pattern.

Part 9 — Determine Whether Privilege Changed

For any IAM modification, document:

Changed By:
[Principal]

Affected Principal:
[Identity]

Affected Resource:
[Resource]

Previous Access:
[If Known]

New Access:
[If Known]

Security Impact:
[Assessment]

Part 10 — Investigate Resource Creation

Look for unexpected new resources.

Examples may include:

Compute instances
Service accounts
Storage resources
Network rules
Serverless services

Ask:

Who created the resource?

When?

Why?

Which identity was attached?

Was it expected?

Suspicious Compute Resource

If an unexpected VM exists, investigate:

VM
│
├── Creator
├── Creation Time
├── Network
├── Public IP
├── Service Account
├── Disks
└── Firewall Exposure

Part 11 — Examine Attached Identity

If the suspicious workload uses a service account, review that service account.

Ask:

What roles does it have?
What resources can it access?
Can it be impersonated?
Does it have more privilege than required?

The attack path may look like:

Compromised User
      ↓
Creates VM
      ↓
VM Uses Privileged Service Account
      ↓
Sensitive Resource Access

Part 12 — Investigate Network Changes

Look for network modifications around the incident time.

Examples:

Firewall rule created
Firewall rule modified
Public exposure added
Network configuration changed

Ask:

What changed?

Who changed it?

Which resource became reachable?

Was the source broad?

Which ports were exposed?

Network Exposure Example

Before

Internet
   X
Private Workload

After

Internet
   ↓
New Firewall Rule
   ↓
Workload

This can significantly alter incident severity.

Part 13 — Review Firewall Exposure

For any suspicious firewall change, assess:

Direction
Source
Destination
Protocol
Port
Target workload
Business justification

Broad internet exposure deserves additional attention.

Part 14 — Investigate Resource-Based Activity

Now pivot from the identity to each affected resource.

For each suspicious resource ask:

Who created it?

Who modified it?

Which identities accessed it?

Which configuration changed?

What happened afterward?

This helps reveal secondary activity.

Part 15 — Review Service Account Activity

If a service account appears in the incident, investigate its activity separately.

Ask:

Was the service account used normally?

Which resources did it access?

Did activity begin only after the suspicious event?

Was it impersonated?

Are keys involved?

Service Account Incident Risk

A compromised service account can allow persistent automated access.

Therefore determine:

What can this identity reach?

not merely:

Which workload uses it?

Part 16 — Investigate Credential Risk

Review whether the incident involves:

Service account keys
Long-lived credentials
External credentials
Impersonation

If a credential may be compromised, containment should account for it.

Part 17 — Look for Security Control Changes

Attackers sometimes modify defensive controls.

Review activity related to:

Logging
Monitoring
IAM
Network controls
Security policies

Ask:

Was visibility reduced?

Was a security control disabled?

Was access broadened?

Was evidence modified?

Part 18 — Preserve Evidence

Before aggressive remediation, preserve relevant evidence.

Depending on the scenario, this may include:

Cloud Audit Logs
Relevant application logs
VM details
Disk snapshots
IAM policies
Firewall configuration
Resource metadata
Screenshots or exported findings

Evidence Principle

Avoid:

See Suspicious VM
        ↓
Immediately Delete

Prefer:

See Suspicious VM
        ↓
Record Configuration
        ↓
Preserve Relevant Evidence
        ↓
Contain
        ↓
Remediate

Part 19 — Build the Incident Timeline

Create a timeline from your findings.

Example:

10:03
Suspicious identity performs IAM change

10:05
New privileged access becomes effective

10:09
Compute instance created

10:12
Firewall rule modified

10:14
Workload begins additional activity

10:20
SOC alert generated

10:25
Investigation begins

The timeline is one of the most important incident artifacts.

Timeline Table

Time	Principal	Event	Resource	Security Relevance
Record	Identity	IAM Change	Project	Privilege change
Record	Identity	VM Created	VM	New resource
Record	Identity	Firewall Modified	Firewall	Exposure
Record	Identity	Resource Access	Resource	Follow-on activity

Part 20 — Determine the Initial Access Point

Based on the evidence, identify the earliest suspicious event.

Ask:

What happened first?

This may be:

Suspicious authentication
IAM modification
Credential use
Service account impersonation
Resource deployment

Do not assume the first alert was the first malicious action.

Part 21 — Determine the Blast Radius

Blast radius answers:

How much of the environment could the compromised identity affect?

Review:

Identity Permissions
      ↓
Accessible Projects
      ↓
Accessible Resources
      ↓
Service Accounts
      ↓
Network Reachability
      ↓
Sensitive Data

Blast Radius Assessment

Document:

Affected Identity:
[Record]

Projects Accessible:
[Record]

High-Privilege Roles:
[Record]

Sensitive Resources Accessible:
[Record]

Potential Data Exposure:
[Record]

Network Reach:
[Record]

Part 22 — Assign Incident Severity

Assess severity based on factors such as:

Privilege level
Sensitive resource access
Data exposure
Persistence
Network exposure
Number of affected resources

A simple model:

Low
Limited impact

Medium
Meaningful unauthorized access

High
Privileged compromise or sensitive-resource impact

Critical
Broad administrative compromise or major data impact

Document your rationale.

Part 23 — Containment Planning

Do not begin containment randomly.

First decide:

What must be stopped immediately?

What evidence must be preserved?

What business services must remain available?

Possible containment actions may include:

Disable or restrict compromised identity
Remove unauthorized IAM bindings
Disable compromised keys
Restrict firewall rules
Isolate workloads
Stop suspicious resources
Remove unauthorized impersonation rights

Part 24 — Contain the Identity

If the identity is confirmed compromised in your lab scenario, restrict its ability to continue operating.

Conceptually:

Compromised Identity
        ↓
Access Restricted
        ↓
Additional Activity Blocked

Only perform actions appropriate to your lab environment.

Part 25 — Contain IAM Abuse

Remove unauthorized IAM changes.

Your objective is:

Unauthorized Privilege
        ↓
Remove Binding
        ↓
Restore Approved Access

Afterward, verify that expected administrators still retain legitimate access.

Part 26 — Contain Service Account Risk

If a service account is involved:

Review roles
Remove unauthorized access
Disable compromised keys where applicable
Review impersonation permissions
Determine whether the workload should continue using the identity

Do not simply delete an account that legitimate production services depend on.

Part 27 — Contain Network Exposure

If an unauthorized firewall rule exists:

Broad Exposure
      ↓
Restrict / Remove Rule
      ↓
Validate Required Connectivity

Security containment should not unintentionally cause unnecessary outages.

Part 28 — Isolate Suspicious Workloads

If a suspicious workload is involved, isolation may be appropriate.

Conceptually:

Suspicious VM
      ↓
Network Isolation
      ↓
Evidence Review

The exact containment method depends on the environment and incident.

Part 29 — Eradicate the Root Cause

Containment stops immediate activity.

Eradication addresses why the incident was possible.

Potential root causes may include:

Excessive IAM
Leaked credentials
Long-lived service account key
Weak access controls
Overly broad network exposure
Missing guardrails

Root Cause Question

Ask:

What security weakness allowed this activity to occur?

Do not stop at:

Which resource was compromised?

Part 30 — Remediate IAM

Possible remediation may include:

Replace broad roles
Reduce scope
Remove unnecessary access
Review group membership
Review impersonation
Remove unused service accounts

Use least privilege.

Part 31 — Remediate Credential Risk

If a credential may have been compromised:

Potentially Compromised Credential
          ↓
Revoke / Rotate
          ↓
Update Workload
          ↓
Validate

Also determine why the credential was exposed.

Part 32 — Remediate Network Security

Review whether the incident reveals weak network controls.

Potential improvements include:

Reduce public exposure
Narrow firewall sources
Limit ports
Improve segmentation
Prefer private connectivity

Part 33 — Review Logging Coverage

Ask:

Were we able to reconstruct the incident?

If not, identify logging gaps.

Examples:

Missing audit visibility
Missing network telemetry
Missing workload logs
Insufficient retention

Incident response often reveals observability weaknesses.

Part 34 — Recovery

After remediation, restore the environment to a trusted state.

Recovery may include:

Re-enable approved user access
Restore secure workload configuration
Recreate compromised workloads from trusted configurations
Validate network access
Validate application functionality
Confirm monitoring

Recovery Principle

Do not simply return to:

System Works

You want:

System Works
        +
Known Security Weakness Fixed
        +
Monitoring Confirmed

Part 35 — Validate IAM

After recovery, confirm:

Unauthorized roles are gone
Required roles remain
Service account access is appropriate
No unexpected identities remain

Part 36 — Validate Networking

Confirm:

Unauthorized firewall rules are removed
Public exposure is appropriate
Required application connectivity still functions

Part 37 — Validate Resources

Inventory suspicious or newly created resources.

Determine whether each should be:

Retain
Rebuild
Remove

Document the decision.

Part 38 — Monitor for Recurrence

After containment and recovery, review logs for new activity.

Ask:

Is the compromised identity still active?

Are similar resources being recreated?

Are additional IAM changes occurring?

This helps confirm containment succeeded.

Part 39 — Build the Final Incident Timeline

Your final timeline should include both attacker/test activity and defender activity.

Example:

10:03 — Suspicious IAM change
10:09 — VM created
10:12 — Firewall modified
10:20 — Alert generated
10:25 — Investigation started
10:38 — Compromised access contained
10:45 — Firewall exposure removed
10:55 — Unauthorized IAM removed
11:10 — Recovery validated

Part 40 — Document Evidence

Create an evidence table.

Evidence ID	Source	Description	Relevance
E01	Cloud Audit Logs	IAM modification event	Privilege change
E02	Cloud Audit Logs	VM creation	Suspicious resource
E03	Network Configuration	Firewall rule	Exposure
E04	IAM	Service account roles	Blast radius

Part 41 — Create an Incident Summary

Use:

Incident:
Unauthorized Google Cloud administrative activity

Severity:
High

Affected Environment:
Training GCP Project

Initial Indicator:
Unexpected IAM and resource activity

Primary Principal:
[Test Identity]

Affected Resources:
[Record]

Root Cause:
[Record]

Containment:
[Record]

Remediation:
[Record]

Recovery:
Completed / In Progress

Current Status:
Contained

Part 42 — Root Cause Analysis

Write a concise root cause statement.

Example:

The incident was enabled by excessive privileges assigned to a test identity. The identity was able to modify IAM and create additional resources beyond its intended business requirement.

Avoid vague explanations such as:

Security Was Weak

Be specific.

Part 43 — Identify Contributing Factors

The root cause may have additional contributing factors.

Examples:

Broad role assignment
Missing access review
Lack of preventive policy
Unnecessary public exposure
Weak service account governance
Insufficient alerting

Part 44 — Create Corrective Actions

Your incident should produce improvements.

Example:

Issue	Corrective Action
Excessive privilege	Implement least privilege
Long-lived credentials	Move to managed identity
Broad firewall rule	Restrict source and ports
Delayed detection	Improve alerting
Missing governance	Add organization guardrails

Part 45 — Prevent Recurrence

The best incident response does more than fix the affected resource.

Think:

Incident
   ↓
Root Cause
   ↓
Security Improvement
   ↓
Prevent Recurrence

Examples include:

IAM reviews
Organization policies
Central logging
Security Command Center
Detection rules
Automated configuration validation

Part 46 — Detection Improvement

Ask:

How could we detect this earlier next time?

Potential detection opportunities include:

Privileged IAM changes
Service account key creation
Unexpected resource creation
Broad firewall rules
Logging configuration changes

Part 47 — Map the Incident to Security Controls

Take each stage of the incident and identify a defensive control.

Example:

Excessive IAM
      ↓
Least Privilege

Suspicious IAM Change
      ↓
Audit Logging + Alerting

Public Exposure
      ↓
Network Policy

Unexpected Resource
      ↓
Asset Monitoring

Incident
      ↓
IR Runbook

This converts incident response into security engineering.

Part 48 — Incident Response Workflow

Build this reusable workflow:

01 Receive Alert
        ↓
02 Confirm Scope
        ↓
03 Set Time Window
        ↓
04 Review Audit Logs
        ↓
05 Identify Principal
        ↓
06 Review IAM Activity
        ↓
07 Identify Resources
        ↓
08 Review Network Changes
        ↓
09 Analyze Service Accounts
        ↓
10 Build Timeline
        ↓
11 Determine Blast Radius
        ↓
12 Preserve Evidence
        ↓
13 Contain
        ↓
14 Remediate
        ↓
15 Recover
        ↓
16 Monitor
        ↓
17 Document
        ↓
18 Improve Controls

Lab Challenge 1 — Find the Initial Suspicious Event

Using your lab logs, identify the earliest suspicious event.

Document:

Time
Principal
Action
Resource
Reason it is suspicious

Lab Challenge 2 — Build a Principal Timeline

Take the test identity and document every relevant action it performed during the lab.

Classify each as:

Expected
Suspicious
Confirmed Incident Activity

Lab Challenge 3 — Identify Blast Radius

Determine:

Which roles the identity had
Which resources it could modify
Which service accounts it could use
Which sensitive assets were potentially reachable

Lab Challenge 4 — Investigate Network Exposure

Find the simulated network change.

Explain:

What changed
Why it created risk
How you would contain it

Lab Challenge 5 — Perform Containment

Safely contain the simulated incident.

Verify that:

Unauthorized access stops
Required legitimate functionality remains
Evidence is retained

Lab Challenge 6 — Write the Incident Report

Produce a one-page summary containing:

Incident Overview
Timeline
Affected Assets
Root Cause
Containment
Remediation
Recovery
Lessons Learned

Incident Report Template

Incident ID:
GCP-IR-LAB-001

Title:
Unauthorized Google Cloud Administrative Activity

Severity:
[Low / Medium / High / Critical]

Date:
[Record]

Affected Project:
[Record]

Summary:
[Short description]

Initial Detection:
[How discovered]

Affected Principal:
[Record]

Affected Resources:
[Record]

Timeline:
[Key events]

Root Cause:
[Record]

Blast Radius:
[Record]

Containment Actions:
[Record]

Remediation:
[Record]

Recovery:
[Record]

Evidence:
[Record]

Lessons Learned:
[Record]

Preventive Actions:
[Record]

Lab Success Criteria

You have successfully completed this lab when you can:

Triage a security alert
Define an investigation window
Identify a suspicious principal
Review related audit activity
Investigate IAM changes
Identify affected resources
Review suspicious network changes
Analyze service account involvement
Build an incident timeline
Determine blast radius
Preserve evidence
Perform safe containment
Remediate the root cause
Validate recovery
Write an incident report

Incident Response Checklist

Recorded the initial alert
Identified affected project
Defined investigation time window
Reviewed Cloud Audit Logs
Identified suspicious principal
Reviewed IAM activity
Identified affected resources
Reviewed network changes
Reviewed service account activity
Built incident timeline
Assessed blast radius
Preserved evidence
Contained suspicious access
Removed unauthorized IAM
Remediated network exposure
Reviewed credentials
Validated recovery
Monitored for recurrence
Completed incident report
Documented lessons learned

Interview Questions to Practice

After completing this lab, you should be able to answer:

How would you investigate suspicious activity in GCP?
What log sources would you review first?
How do Cloud Audit Logs support incident response?
How would you identify the principal behind an IAM change?
How would you investigate a compromised service account?
What is blast radius?
How would you determine whether privilege escalation occurred?
What evidence would you preserve during a cloud incident?
How would you contain a compromised cloud identity?
How would you respond to unauthorized IAM changes?
How would you investigate an unexpected VM?
How would you respond to an exposed firewall rule?
Why should you build an incident timeline?
What is the difference between containment and remediation?
Why should credentials be rotated after compromise?
How would you validate cloud recovery?
How can incident response improve preventive security?
Why is centralized logging important?
How would you identify the root cause of a GCP incident?
How would you prevent recurrence?

Cloud Incident Responder Mindset

When an alert appears, avoid jumping randomly between resources.

Use a structured thought process:

Alert
  ↓
Identity
  ↓
Activity
  ↓
Resources
  ↓
Privilege
  ↓
Network
  ↓
Data
  ↓
Blast Radius
  ↓
Containment

Then ask:

What happened before this?

What happened after this?

What else could the identity access?

What must be contained first?

Key Takeaway

Cloud incident response is an evidence-driven process.

Your investigation should move from:

Suspicion

to:

Evidence
   ↓
Timeline
   ↓
Scope
   ↓
Impact
   ↓
Containment
   ↓
Recovery

The objective is not merely to stop suspicious activity.

The objective is to understand the incident well enough to contain it safely, recover confidently, and prevent it from happening again.

Clean Up

After completing the lab:

Remove test IAM bindings
Remove temporary service accounts if appropriate
Remove test firewall changes
Remove disposable resources
Remove lab credentials that are no longer required
Confirm the project has returned to its secure baseline

Keep your incident notes and evidence summary for your portfolio.

What’s Next?

➡️ Lab 04 — GCP Network Security

In the next lab, you will focus deeply on protecting and assessing Google Cloud network architecture.

You will review:

VPC
  ↓
Subnets
  ↓
Routes
  ↓
Firewall Rules
  ↓
Public IP Exposure
  ↓
Private Connectivity
  ↓
Cloud NAT
  ↓
Network Logging
  ↓
Security Assessment

You will learn how to identify overly broad access, review internet exposure, validate segmentation, analyze network security controls, and document findings like a Google Cloud Security Engineer.