Runbook 01 — Linux Incident Investigation

A security alert tells you:

Something May Be Wrong

It does not automatically tell you:

What Happened

Who Did It

How It Happened

How Far It Spread

Whether It Is Still Active

That is the purpose of incident investigation.

This runbook provides a repeatable workflow for investigating suspicious activity on an authorized Linux system.

Runbook Information

Runbook: Linux Incident Investigation
Level: Intermediate → Advanced
Primary Role: SOC Analyst / Incident Responder
Supporting Roles: Linux Security Engineer, Cloud Security Engineer, DFIR Analyst, Security Consultant
Environment: Authorized Linux systems
Objective: Determine what happened, establish scope and impact, preserve useful evidence, and support safe containment and remediation decisions.

Mission Scenario

Your SOC receives an alert:

Suspicious Outbound Connection
Detected from Linux Server

The affected host supports an important business application.

Initial monitoring shows:

Unexpected External IP
        +
Unusual Process Activity
        +
Recent Administrative Login

You are assigned to investigate.

Your job is not simply to:

Block the IP

or:

Kill the Process

Your job is to answer:

What Happened?

When Did It Start?

Which Identity Was Involved?

Which Process Was Responsible?

How Was It Started?

Did It Establish Persistence?

Which Systems Were Affected?

Is the Activity Still Active?

What Should We Contain?

What Evidence Must Be Preserved?

Incident Investigation Philosophy

Use:

ALERT
  ↓
VALIDATE
  ↓
PRESERVE
  ↓
TRIAGE
  ↓
INVESTIGATE
  ↓
CORRELATE
  ↓
SCOPE
  ↓
CONTAIN
  ↓
REMEDIATE
  ↓
RECOVER
  ↓
DOCUMENT

Core Principle

During an investigation:

Evidence
   >
Assumption

Never conclude:

Unfamiliar Process
=
Malware

Instead:

Observe
   ↓
Collect
   ↓
Correlate
   ↓
Validate
   ↓
Conclude

Golden Rule — Do Not Destroy Evidence

When suspicious activity is discovered, avoid immediately:

Rebooting

Deleting Files

Killing Processes

Clearing Logs

Removing Users

Changing Every Configuration

Reinstalling the Server

unless your approved incident-response procedure requires immediate containment.

These actions may destroy:

Running Process Evidence

Active Connections

Process Relationships

Memory-Resident Information

Temporary Files

Useful Timestamps

Investigation Priorities

Think about evidence volatility.

A useful high-level order is:

Current Time
      ↓
Logged-In Users
      ↓
Running Processes
      ↓
Network Connections
      ↓
Open / Runtime Information
      ↓
Services
      ↓
Filesystem Metadata
      ↓
Logs
      ↓
Persistent Configuration

Exact evidence-collection order should follow organizational DFIR procedures.

Phase 01 — Receive the Alert

Start with the alert itself.

Document:

Alert Name:

Alert Time:

Detection Source:

Hostname:

IP Address:

User:

Process:

Destination:

Severity:

Alert Identifier:

Initial Questions

Ask:

What triggered the alert?

Which system generated it?

Which asset is affected?

How critical is the asset?

Which identity is involved?

Is activity still occurring?

Is the alert based on one event or many?

Are other systems generating similar alerts?

Phase 02 — Confirm Scope and Authorization

Before interacting with the system, confirm:

Host Is In Scope

Investigation Is Authorized

Permitted Actions Are Known

Evidence Requirements Are Known

Escalation Contacts Are Known

Production Safety

On production Linux servers, some actions may affect:

Availability

Application Performance

Evidence Integrity

Customer Services

Use approved procedures.

Phase 03 — Start the Incident Record

Create an investigation directory in your authorized environment:

mkdir -p ~/linux-incident-investigation/{system,identity,process,network,services,logs,persistence,files,timeline,reports}

Move into it:

cd ~/linux-incident-investigation

Incident Notes

Create:

Incident ID:

Analyst:

Start Time:

Affected Host:

Business Owner:

Detection Source:

Initial Alert:

Current Status:

Maintain a record of:

What You Did

When You Did It

Why You Did It

What You Observed

Phase 04 — Record System Time

Capture:

date

Then:

date -u

Review:

timedatectl

Save where appropriate:

timedatectl > system/time-status.txt

Why This Matters

You may later correlate:

Linux Logs

SIEM

Firewall

Cloud Logs

Identity Provider

Application Logs

Time consistency is essential.

Phase 05 — Identify the Host

Collect:

hostname

hostnamectl

cat /etc/os-release

uname -a

Record:

Hostname

Distribution

OS Version

Kernel

Architecture

Virtual / Physical / Cloud Context

Asset Context

Determine:

What Does This Server Do?

Who Owns It?

Is It Production?

What Data Does It Process?

Is It Internet-Facing?

What Is Its Criticality?

Technical severity and business severity may differ.

Phase 06 — Review Uptime

Run:

uptime

Record:

Current Time

Uptime

Load

Investigation Value

Unexpected recent reboot may be relevant.

But:

Recent Reboot

does not automatically mean:

Attacker Rebooted Server

Correlate with:

Maintenance

Patching

Administrator Activity

Cloud Events

Phase 07 — Identify Current User

Run:

whoami

Then:

id

Document the identity under which evidence is being collected.

This supports:

Accountability

Evidence Interpretation

Audit Trail

Phase 08 — Review Logged-In Users

Run:

who

Then:

Review:

Username

Terminal

Login Source

Login Time

Current Activity

Investigation Questions

Are These Users Expected?

Are Sources Expected?

Is a Privileged User Logged In?

Does Login Time Match the Alert?

Is an Unknown Session Active?

Run where appropriate:

last

Review:

Username

Source

Login Time

Session Duration

Reboot Events

Authentication Timeline

You may discover:

01:55 Failed Login

02:01 Failed Login

02:06 Successful Login

02:08 sudo Activity

02:11 New Process

02:14 External Connection

The relationship between events is often more important than any single event.

Phase 10 — Review Authentication Failures

Relevant evidence may exist in:

systemd Journal

/var/log/auth.log

/var/log/secure

depending on distribution.

Search for patterns rather than blindly copying entire sensitive logs.

Questions

Which Account Was Targeted?

Which Source Generated Failures?

How Many Failures?

Was There a Later Success?

Was the Account Privileged?

Was the Source Expected?

Suspicious Pattern Example

Repeated Authentication Failures
        ↓
Successful Authentication
        ↓
Privilege Escalation
        ↓
New Process

This warrants deeper investigation.

Phase 11 — Review Privilege Activity

Investigate:

sudo

su

Administrative Sessions

UID 0 Activity

Your objective:

WHO
   ↓
Gained Which Privilege
   ↓
WHEN
   ↓
Performed What Action

Privilege Questions

Was Privilege Expected?

Was It Approved?

Was It Used by the Logged-In User?

Did It Occur Immediately Before Suspicious Activity?

Phase 12 — Review Users

Collect an account inventory:

getent passwd

Identify UID 0 identities:

awk -F: '$3 == 0 {print $1 ":" $3 ":" $7}' /etc/passwd

Look for

Unexpected User

Recently Created User

Unexpected UID 0

Unexpected Login Shell

Unexpected Service Account

Changed Group Membership

Important

An unfamiliar account may belong to:

Application

Monitoring Agent

Backup Software

Automation

Vendor Product

Validate before classifying.

Phase 13 — Review Privileged Groups

Review:

getent group

Pay attention to administrative or security-sensitive groups appropriate to the distribution.

Compare:

Expected Membership

against:

Actual Membership

Potential Finding

Unexpected Privileged Group Membership

may indicate:

Misconfiguration

Privilege Creep

Unauthorized Change

Potential Persistence

Phase 14 — Review Running Processes

Run:

ps aux

Then:

ps -ef

Capture:

ps -ef > process/process-list.txt

Process Investigation Questions

For suspicious processes determine:

PID

PPID

User

Start Time

Command

Executable

Parent

Network Activity

Phase 15 — Identify High Resource Usage

Run:

ps aux --sort=-%cpu | head -20

Then:

ps aux --sort=-%mem | head -20

High CPU or memory may indicate:

Normal Workload

Application Failure

Runaway Process

Unexpected Software

It is a lead, not a verdict.

Phase 16 — Investigate a Suspicious PID

Suppose the alert identifies:

PID 1234

Use:

ps -fp 1234

Then:

ps -o pid,ppid,user,lstart,cmd -p 1234

Investigation Chain

PID
 ↓
USER
 ↓
PPID
 ↓
PARENT
 ↓
EXECUTABLE
 ↓
START TIME
 ↓
NETWORK

Phase 17 — Identify the Executable

For an authorized running process:

readlink -f /proc/1234/exe

Review command-line information:

tr '\0' ' ' < /proc/1234/cmdline

Be careful when collecting process data because arguments may contain:

Passwords

Tokens

Secrets

API Keys

Minimize sensitive evidence.

Phase 18 — Review Parent Process

If:

Suspicious PID = 1234

and:

PPID = 900

investigate:

ps -fp 900

Why Parent Processes Matter

Suppose:

sshd
  ↓
shell
  ↓
unexpected process

This provides different context than:

systemd
  ↓
approved service

Process ancestry helps reconstruct execution.

Phase 19 — Build the Process Tree

Where available:

pstree -p

Look for relationships such as:

sshd
 └── shell
      └── process

or:

systemd
 └── service
      └── worker

Phase 20 — Review Deleted Executables

Linux may continue running a process even after its executable file has been deleted.

Where authorized, investigate process executable links and open-file evidence.

A deleted executable associated with an active process deserves attention but is not automatically malicious.

Legitimate software upgrades can also produce this condition.

Phase 21 — Review Open Files

Where available:

sudo lsof -p 1234

This may show:

Executable

Libraries

Configuration

Files

Network Sockets

Treat output carefully because it may expose sensitive file paths or data relationships.

Phase 22 — Review Network Interfaces

Run:

ip -brief address

Then:

ip route

Document:

Interfaces

Addresses

Routes

Default Gateway

This establishes the network context for the incident.

Phase 23 — Review Listening Ports

Run:

sudo ss -lntup

Look for:

Unexpected Ports

Unexpected Bind Addresses

Unknown Processes

Administrative Services

Recently Introduced Services

Investigation Workflow

PORT
 ↓
PROCESS
 ↓
PID
 ↓
USER
 ↓
EXECUTABLE
 ↓
SERVICE
 ↓
BUSINESS REQUIREMENT

Phase 24 — Review Active Connections

Run:

sudo ss -ntp

Review:

Local Address

Local Port

Remote Address

Remote Port

State

PID

Process

Network Investigation Questions

Is the Destination Expected?

Does the Process Normally Communicate There?

Does the Port Match the Application?

When Did Communication Begin?

Are Other Hosts Contacting the Same Destination?

Phase 25 — Correlate Process and Network

Suppose you identify:

External IP
   ↓
Connection
   ↓
PID 1234
   ↓
Process X
   ↓
User alice

Now ask:

How Did Alice Authenticate?

How Did Process X Start?

What Is Its Parent?

What Files Did It Access?

Does It Persist?

This is how investigation expands from an alert into an incident story.

Phase 26 — Review DNS Context

Where relevant, investigate:

Destination Hostname

DNS Queries

Resolver Configuration

Application Logs

Central DNS Logs

Do not rely exclusively on current DNS resolution because mappings can change.

Historical DNS telemetry may provide stronger incident context.

Phase 27 — Review Firewall State

Determine the host firewall technology.

Examples:

sudo firewall-cmd --list-all

or:

sudo ufw status verbose

or:

sudo nft list ruleset

Questions

Was Suspicious Traffic Allowed?

Should It Have Been Allowed?

Were Firewall Rules Changed?

Is Administrative Access Overly Broad?

Phase 28 — Review Running Services

Run:

systemctl --type=service --state=running

Review:

Known Services

Unknown Services

Recently Introduced Services

Privileged Services

Phase 29 — Review Enabled Services

Run:

systemctl list-unit-files --type=service --state=enabled

An unexpected enabled service can be significant because it may:

Restart Automatically
After Reboot

Phase 30 — Investigate Suspicious Service

For an authorized service:

systemctl status <service>

Then:

systemctl cat <service>

Review:

Executable

User

Arguments

Dependencies

Restart Behavior

Environment

Configuration

Do not expose secrets contained in environment or configuration data.

Phase 31 — Persistence Investigation

Review common Linux mechanisms that can cause execution to continue across sessions or reboots.

Think:

SYSTEMD
   ↓
SCHEDULED TASKS
   ↓
SSH KEYS
   ↓
ACCOUNTS
   ↓
SHELL STARTUP
   ↓
APPLICATION STARTUP

Persistence Does Not Equal Malware

These mechanisms are used extensively by legitimate software.

Your question is:

Is This Mechanism Expected?

Phase 32 — Review Cron

For your authorized account:

crontab -l

Review appropriate system scheduling locations.

For another authorized account:

sudo crontab -u <user> -l

Cron Questions

Which User?

Which Command?

Which Script?

Who Owns the Script?

Can It Be Modified by Others?

When Was It Added?

Is It Expected?

Phase 33 — Review systemd Timers

Run:

systemctl list-timers --all

Investigate unexpected timers.

A timer may launch:

Maintenance

Backups

Updates

Monitoring

Custom Scripts

or unauthorized execution.

Phase 34 — Review SSH Authorized Keys

For authorized accounts, review:

~/.ssh/authorized_keys

Focus on:

Unknown Key

Recently Added Key

Privileged Account Key

Shared Key

Key Without Owner

Never collect private keys into routine incident notes.

Phase 35 — Review Shell Startup Configuration

Potential user startup files include:

.profile

.bash_profile

.bashrc

depending on the shell and distribution.

Investigate:

Unexpected Commands

External Connections

Unexpected Executables

Recent Changes

Phase 36 — Review systemd Unit Files

Potential locations include:

/etc/systemd/system/

/usr/lib/systemd/system/

/lib/systemd/system/

depending on the distribution.

Pay particular attention to:

Custom Units

Recent Changes

Unexpected Executables

Unexpected Users

Phase 37 — Review Temporary Locations

Review relevant metadata in:

/tmp

/var/tmp

Look for:

Unexpected Executables

Recent Files

Unexpected Ownership

Files Related to Suspicious Processes

Do not delete potential evidence during collection.

Phase 38 — Review Filesystem Changes

For targeted locations, review recent modification times.

Example:

find /etc -type f -mtime -2 -print 2>/dev/null

Also consider:

Application Directories

User Home

Service Configuration

Scheduled Tasks

Important

Timestamp interpretation requires care.

Timestamps can be affected by:

Normal Administration

Package Updates

Deployment

File Copying

Backup/Restore

System Behavior

Phase 39 — Review File Metadata

For a suspicious file:

stat <file>

Record:

Path

Owner

Group

Permissions

Size

Timestamps

Evidence Rule

Prefer:

Metadata First

before modifying the file.

Phase 40 — Hash Relevant Files

For an authorized suspicious file:

sha256sum <file>

Record:

SHA-256

File Path

Collection Time

Why Hashes Matter

A hash helps identify:

Which Exact File
Was Examined

and can support:

Integrity Verification

Threat Intelligence Correlation

Case Tracking

A hash alone does not determine whether a file is malicious.

Phase 41 — Review Package Ownership

If the suspicious file appears to belong to installed software, determine whether the package manager recognizes it.

On RPM-based systems, package-management queries can help.

On Debian-based systems, package-management queries can help.

Your objective:

File
 ↓
Package?
 ↓
Approved Software?
 ↓
Expected Version?

Phase 42 — Review Software Changes

Check package-management history using the facilities available on your distribution.

Questions:

Was Software Installed Recently?

Who Installed It?

Was It Approved?

Does the Time Match the Incident?

Did Installation Create a Service?

Phase 43 — Review Logs

Relevant sources may include:

System Journal

Authentication Logs

Application Logs

Web Logs

Firewall Logs

Audit Logs

Package Logs

Cloud Logs

Log Strategy

Do not search randomly.

Start from:

Known Alert Time

Then expand:

Before Alert

During Alert

After Alert

Phase 44 — Query the Journal

Review recent events:

journalctl --since "1 hour ago"

Or use an approved incident-specific time range.

For a service:

journalctl -u <service>

Investigation Approach

Known Timestamp
      ↓
Known User
      ↓
Known Process
      ↓
Known Service
      ↓
Related Events

Phase 45 — Review Kernel Events

Where relevant:

journalctl -k

Kernel evidence may include:

Device Events

Network Events

Security Control Messages

Resource Problems

Phase 46 — Review SELinux Evidence

On SELinux systems:

getenforce

Review relevant security denials using approved platform tools.

Security denials can help identify:

Unexpected Application Behavior

Incorrect Context

Blocked Unauthorized Activity

Do not automatically disable SELinux.

Phase 47 — Review AppArmor Evidence

On AppArmor-based systems, review:

Profile Status

Relevant Denials

Affected Application

Mandatory access-control evidence may help explain what an application attempted to do.

Phase 48 — Review Audit Evidence

If Linux auditing is configured:

systemctl status auditd

Where authorized:

sudo auditctl -l

Audit data may help answer:

Who Changed a File?

Who Changed an Account?

Which Process Performed an Action?

Which Security-Relevant Event Occurred?

Phase 49 — Build an Event Timeline

Create:

timeline/incident-timeline.md

Use a structure such as:

Time	Event	User	Process	Source	Evidence
01:55	Login failures	alice	sshd	External IP	Auth
02:06	Login success	alice	sshd	External IP	Auth
02:08	sudo used	alice	sudo	Local	Auth
02:11	Process started	root	process	Local	Process
02:14	External connection	root	process	External IP	Network

Timeline Principle

A useful incident timeline is:

Evidence-Based

Chronological

Source-Referenced

Timezone-Aware

Phase 50 — Correlate Events

Do not analyze events in isolation.

Example:

Login
  +
sudo
  +
New File
  +
New Service
  +
Outbound Connection
        ↓
Potential Incident Chain

Correlation Questions

Did the Same User Perform the Actions?

Did Events Occur Close Together?

Does the Parent Process Match the Login?

Did the New Service Execute the File?

Did the File Initiate the Connection?

Phase 51 — Develop Hypotheses

Examples:

Hypothesis A:
Legitimate Administrator Activity

Hypothesis B:
Application Deployment

Hypothesis C:
Compromised User Account

Hypothesis D:
Unauthorized Software Execution

Then test each hypothesis against evidence.

Good Investigation Practice

Do not search only for evidence supporting your first theory.

Also ask:

What Evidence Would Disprove
My Hypothesis?

Phase 52 — Determine Initial Access

Where evidence permits, determine how suspicious activity began.

Possible categories include:

Valid Credentials

Vulnerable Service

Application Weakness

Misconfiguration

Exposed Administrative Access

Software Supply Chain

Existing Internal Access

Do not claim an initial-access mechanism without supporting evidence.

Phase 53 — Determine Privilege Escalation

Ask:

Did Activity Remain Under
the Original Account?

or:

Did It Gain Additional Privilege?

Review:

sudo

su

UID Changes

Privileged Services

Account Changes

Security Logs

Phase 54 — Determine Persistence

Ask:

Would Access Survive
Logout or Reboot?

Review:

New Users

SSH Keys

Services

Timers

Cron

Startup Files

Application Startup

Phase 55 — Determine Lateral Movement

Ask whether the affected Linux host communicated with other internal systems.

Review:

SSH Activity

Application Connections

Internal Destinations

Authentication Logs

Network Telemetry

Cloud Flow Logs

Scope Question

Is This:
One Host
or
Multiple Hosts?

This distinction can radically change incident severity.

Phase 56 — Determine External Communication

Review:

External IPs

Domains

Ports

Processes

Connection Frequency

Transferred Data Where Observable

Ask:

Expected Vendor?

Package Repository?

Monitoring Platform?

Backup Provider?

Unknown Destination?

Phase 57 — Determine Data Access

Where relevant, assess:

Which Files Were Accessed?

Which Databases?

Which Credentials?

Which Application Data?

Which Secrets?

Do not claim:

Data Exfiltration

simply because an outbound connection exists.

Distinguish:

Possible Access

Observed Access

Confirmed Transfer

Phase 58 — Determine Impact

Evaluate:

CONFIDENTIALITY

INTEGRITY

AVAILABILITY

Questions:

Was Sensitive Data Accessed?

Was Configuration Modified?

Was Service Availability Affected?

Were Credentials Exposed?

Was Persistence Established?

Phase 59 — Determine Incident Scope

Build a scope table:

Asset	Evidence	Status	Priority
Linux Server A	Confirmed activity	Affected	High
Linux Server B	Same indicator	Investigate	High
Database	Connection observed	Validate	High
Admin Account	Suspicious login	Investigate	High

Scope Expansion

Search across:

SIEM

EDR

Firewall

Identity Logs

Cloud Logs

DNS

Proxy

Other Linux Hosts

for confirmed indicators.

Phase 60 — Classify Evidence

Use categories such as:

Confirmed

Probable

Possible

Unknown

Example:

Confirmed:
Process contacted external IP.

Probable:
Process was started from suspicious session.

Possible:
Credentials may have been exposed.

Unknown:
Whether sensitive data left the environment.

This prevents overstatement.

Phase 61 — Decide Whether Containment Is Required

Containment depends on:

Incident Severity

Business Impact

Active Threat

Evidence Requirements

System Criticality

Available Redundancy

Potential Containment Options

Depending on organizational procedures:

Restrict Network Access

Disable Compromised Identity

Block Confirmed Malicious Destination

Stop Malicious Service

Isolate Host

Revoke Credentials

Remove Temporary Access

Important

Containment actions can:

Disrupt Business

Alert an Adversary

Destroy Volatile Evidence

Change the Incident State

Coordinate with the incident-response lead.

Phase 62 — Short-Term vs Long-Term Containment

Short-Term

Stop Immediate Threat

Examples:

Isolate Host

Disable Account

Block Confirmed Indicator

Long-Term

Restore Controlled Operations

Examples:

Network Segmentation

Credential Rotation

Security Policy Changes

Temporary Hardened Replacement

Phase 63 — Credential Containment

If credentials are confirmed or reasonably suspected to be compromised, follow approved identity-response procedures.

Consider:

Password

SSH Keys

API Tokens

Cloud Credentials

Application Secrets

Service Credentials

Do not focus only on the Linux password.

Phase 64 — Cloud Linux Incident

If the affected host is a cloud VM, expand the investigation.

AWS Example

Review:

CloudTrail

EC2 Metadata/Role Context

Security Groups

VPC Flow Logs

IAM Activity

Snapshots

CloudWatch

Azure Example

Review:

Azure Activity Logs

Entra Sign-In Logs

NSGs

Network Logs

Managed Identity

Defender Telemetry

Google Cloud Example

Review:

Cloud Audit Logs

IAM

VPC Flow Logs

Firewall Logs

Service Accounts

Security Command Center

Cloud Incident Model

CLOUD CONTROL PLANE
        +
LINUX OPERATING SYSTEM
        =
COMPLETE INVESTIGATION

Phase 65 — Containerized Linux Workloads

If containers are involved, expand to:

Container

Image

Runtime

Host

Registry

Orchestrator

Ask:

Was the Process Inside a Container?

Which Image?

Which User?

Which Mounts?

Which Network?

Did Activity Reach the Host?

Phase 66 — Kubernetes Context

For Kubernetes workloads:

Alert
  ↓
Pod
  ↓
Container
  ↓
Node
  ↓
Service Account
  ↓
Kubernetes Audit
  ↓
Cloud IAM

Do not investigate only the Linux node if orchestration evidence is available.

Phase 67 — Eradication

After containment and evidence collection, remove confirmed incident artifacts according to the approved plan.

This may include:

Unauthorized Account

Unauthorized SSH Key

Malicious File

Malicious Service

Scheduled Persistence

Compromised Package

Unsafe Configuration

Important

Do not confuse:

Delete Artifact

with:

Remove Root Cause

If the original access path remains open, the incident can recur.

Phase 68 — Root Cause

Ask:

Why Was This Possible?

Examples:

Exposed SSH

Compromised Credentials

Excessive sudo

Missing Patch

Weak Application Configuration

Poor Network Segmentation

Unmanaged SSH Key

Missing MFA Upstream

Overly Broad Cloud IAM

Root Cause vs Symptom

Suspicious Process
=
Symptom

Compromised Credential
=
Potential Cause

Remediation should address both.

Phase 69 — Recovery

Recovery may include:

Restore Service

Rebuild Host

Restore Clean Data

Apply Patches

Rotate Credentials

Restore Security Controls

Validate Configuration

Increase Monitoring

Rebuild Decision

For serious compromise, organizations may prefer:

Known-Good Rebuild

over:

Attempt to Clean
Unknown Compromise

The decision depends on evidence, severity, system architecture, and organizational policy.

Phase 70 — Validate Recovery

Before returning the system to normal operation, verify:

Unauthorized Access Removed

Credentials Rotated

Persistence Removed

Security Controls Active

Required Services Healthy

Network Exposure Correct

Logging Working

Monitoring Active

Phase 71 — Heightened Monitoring

After recovery, temporarily increase monitoring for:

Repeated Indicators

Authentication Attempts

New Processes

Unexpected Services

Outbound Connections

Configuration Changes

Recovery does not end the investigation immediately.

Phase 72 — Document Indicators

Document confirmed indicators such as:

IP Addresses

Domains

File Hashes

File Paths

Process Names

Usernames

Service Names

Clearly label whether each indicator is:

Confirmed Malicious

Suspicious

Contextual

Benign

Phase 73 — Document Findings

Use:

Finding Title

Observation

Evidence

Risk

Impact

Recommendation

Priority

Status

Finding Example — Suspicious Account Activity

Finding:
Suspicious Administrative Authentication

Observation:
An administrative account successfully
authenticated from a source not associated
with normal administrative activity.

Evidence:
Authentication logs and centralized
security telemetry correlate the login
with subsequent privileged activity.

Risk:
Unauthorized use of administrative
credentials could permit system-level
changes and access to sensitive resources.

Recommendation:
Validate account ownership, revoke active
sessions where appropriate, rotate affected
credentials, review privilege assignments,
and investigate related activity.

Finding Example — Unauthorized Persistence

Finding:
Unauthorized Persistent Service

Observation:
A previously undocumented systemd service
was configured to start automatically and
execute an unapproved binary.

Risk:
The service could provide continued
execution after reboot and maintain
unauthorized access.

Recommendation:
Preserve required evidence, identify the
installation source and related activity,
remove the service through the approved
incident process, and address the original
access mechanism.

Finding Example — Insufficient Logging

Finding:
Insufficient Linux Security Telemetry

Observation:
Authentication and process activity could
not be fully reconstructed because required
security events were not centrally retained.

Risk:
Limited telemetry reduces detection,
investigation, and incident reconstruction
capabilities.

Recommendation:
Implement centralized security logging,
appropriate audit controls, time
synchronization, retention, and monitoring.

Phase 74 — Write the Incident Report

Your report should contain:

1. Executive Summary

Explain:

What Happened

Affected Assets

Business Impact

Current Status

2. Detection

Document:

Detection Source

Alert

Timestamp

Initial Indicator

3. Scope

Document:

Affected Hosts

Affected Accounts

Affected Applications

Affected Data

4. Investigation

Describe:

Authentication

Privilege

Processes

Network

Files

Persistence

Logs

5. Timeline

Provide the evidence-based chronology.

6. Indicators

Document relevant:

IPs

Domains

Hashes

Accounts

Processes

Services

7. Impact

Assess:

Confidentiality

Integrity

Availability

8. Containment

Document:

Action

Time

Owner

Reason

Result

9. Eradication

Document what was removed or corrected.

10. Recovery

Document restoration and validation.

11. Root Cause

State only what the evidence supports.

12. Recommendations

Prioritize:

Immediate

Short-Term

Long-Term

Phase 75 — Incident Timeline Template

Incident ID:

Timezone:

--------------------------------------------------

Time:
Event:
User:
Host:
Process:
Source:
Evidence:
Analyst Notes:

--------------------------------------------------

Phase 76 — Analyst Investigation Notes

Use:

Timestamp

Action Performed

Command / Tool

Observation

Evidence Location

Interpretation

Next Step

Separate:

Observation

from:

Interpretation

Example:

Observation:
TCP connection exists from PID 1234
to external address.

Interpretation:
Connection is not present in documented
application baseline and requires further
investigation.

Phase 77 — Chain of Custody

For formal investigations, evidence handling may require:

Evidence Identifier

Collector

Collection Time

Source

Hash

Storage Location

Transfers

Access History

Follow organizational and legal requirements.

Phase 78 — Escalation Criteria

Escalate when evidence indicates or strongly suggests:

Root-Level Compromise

Credential Theft

Multiple Hosts

Sensitive Data Exposure

Active External Communication

Persistence

Production Impact

Regulated Data

Cloud Control-Plane Compromise

Phase 79 — Communication

Incident communication should be:

Accurate

Evidence-Based

Timely

Need-to-Know

Free of Unsupported Conclusions

Avoid statements like:

"The attacker stole all data."

when the evidence only shows:

"An unauthorized process established
an outbound connection."

Phase 80 — Incident Severity

Severity should consider:

Asset Criticality

Privilege Level

Scope

Data Sensitivity

Persistence

External Communication

Business Impact

Active Threat

Phase 81 — False Positive Handling

If investigation determines activity is legitimate:

Document Evidence
      ↓
Identify Business Process
      ↓
Close Incident
      ↓
Tune Detection if Appropriate

Do not simply write:

False Positive

without evidence.

Phase 82 — Lessons Learned

After closure ask:

Why Did Detection Work?

What Detection Was Missing?

Was Evidence Sufficient?

Was Containment Fast Enough?

Were Contacts Clear?

Was the Asset Baseline Accurate?

Could the Incident Have Been Prevented?

Phase 83 — Improve Security Controls

Lessons may lead to improvements in:

Linux Hardening

IAM

SSH

Patching

Firewall

Segmentation

Logging

Audit

EDR

SIEM

Cloud IAM

Monitoring

Phase 84 — Detection Engineering

Turn investigation knowledge into detection opportunities.

Examples:

New UID 0 Account

Unexpected sudo Change

New SSH Key

New Enabled Service

Unexpected External Connection

New Listening Port

Security Control Disabled

The goal is:

Incident
   ↓
Knowledge
   ↓
Detection Improvement

Phase 85 — Investigation Decision Tree

ALERT RECEIVED
      |
      v
VALID?
  |       |
 NO      YES
  |       |
Close     v
      COLLECT CONTEXT
          |
          v
     ACTIVE THREAT?
       |       |
      NO      YES
       |       |
       |    Evaluate
       |   Containment
       |       |
       +---+---+
           |
           v
      INVESTIGATE
           |
           v
      BUILD TIMELINE
           |
           v
      DETERMINE SCOPE
           |
           v
      ROOT CAUSE
           |
           v
       REMEDIATE
           |
           v
        RECOVER
           |
           v
       VALIDATE
           |
           v
        REPORT

Phase 86 — Quick Linux Triage Checklist

System

Identity

Processes

Network

Services

Review running services
Review enabled services
Investigate unknown services
Review service configuration

Persistence

Files

Logs

Scope

Determine affected users
Determine affected hosts
Determine affected applications
Determine external destinations
Determine potential data impact
Search confirmed indicators

Response

Common Investigation Mistakes

Avoid:

Rebooting Too Early

Killing Processes Before Collection

Deleting Suspicious Files Immediately

Clearing Logs

Assuming Unfamiliar = Malicious

Ignoring Process Parents

Ignoring Outbound Connections

Ignoring SSH Keys

Ignoring Service Accounts

Ignoring Scheduled Tasks

Ignoring Cloud Evidence

Ignoring Timezone Differences

Focusing Only on One Indicator

Stopping After Containment

Confusing Symptom with Root Cause

Overstating Data Loss

Failing to Document Actions

Linux Investigation Mental Model

Memorize:

USER
  ↓
PROCESS
  ↓
PARENT
  ↓
FILE
  ↓
SERVICE
  ↓
NETWORK
  ↓
PERSISTENCE
  ↓
LOGS
  ↓
TIMELINE

Investigation Questions

Every Linux incident should eventually answer:

WHO?

WHAT?

WHEN?

WHERE?

HOW?

WHY?

HOW FAR?

WHAT IMPACT?

WHAT REMAINS?

Interview Scenario 01

A SOC alert reports an unexpected outbound connection from a Linux server. What do you investigate first?

Start by validating:

Host

Connection

PID

Process

User

Then expand into:

Parent

Executable

Authentication

Persistence

Timeline

Interview Scenario 02

You identify a suspicious process. Should you kill it immediately?

Not automatically.

First consider:

Active Risk

Evidence Preservation

Process Information

Connections

Parent Process

Incident Procedure

Immediate containment may still be necessary for a high-risk active incident.

Interview Scenario 03

Why is PPID useful?

Because it helps identify:

How the Process
Was Started

and supports reconstruction of execution relationships.

Interview Scenario 04

Why review SSH authorized keys?

Because unauthorized keys may provide:

Persistent Remote Access

even after a password is changed.

Interview Scenario 05

Why isn’t an outbound connection proof of data exfiltration?

Because it proves communication occurred, not necessarily:

Which Data

How Much Data

Whether Sensitive Data

Was Transferred

Additional evidence is required.

40 Linux Incident Response Interview Questions

What is Linux incident triage?
What should you record when an alert arrives?
Why is system time important?
Why is evidence volatility important?
Why should you avoid rebooting immediately?
How do you identify logged-in users?
How do you review login history?
How would you investigate authentication failures?
How do you identify UID 0 accounts?
Why should privileged groups be reviewed?
How do you capture running processes?
Why is PPID important?
How do you identify a process executable?
What can /proc tell you?
Why should process arguments be handled carefully?
How do you identify listening ports?
How do you map connections to processes?
How do you investigate an unexpected external connection?
Why review the host firewall?
How do you identify running services?
Why review enabled services?
What Linux mechanisms can provide persistence?
Why review cron?
Why review systemd timers?
Why review SSH authorized keys?
Why review startup files?
Why are file timestamps useful?
Why calculate file hashes?
What is event correlation?
What is an incident timeline?
How do you determine initial access?
How do you investigate privilege escalation?
How do you determine incident scope?
What is containment?
What is eradication?
What is recovery?
What is root cause?
Why should cloud logs be reviewed for cloud Linux incidents?
What should an incident report contain?
What are the most common Linux incident-investigation mistakes?

Runbook Completion Criteria

The investigation is not complete merely because:

Alert Stopped

You should be able to explain:

What Happened

Which Assets Were Affected

Which Accounts Were Involved

Which Processes Were Involved

Which Connections Occurred

Whether Persistence Existed

How Access Was Obtained Where Known

What Impact Occurred

What Was Contained

What Was Remediated

How Recovery Was Validated

Runbook Success Criteria

A successful Linux investigation produces:

Validated Alert

Preserved Evidence

Identity Analysis

Process Analysis

Network Analysis

Persistence Analysis

Event Timeline

Incident Scope

Impact Assessment

Containment Decision

Root-Cause Analysis

Recovery Validation

Incident Report

Final Operational Workflow

DETECT
  ↓
VALIDATE
  ↓
PRESERVE
  ↓
IDENTIFY
  ↓
TRIAGE
  ↓
INVESTIGATE
  ↓
CORRELATE
  ↓
TIMELINE
  ↓
SCOPE
  ↓
CONTAIN
  ↓
ERADICATE
  ↓
RECOVER
  ↓
VALIDATE
  ↓
MONITOR
  ↓
REPORT
  ↓
IMPROVE

Runbook Complete

You now have a repeatable procedure for investigating suspicious activity on Linux systems.

The key transition is from:

"I Found Something Suspicious"

to:

I Can Explain:

What Happened

When It Happened

Which Identity Was Involved

Which Process Was Involved

How It Communicated

Whether It Persisted

How Far It Spread

What Evidence Supports the Conclusion

What Response Is Required

This is the difference between simply operating security tools and performing professional incident investigation.

What’s Next?

➡️ Runbook 02 — Linux Security Assessment

In the next runbook, you will move from:

Reactive Investigation

to:

Proactive Security Assessment

You will build a repeatable procedure for assessing:

Asset Context
      ↓
System Configuration
      ↓
Identity and Access
      ↓
Privileged Access
      ↓
Authentication
      ↓
Filesystem Security
      ↓
Services
      ↓
Network Exposure
      ↓
Patching
      ↓
Security Controls
      ↓
Logging and Monitoring
      ↓
Persistence Risk
      ↓
Backup and Recovery
      ↓
Findings
      ↓
Risk Prioritization
      ↓
Remediation Roadmap

Your Linux operational-security sequence is:

Runbook 01 — Linux Incident Investigation
        ↓
Runbook 02 — Linux Security Assessment
        ↓
Runbook 03 — Linux Server Hardening

Runbook 01 — Linux Incident Investigation

Runbook Information

Mission Scenario

Incident Investigation Philosophy

Core Principle

Golden Rule — Do Not Destroy Evidence

Investigation Priorities

Phase 01 — Receive the Alert

Initial Questions

Phase 02 — Confirm Scope and Authorization

Production Safety

Phase 03 — Start the Incident Record

Incident Notes

Phase 04 — Record System Time

Why This Matters

Phase 05 — Identify the Host

Asset Context

Phase 06 — Review Uptime

Investigation Value

Phase 07 — Identify Current User

Phase 08 — Review Logged-In Users

Investigation Questions

Phase 09 — Review Login History

Authentication Timeline

Phase 10 — Review Authentication Failures

Questions

Suspicious Pattern Example

Phase 11 — Review Privilege Activity

Privilege Questions

Phase 12 — Review Users

Look for

Important

Phase 13 — Review Privileged Groups

Potential Finding

Phase 14 — Review Running Processes

Process Investigation Questions

Phase 15 — Identify High Resource Usage

Phase 16 — Investigate a Suspicious PID

Investigation Chain

Phase 17 — Identify the Executable

Phase 18 — Review Parent Process

Why Parent Processes Matter

Phase 19 — Build the Process Tree

Phase 20 — Review Deleted Executables

Phase 21 — Review Open Files

Phase 22 — Review Network Interfaces

Phase 23 — Review Listening Ports

Investigation Workflow

Phase 24 — Review Active Connections

Network Investigation Questions

Phase 25 — Correlate Process and Network

Phase 26 — Review DNS Context

Phase 27 — Review Firewall State

Questions

Phase 28 — Review Running Services

Phase 29 — Review Enabled Services

Phase 30 — Investigate Suspicious Service

Phase 31 — Persistence Investigation

Persistence Does Not Equal Malware

Phase 32 — Review Cron

Cron Questions

Phase 33 — Review systemd Timers

Phase 34 — Review SSH Authorized Keys

Phase 35 — Review Shell Startup Configuration

Phase 36 — Review systemd Unit Files

Phase 37 — Review Temporary Locations

Phase 38 — Review Filesystem Changes

Important

Phase 39 — Review File Metadata

Evidence Rule

Phase 40 — Hash Relevant Files

Why Hashes Matter

Phase 41 — Review Package Ownership

Phase 42 — Review Software Changes

Phase 43 — Review Logs

Log Strategy

Phase 44 — Query the Journal

Investigation Approach

Phase 45 — Review Kernel Events

Phase 46 — Review SELinux Evidence