Lab 05 — Linux Security
Welcome to the final hands-on lab in the Linux lab series.
You have already worked through:
Lab 01 — Linux Administration ↓Lab 02 — Linux Hardening ↓Lab 03 — Linux IAM ↓Lab 04 — Linux NetworkingNow you will bring those skills together into:
Lab 05 — Linux SecurityThis lab changes your perspective from:
How Do I AdministerThis Linux Server?to:
Is This Linux Server Secure?
What Is Exposed?
Who Has Access?
What Is Running?
What Has Changed?
What Looks Suspicious?
Can We Investigate It?
How Should We Remediate It?Mission Information
Section titled “Mission Information”Lab: Linux Security
Level: Intermediate
Estimated Time: 150–210 minutes
Environment: Authorized disposable Linux VM
Primary Role: Linux Security Engineer
Secondary Roles: SOC Analyst, Incident Responder, Cloud Security Engineer, Security Consultant, DevSecOps Engineer
Mission Scenario
Section titled “Mission Scenario”Your organization is preparing an important Linux application server for a security review.
The server is operational, but the security team wants an independent assessment before approving it for production.
You have been assigned to review:
Operating System ↓Accounts ↓Privileges ↓Authentication ↓Filesystem ↓Processes ↓Services ↓Software ↓Network ↓Firewall ↓Logging ↓Auditing ↓Scheduled Execution ↓Security Controls ↓Suspicious ActivityYou must distinguish:
Expectedfrom:
Unexpectedand convert technical observations into professional security findings.
Mission Objectives
Section titled “Mission Objectives”By completing this lab, you should be able to:
- Establish a Linux security baseline
- Identify the operating system and kernel
- Review user and service identities
- Identify privileged accounts
- Review groups and sudo access
- Evaluate authentication exposure
- Review SSH security
- Assess filesystem permissions
- Identify world-writable resources
- Review SUID and SGID executables
- Identify unusual processes
- Review running and enabled services
- Assess installed software
- Review patch status
- Identify network listeners
- Investigate active connections
- Review host firewall controls
- Evaluate SELinux or AppArmor
- Review system and authentication logs
- Review Linux auditing
- Investigate scheduled execution
- Identify basic persistence indicators
- Build a security timeline
- Document findings and remediation
- Produce a Linux security assessment report
Lab Architecture
Section titled “Lab Architecture”+----------------------------------+| Security Analyst |+----------------+-----------------+ | | SSH / Console | v+----------------------------------+| Linux Server || || Identity Processes || Privilege Services || Filesystem Packages || Network Firewall || SSH Logs || Audit Persistence || SELinux / AppArmor |+----------------+-----------------+ | v+----------------------------------+| Security Findings & Evidence |+----------------------------------+Security Assessment Mental Model
Section titled “Security Assessment Mental Model”Use:
ASSET ↓IDENTITY ↓PRIVILEGE ↓SOFTWARE ↓PROCESS ↓SERVICE ↓NETWORK ↓LOGGING ↓SECURITY CONTROL ↓EVIDENCEGolden Rule
Section titled “Golden Rule”Do not assume:
Unfamiliar=MaliciousInstead:
Observe ↓Collect Evidence ↓Compare Baseline ↓Establish Context ↓Determine RiskPart 01 — Prepare the Security Workspace
Section titled “Part 01 — Prepare the Security Workspace”Create:
mkdir -p ~/linux-security-labEnter it:
cd ~/linux-security-labCreate:
mkdir baseline evidence findings reportsYour structure becomes:
linux-security-lab/├── baseline/├── evidence/├── findings/└── reports/Part 02 — Confirm Authorization
Section titled “Part 02 — Confirm Authorization”Before conducting any security assessment, confirm:
System Is Authorized
Scope Is Known
Assessment Window Is Known
Permitted Actions Are Known
Evidence Handling Is DefinedThis lab should be performed only against:
Your Own Linux VM
Authorized Training Infrastructure
Approved Enterprise SystemsPart 03 — Identify the System
Section titled “Part 03 — Identify the System”Run:
hostnameThen:
cat /etc/os-releaseThen:
uname -aReview:
Hostname
Distribution
Version
Kernel
ArchitectureStep 01 — Capture System Baseline
Section titled “Step 01 — Capture System Baseline”Create:
{ echo "Linux Security Baseline" echo "=======================" echo "Date: $(date)" echo "Hostname: $(hostname)" echo "Kernel: $(uname -r)" echo "User: $(whoami)" echo cat /etc/os-release} > baseline/system.txtWhy This Matters
Section titled “Why This Matters”Security findings require context.
For example:
Vulnerability ↓Affected Software Version ↓Specific Operating SystemWithout an accurate inventory, vulnerability and configuration analysis becomes unreliable.
Part 04 — Review System Uptime
Section titled “Part 04 — Review System Uptime”Run:
uptimeRecord:
Uptime
Load Average
Current TimeSecurity Perspective
Section titled “Security Perspective”Uptime can help answer questions such as:
Was the System Recently Rebooted?
Could a Security Update Be Waiting for Reboot?
Does the Timeline Match the Incident?It is evidence, not proof by itself.
Part 05 — Review Current Identity
Section titled “Part 05 — Review Current Identity”Run:
whoamiThen:
idDocument:
Username
UID
Primary Group
Supplementary Groups
Administrative CapabilityPart 06 — Inventory Users
Section titled “Part 06 — Inventory Users”Run:
getent passwdSave:
getent passwd > baseline/users.txtReview accounts as:
Human Accounts
Administrative Accounts
Service Accounts
System AccountsStep 02 — Identify Interactive Accounts
Section titled “Step 02 — Identify Interactive Accounts”A useful starting point is:
getent passwd | grep -E '/bin/(bash|sh|zsh|fish)$'Exact shell paths vary by environment.
Ask:
Who Owns This Account?
Does It Need Interactive Login?
Is It Still Required?
Does Its Access Match Its Role?Part 07 — Identify UID 0 Accounts
Section titled “Part 07 — Identify UID 0 Accounts”Run:
awk -F: '$3 == 0 {print $1 ":" $3 ":" $7}' /etc/passwdUID 0 represents root-equivalent identity.
Security Finding
Section titled “Security Finding”Unexpected UID 0 accounts should receive immediate review.
Finding:Unexpected Privileged Identity
Observation:An account other than the approved rootidentity has UID 0.
Risk:The account possesses root-equivalentoperating-system privileges.
Recommendation:Validate ownership and business need andremove unauthorized root-equivalentidentity assignments through the approvedchange process.Part 08 — Review Groups
Section titled “Part 08 — Review Groups”Run:
getent groupSave:
getent group > baseline/groups.txtFocus on groups that may grant:
Administrative Access
Application Access
Sensitive File Access
Device AccessPart 09 — Review sudo
Section titled “Part 09 — Review sudo”For your authorized account:
sudo -lReview appropriate administrative groups and sudo configuration.
Security questions:
Who Can Elevate Privilege?
What Can They Execute?
Is Broad Access Necessary?
Is Privileged Activity Logged?
Is Access Individually Attributable?Privilege Model
Section titled “Privilege Model”Standard User ↓Approved Administrative Requirement ↓Controlled sudo ↓Logging ↓ReviewPart 10 — Look for Shared Administrative Access
Section titled “Part 10 — Look for Shared Administrative Access”Shared privileged identities weaken accountability.
Poor model:
Admin A ─┐Admin B ─┼──> shared-admin ──> rootAdmin C ─┘Better:
Admin A ──> Individual Identity ──> sudoAdmin B ──> Individual Identity ──> sudoAdmin C ──> Individual Identity ──> sudoFinding Example
Section titled “Finding Example”Finding:Shared Privileged Account
Risk:Administrative activity cannot bereliably attributed to an individual.
Recommendation:Use individually attributable identitieswith controlled privilege escalation.Part 11 — Review Service Accounts
Section titled “Part 11 — Review Service Accounts”Identify accounts associated with:
Web Servers
Databases
Monitoring Agents
Backup Agents
Application ServicesAsk:
Does the Service Need a Login Shell?
Does It Need a Home Directory?
Does It Need sudo?
Which Files Does It Own?
Which Network Resources Does It Access?Security Principle
Section titled “Security Principle”Prefer:
Application ↓Dedicated Service Account ↓Minimum Required Privilegerather than:
Application ↓rootunless a legitimate technical requirement exists.
Part 12 — Review Logged-In Users
Section titled “Part 12 — Review Logged-In Users”Run:
whoThen:
wReview:
User
Terminal
Source
Login Time
Current ActivityStep 03 — Review Login History
Section titled “Step 03 — Review Login History”Where available:
lastAsk:
Are Sources Expected?
Are Login Times Expected?
Are Privileged Users Expected?
Are There Unusual Patterns?Part 13 — Review Authentication Failures
Section titled “Part 13 — Review Authentication Failures”Authentication evidence varies by distribution.
Possible sources include:
systemd Journal
/var/log/auth.log
/var/log/secureUse the appropriate source for your system.
Look for patterns such as:
Repeated Failures
Unknown Users
Unexpected Sources
Successful Login After Many FailuresImportant
Section titled “Important”A failed-login spike can represent:
User Error
Automation Failure
Credential Misconfiguration
Scanning
Password GuessingInvestigate context before assigning cause.
Part 14 — Review SSH
Section titled “Part 14 — Review SSH”Determine whether SSH is active:
systemctl status sshdor:
systemctl status sshdepending on the distribution.
Review the effective SSH configuration where supported:
sudo sshd -TSSH Security Questions
Section titled “SSH Security Questions”Ask:
Is Root Remote Login Required?
Which Authentication Methods Are Enabled?
Are Unnecessary Users Allowed?
Which Networks Can Reach SSH?
Are SSH Events Logged?
Are Authorized Keys Managed?Part 15 — Review SSH Keys
Section titled “Part 15 — Review SSH Keys”For authorized identities, review:
~/.ssh/
authorized_keysDo not copy private key material into your evidence.
Ask:
Who Owns Each Key?
Is It Still Required?
Is It Shared?
Is It Stale?
Can It Be Revoked?Finding Example
Section titled “Finding Example”Finding:Unmanaged SSH Authorization
Observation:An SSH public key is authorized for anadministrative account but its currentowner cannot be confirmed.
Risk:An unauthorized or former key holdermay retain remote access.
Recommendation:Validate key ownership and remove staleor unauthorized keys.Part 16 — Filesystem Security
Section titled “Part 16 — Filesystem Security”Linux filesystem permissions are fundamental security controls.
Start with sensitive locations such as:
/etc
/root
/home
/var
Application Directories
SSH Directories
Log DirectoriesStep 04 — Review Sensitive Files
Section titled “Step 04 — Review Sensitive Files”Check metadata rather than exposing sensitive contents.
For example:
ls -l /etc/passwdand:
sudo ls -l /etc/shadowDo not copy password hashes into reports.
Part 17 — Review Home Directories
Section titled “Part 17 — Review Home Directories”Run:
ls -ld /home/*Review:
Owner
Group
PermissionsAsk:
Can Other Users Read DataThey Do Not Need?Part 18 — Search for World-Writable Files
Section titled “Part 18 — Search for World-Writable Files”In an authorized disposable lab:
sudo find / -xdev -type f -perm -0002 -print 2>/dev/nullDo not classify every result as a vulnerability automatically.
For each result determine:
Purpose
Owner
Location
Business Requirement
ExposurePart 19 — Review World-Writable Directories
Section titled “Part 19 — Review World-Writable Directories”Shared writable directories may be legitimate.
Review:
Permissions
Sticky Bit
Ownership
PurposeCheck:
ls -ld /tmpA common secure shared-directory pattern includes the sticky bit.
Part 20 — Review SUID Files
Section titled “Part 20 — Review SUID Files”Run:
sudo find / -xdev -type f -perm -4000 -print 2>/dev/nullSUID files can execute with the effective privileges of their owner.
Do not remove SUID permissions blindly.
Investigate:
File
Owner
Package
Purpose
Baseline
Business RequirementPart 21 — Review SGID Files
Section titled “Part 21 — Review SGID Files”Run:
sudo find / -xdev -type f -perm -2000 -print 2>/dev/nullAgain:
Expected? ↓Known Package? ↓Required? ↓Approved?Part 22 — Identify Orphaned Ownership
Section titled “Part 22 — Identify Orphaned Ownership”Security assessments should consider files whose UID or GID no longer maps cleanly to an active identity.
Why?
Deleted Identity ↓Files Remain ↓UID/GID Reused ↓Unexpected OwnershipReview orphaned ownership carefully in authorized environments.
Part 23 — Review Recently Modified Files
Section titled “Part 23 — Review Recently Modified Files”During investigation, file timestamps can provide useful leads.
For a controlled directory:
find /etc -type f -mtime -7 -print 2>/dev/nullThis example identifies files with modification times in a recent window.
Important
Section titled “Important”Recent modification does not equal compromise.
It may represent:
Patch
Configuration Change
Deployment
Administrator Activity
AutomationCorrelate timestamps with approved changes and logs.
Part 24 — Review File Integrity Concept
Section titled “Part 24 — Review File Integrity Concept”Security-sensitive files may require monitoring for unexpected change.
Examples:
SSH Configuration
sudo Configuration
Authentication Configuration
Application Configuration
Scheduled TasksA mature environment may use:
File Integrity Monitoringto detect unauthorized modifications.
Part 25 — Review Processes
Section titled “Part 25 — Review Processes”Run:
ps auxThen:
ps -efReview:
PID
PPID
User
CPU
Memory
CommandProcess Investigation Model
Section titled “Process Investigation Model”PROCESS ↓PID ↓USER ↓PARENT ↓EXECUTABLE ↓NETWORK ↓FILES ↓TIMELINEPart 26 — Identify Resource-Heavy Processes
Section titled “Part 26 — Identify Resource-Heavy Processes”Run:
ps aux --sort=-%cpu | headThen:
ps aux --sort=-%mem | headHigh resource usage can indicate:
Normal Application Load
Runaway Process
Misconfiguration
Unexpected SoftwareDo not conclude malicious activity from resource usage alone.
Part 27 — Investigate a Process
Section titled “Part 27 — Investigate a Process”For an authorized PID:
ps -fp <PID>You can inspect its parent relationship:
ps -o pid,ppid,user,lstart,cmd -p <PID>Ask:
Which User Started It?
What Is Its Parent?
When Did It Start?
What Executable Is Running?
Is It Expected?Part 28 — /proc
Section titled “Part 28 — /proc”Linux exposes runtime process information through:
/procFor an authorized PID, information may be available under:
/proc/<PID>/This can help analysts understand:
Executable
Command Line
Environment
Open Resources
Process StateBe careful: process environments can contain sensitive information.
Do not copy secrets into evidence unnecessarily.
Part 29 — Review Services
Section titled “Part 29 — Review Services”Run:
systemctl --type=service --state=runningThen review enabled services:
systemctl list-unit-files --type=service --state=enabledAsk:
What Is Running?
Why?
Who Owns It?
Does It Need Network Access?
Should It Start Automatically?Part 30 — Correlate Services and Ports
Section titled “Part 30 — Correlate Services and Ports”Run:
sudo ss -lntupCreate a matrix:
| Service | Process | Port | Bind Address | Required |
|---|---|---|---|---|
| SSH | sshd | 22 | Review | Yes |
| Application | Review | Review | Review | Yes |
| Unknown | Review | Review | Review | Investigate |
Security Principle
Section titled “Security Principle”Every network listener should have:
Owner
Purpose
Approved ExposurePart 31 — Investigate Unknown Listener
Section titled “Part 31 — Investigate Unknown Listener”If you discover an unexpected listener:
PORT ↓PID ↓PROCESS ↓USER ↓EXECUTABLE ↓SERVICE ↓START METHOD ↓BUSINESS REQUIREMENTDo not identify software as malicious solely because it uses an unusual port.
Part 32 — Review Active Network Connections
Section titled “Part 32 — Review Active Network Connections”Run:
sudo ss -ntpReview:
Local Address
Remote Address
State
Process
PIDSecurity Questions
Section titled “Security Questions”For unexpected outbound communication ask:
Which Process?
Which User?
Which Destination?
Which Port?
When Did It Begin?
Is It Required?
Does It Match the Baseline?Part 33 — Investigate an Outbound Connection
Section titled “Part 33 — Investigate an Outbound Connection”Use:
NETWORK CONNECTION ↓PID ↓PROCESS ↓PARENT PROCESS ↓USER ↓EXECUTABLE ↓LOGS ↓BUSINESS CONTEXTThis is a powerful Linux incident-analysis workflow.
Part 34 — Review Firewall
Section titled “Part 34 — Review Firewall”Determine the active host-firewall technology.
Possible examples:
firewalld
nftables
ufwFor firewalld:
sudo firewall-cmd --list-allFor UFW:
sudo ufw status verboseFor nftables:
sudo nft list rulesetFirewall Assessment Questions
Section titled “Firewall Assessment Questions”Is the Firewall Active?
Which Services Are Allowed?
Which Sources Are Allowed?
Is Administrative Access Restricted?
Are Unnecessary Ports Exposed?
Does IPv6 Receive Equivalent Protection?Finding Example
Section titled “Finding Example”Finding:Administrative Service Broadly Accessible
Observation:The Linux remote-administration serviceis reachable from a broader network rangethan required.
Risk:Unnecessary exposure increases theopportunity for authentication attacksand exploitation.
Recommendation:Restrict administrative access toapproved management sources and enforcestrong authentication and monitoring.Part 35 — Review Installed Packages
Section titled “Part 35 — Review Installed Packages”Debian-family example:
dpkg -lRPM-family example:
rpm -qaSave an inventory where appropriate.
Package Security Questions
Section titled “Package Security Questions”Is the Package Required?
Is It Supported?
Is It Updated?
Which Repository Supplied It?
Does It Introduce a Service?Part 36 — Review Patch Status
Section titled “Part 36 — Review Patch Status”Use the package-management tools appropriate to your distribution to identify available updates.
Do not automatically apply production updates during an assessment.
Follow:
Discover ↓Assess ↓Prioritize ↓Test ↓Approve ↓Deploy ↓ValidateVulnerability vs Patch
Section titled “Vulnerability vs Patch”Remember:
Available Updatedoes not automatically mean:
Currently Exploitable VulnerabilitySimilarly:
No Available Package Updatedoes not prove:
No Security RiskSecurity analysis requires context.
Part 37 — Review Package Repositories
Section titled “Part 37 — Review Package Repositories”Review configured repositories.
Ask:
Are They Approved?
Are They Expected?
Are They Trusted?
Are Unsupported Sources Present?Finding Example
Section titled “Finding Example”Finding:Unapproved Software Repository
Observation:The server references a package repositorythat is not part of the approved softwaresupply-chain configuration.
Risk:Software from an untrusted or unmanagedsource could introduce unsupported ormalicious components.
Recommendation:Validate the repository requirement andrestrict software installation to approvedtrusted sources.Part 38 — Review SELinux
Section titled “Part 38 — Review SELinux”On systems using SELinux:
getenforceThen, where available:
sestatusPossible states include:
Enforcing
Permissive
DisabledSecurity Principle
Section titled “Security Principle”Do not use:
Disable SELinuxas a generic troubleshooting solution.
Instead:
Application Problem ↓Standard Permissions ↓Configuration ↓SELinux Evidence ↓Context / Policy ↓Correct RemediationPart 39 — Review SELinux Context
Section titled “Part 39 — Review SELinux Context”For an authorized file:
ls -Z <file>For a directory:
ls -Zd <directory>When a legitimate application is blocked, investigate whether:
Label Is Incorrect
Boolean Is Required
Policy Is Missing
Application Is Behaving UnexpectedlyPart 40 — Review AppArmor
Section titled “Part 40 — Review AppArmor”On distributions using AppArmor, review active profiles with the supported tools available on the platform.
Understand:
Enforce
Complain
UnconfinedThe security objective is:
Application ↓Minimum Required CapabilityPart 41 — Review System Logs
Section titled “Part 41 — Review System Logs”Start with:
journalctl -n 100Review:
Errors
Warnings
Service Activity
Authentication
Kernel EventsStep 05 — Review Boot Events
Section titled “Step 05 — Review Boot Events”journalctl -bThis may reveal:
Service Failures
Filesystem Problems
Device Problems
Security Control IssuesPart 42 — Review Authentication Logs
Section titled “Part 42 — Review Authentication Logs”Depending on distribution:
/var/log/auth.logor:
/var/log/securemay contain authentication-related information.
The system journal may also contain relevant evidence.
Authentication Investigation
Section titled “Authentication Investigation”Look for:
Failed Login
Successful Login
sudo Activity
Account Changes
SSH SessionsPart 43 — Review sudo Activity
Section titled “Part 43 — Review sudo Activity”Your goal is to answer:
Who Elevated Privilege?
When?
What Happened?
Was It Expected?Correlate:
Authentication Logs
sudo Logs
Shell History Where Appropriate
Change Records
System LogsDo not treat shell history as authoritative forensic evidence by itself.
Part 44 — Review Audit Framework
Section titled “Part 44 — Review Audit Framework”Some Linux systems use:
auditdCheck:
systemctl status auditdwhere available.
Review configured rules:
sudo auditctl -lwhere authorized.
Audit Value
Section titled “Audit Value”Linux auditing can provide evidence related to:
Security-Relevant System Calls
Identity Changes
Sensitive File Activity
Administrative ActionsPart 45 — Logging vs Auditing
Section titled “Part 45 — Logging vs Auditing”Understand the distinction:
Application/System Logs ↓What Components Reportedversus:
Audit Framework ↓Security-Relevant OS ActivityBoth can be useful.
Part 46 — Review Time Synchronization
Section titled “Part 46 — Review Time Synchronization”Run:
timedatectlReview:
System Time
Timezone
SynchronizationWhy Time Matters
Section titled “Why Time Matters”Incident timelines may combine:
Linux Logs
Firewall Logs
Cloud Logs
SIEM Events
Application LogsIf clocks disagree significantly:
Timeline Reconstructionbecomes much harder.
Part 47 — Review Scheduled Tasks
Section titled “Part 47 — Review Scheduled Tasks”Run:
crontab -lReview authorized system scheduling locations.
Also check:
systemctl list-timers --allSecurity Questions
Section titled “Security Questions”For every scheduled execution ask:
Which User?
Which Command?
Which Script?
Who Owns the Script?
Can Another User Modify It?
Is It Expected?
When Does It Run?Part 48 — Persistence Concept
Section titled “Part 48 — Persistence Concept”Attackers and legitimate administrators may use some of the same operating-system mechanisms.
Potential persistence locations can include:
Scheduled Tasks
systemd Services
Startup Configuration
SSH Authorized Keys
User Accounts
Application Startup MechanismsThe presence of one does not prove malicious activity.
The question is:
Is It Expected?Part 49 — Review Enabled Services
Section titled “Part 49 — Review Enabled Services”An unexpected enabled service may start automatically after reboot.
Review:
systemctl list-unit-files --type=service --state=enabledCompare against:
Approved Server BaselinePart 50 — Review systemd Unit Locations
Section titled “Part 50 — Review systemd Unit Locations”Systemd configuration can exist in locations such as:
/etc/systemd/system
/usr/lib/systemd/system
/lib/systemd/systemdepending on distribution.
Pay special attention to:
Unexpected Custom Units
Recently Modified Units
Unknown Executables
Unusual UsersPart 51 — Review Authorized Keys as Persistence
Section titled “Part 51 — Review Authorized Keys as Persistence”An unauthorized public key added to:
authorized_keyscan create continued remote access.
Review:
Key Owner
Account
File Modification Time
Business Requirement
Associated Change RecordNever expose private keys.
Part 52 — Review Shell Startup Files
Section titled “Part 52 — Review Shell Startup Files”User shell startup files may include:
.bashrc
.profile
.bash_profiledepending on the environment.
Unexpected commands in startup files may require investigation.
Again:
Customdoes not mean:
MaliciousPart 53 — Review Temporary Locations
Section titled “Part 53 — Review Temporary Locations”Security investigations may include directories such as:
/tmp
/var/tmpbecause applications and users frequently write there.
Review:
Unexpected Executables
Unexpected Ownership
Unusual Modification TimesDo not delete suspicious files before deciding whether they are needed as evidence.
Part 54 — Evidence Preservation
Section titled “Part 54 — Evidence Preservation”When suspicious activity is discovered, avoid unnecessary destructive actions.
Before remediation consider capturing:
Process Information
Network Connections
File Metadata
Relevant Logs
User Information
Service Configuration
TimestampsIncident Principle
Section titled “Incident Principle”Detect ↓Preserve ↓Understand ↓Contain ↓Remediatenot:
Detect ↓Delete EverythingPart 55 — Build a Timeline
Section titled “Part 55 — Build a Timeline”Suppose you observe:
02:10 Failed SSH Logins
02:16 Successful Login
02:18 sudo Activity
02:21 New User Created
02:24 New Service Started
02:27 External ConnectionIndividually these events may have multiple explanations.
Together they create:
Investigation ContextTimeline Table
Section titled “Timeline Table”| Time | Event | Identity | Source | Evidence |
|---|---|---|---|---|
| 02:10 | Failed login | user | IP | Auth log |
| 02:16 | Successful login | user | IP | Auth log |
| 02:18 | Privilege use | user | Local | sudo log |
| 02:21 | Account creation | root | Local | Audit/log |
| 02:24 | Service start | service | Local | Journal |
| 02:27 | Connection | process | Remote IP | Socket/network evidence |
Part 56 — Correlation
Section titled “Part 56 — Correlation”Security analysis becomes stronger when multiple sources agree.
Authentication Log +Process Information +Network Connection +Filesystem Metadata +Audit Evidence ↓Higher-Confidence ConclusionPart 57 — Security Investigation Scenario
Section titled “Part 57 — Security Investigation Scenario”Scenario
Section titled “Scenario”Monitoring reports:
Unexpected Outbound Connectionfrom Linux ServerYour workflow:
01 Confirm Alert
02 Identify Destination
03 Identify Local Port
04 Identify PID
05 Identify Process
06 Identify User
07 Identify Parent Process
08 Review Executable
09 Review Start Time
10 Review Authentication Activity
11 Review Persistence
12 Build Timeline
13 Determine Business ContextPart 58 — Investigate Without Immediate Destruction
Section titled “Part 58 — Investigate Without Immediate Destruction”Do not immediately:
Kill Process
Delete File
Remove User
Reboot Serverunless immediate containment is required by your incident-response procedure.
These actions can alter:
Volatile Evidence
Timestamps
Connections
Process StatePart 59 — Security Finding: Unexpected Service
Section titled “Part 59 — Security Finding: Unexpected Service”Finding:Unexpected System Service
Observation:A running and enabled service is notincluded in the approved server baselineand its business owner cannot initiallybe identified.
Risk:An unauthorized service may increaseattack surface or provide persistentexecution.
Recommendation:Identify the service owner, executable,configuration, network exposure, andinstallation source. Disable or removethe service if it is confirmed to beunauthorized.Part 60 — Security Finding: Excessive Privilege
Section titled “Part 60 — Security Finding: Excessive Privilege”Finding:Excessive Administrative Access
Observation:A standard user has broad administrativeprivileges beyond the requirements oftheir documented role.
Risk:Compromise of the account could providefull system control.
Recommendation:Reduce privileges to the minimum requiredand periodically review privileged access.Part 61 — Security Finding: Weak File Permissions
Section titled “Part 61 — Security Finding: Weak File Permissions”Finding:Sensitive File Permissions Too Broad
Observation:A security-sensitive file can be accessedby users who do not require that access.
Risk:Unauthorized users may read or modifysensitive configuration or data.
Recommendation:Restrict ownership and permissionsaccording to least privilege and validateapplication compatibility.Part 62 — Security Finding: Missing Security Logging
Section titled “Part 62 — Security Finding: Missing Security Logging”Finding:Insufficient Security Event Visibility
Observation:Important authentication oradministrative activity is not retainedwith sufficient detail for investigation.
Risk:Security incidents may be difficult todetect, investigate, or reconstruct.
Recommendation:Enable appropriate system and securitylogging, protect logs, synchronize time,and forward required events to acentralized monitoring platform.Part 63 — Security Finding: Unexpected External Connection
Section titled “Part 63 — Security Finding: Unexpected External Connection”Finding:Unexpected External Communication
Observation:A Linux process communicates with anexternal destination that is not partof the documented server workflow.
Risk:The communication could representmisconfiguration, unauthorized software,or malicious activity.
Recommendation:Correlate the connection with process,identity, executable, destination,timeline, and business context beforedetermining containment and remediation.Part 64 — Security Finding: Mandatory Access Control
Section titled “Part 64 — Security Finding: Mandatory Access Control”Finding:Mandatory Access Control Not Enforcing
Observation:The platform's expected SELinux orAppArmor security control is not operatingin its approved enforcement state.
Risk:Processes may have fewer restrictionsthan required by the security baseline.
Recommendation:Determine why enforcement is unavailable,resolve compatibility issues, and restorethe approved security state.Part 65 — Risk Rating
Section titled “Part 65 — Risk Rating”Prioritize findings using context.
Consider:
Exposure
Privilege
Exploitability
Data Sensitivity
Business Criticality
Existing Controls
Potential ImpactA simple model:
LIKELIHOOD ×IMPACT =RISKExample
Section titled “Example”Finding:Unused Local Package
Exposure:Low
Privilege:Low
Impact:Limited
Priority:LowerCompare with:
Finding:Internet-Reachable Administrative Service
Exposure:High
Privilege:High
Impact:High
Priority:HighPart 66 — Validate Security Controls
Section titled “Part 66 — Validate Security Controls”Your assessment should determine whether controls are merely:
Configuredor actually:
EffectiveExample:
Firewall Installeddoes not prove:
Firewall Protecting ServerLikewise:
auditd Installeddoes not prove:
Useful Audit Rules ExistPart 67 — Negative Testing
Section titled “Part 67 — Negative Testing”Security testing should validate denied behavior where safe and authorized.
Examples:
Unauthorized UserCannot Access Sensitive Directory
Unapproved Network SourceCannot Reach Administrative Service
Service AccountCannot Log In Interactively
Standard UserCannot Perform Administrative ActionPart 68 — Review Defense in Depth
Section titled “Part 68 — Review Defense in Depth”A secure Linux server should not depend on one control.
Think:
Identity +Least Privilege +Filesystem Permissions +SELinux/AppArmor +Firewall +Patching +Logging +Monitoring +BackupsPart 69 — Linux Security in AWS
Section titled “Part 69 — Linux Security in AWS”For an EC2 Linux workload:
AWS IAM ↓VPC ↓Security Group ↓EC2 Instance ↓Linux IAM ↓Linux Firewall ↓ApplicationA Linux assessment must consider that some controls may exist outside the operating system.
Part 70 — Linux Security in Azure
Section titled “Part 70 — Linux Security in Azure”Microsoft Entra ID ↓Azure RBAC ↓Virtual Network ↓NSG ↓Linux VM ↓Linux SecurityPart 71 — Linux Security in Google Cloud
Section titled “Part 71 — Linux Security in Google Cloud”Google Cloud IAM ↓VPC Firewall ↓Compute Engine ↓Linux IAM ↓Host SecurityCloud Security Lesson
Section titled “Cloud Security Lesson”Do not assess:
Linux VMin isolation from:
Cloud Control PlanePart 72 — Linux Security and Containers
Section titled “Part 72 — Linux Security and Containers”Container security depends heavily on Linux.
Container ↓Runtime ↓Linux KernelRelevant Linux controls include:
Namespaces
Capabilities
Filesystem Permissions
SELinux/AppArmor
Seccomp
Users
NetworkingPart 73 — Linux Security and Kubernetes
Section titled “Part 73 — Linux Security and Kubernetes”Kubernetes workloads ultimately execute on Linux nodes in many environments.
Kubernetes ↓Container Runtime ↓Linux Node ↓KernelLinux security therefore supports:
Node Hardening
Runtime Security
Container Isolation
Incident ResponsePart 74 — Linux Security and SOC Operations
Section titled “Part 74 — Linux Security and SOC Operations”Linux telemetry can help SOC teams investigate:
Suspicious Login
Privilege Escalation
New User
New Service
Unexpected Process
Unknown Port
Outbound Connection
Persistence
Configuration ChangeSOC Investigation Model
Section titled “SOC Investigation Model”ALERT ↓HOST ↓USER ↓PROCESS ↓NETWORK ↓FILES ↓LOGS ↓TIMELINE ↓DECISIONPart 75 — Linux Security and Incident Response
Section titled “Part 75 — Linux Security and Incident Response”Security assessment asks:
Is the Server Secure?Incident response asks:
What Happened?Strong Linux skills support both.
Part 76 — Build the Linux Security Report
Section titled “Part 76 — Build the Linux Security Report”Create:
Linux Security Assessment ReportYour report should contain the following sections.
1. Executive Summary
Section titled “1. Executive Summary”Include:
Assessment Scope
Overall Security Posture
Critical Observations
Highest-Risk Findings
Recommended Priorities2. System Information
Section titled “2. System Information”Document:
Hostname
Distribution
Version
Kernel
Server Role
Assessment Date3. Identity Review
Section titled “3. Identity Review”Document:
Human Accounts
Service Accounts
UID 0 Accounts
Dormant Accounts
Group Memberships4. Privileged Access
Section titled “4. Privileged Access”Document:
sudo
Administrative Groups
Shared Accounts
SUID/SGID
Root Access5. Authentication
Section titled “5. Authentication”Document:
SSH
Authorized Keys
Authentication Logs
Remote Access Exposure6. Filesystem Security
Section titled “6. Filesystem Security”Document:
Sensitive Permissions
World-Writable Resources
Home Directories
Special Permissions
Unexpected Changes7. Process Review
Section titled “7. Process Review”Document:
Unexpected Processes
High Resource Processes
Process Ownership
Parent/Child Relationships8. Service Review
Section titled “8. Service Review”Document:
Running Services
Enabled Services
Unexpected Services
Business Requirements9. Network Security
Section titled “9. Network Security”Document:
Listening Ports
Bind Addresses
Active Connections
Firewall
Unexpected Destinations10. Software Security
Section titled “10. Software Security”Document:
Packages
Pending Updates
Repositories
Unsupported Software11. Security Controls
Section titled “11. Security Controls”Document:
SELinux/AppArmor
Logging
Auditing
Time Synchronization12. Persistence Review
Section titled “12. Persistence Review”Document:
Scheduled Tasks
systemd Units
SSH Keys
Startup Configuration13. Findings
Section titled “13. Findings”For every finding:
Title
Severity
Observation
Evidence
Risk
Recommendation
Owner
StatusPart 77 — Sample Findings Table
Section titled “Part 77 — Sample Findings Table”| Finding | Risk | Priority |
|---|---|---|
| Excessive sudo | Privilege abuse | High |
| Broad SSH exposure | Remote attack surface | High |
| Unknown listener | Unapproved exposure | High |
| Stale account | Unauthorized access | Medium |
| Weak permissions | Data exposure | Medium |
| Unnecessary package | Attack surface | Low/Medium |
| Missing centralized logging | Detection gap | Medium/High |
Risk depends on actual context.
Part 78 — Evidence Handling
Section titled “Part 78 — Evidence Handling”Security evidence should be:
Relevant
Accurate
Timestamped
Protected
Minimized
TraceableDo not place:
Passwords
Password Hashes
Private Keys
API Tokens
Application Secretsinto ordinary assessment reports.
Part 79 — Assessment Evidence Checklist
Section titled “Part 79 — Assessment Evidence Checklist”Capture appropriate evidence for:
- System identity
- OS and kernel
- Uptime
- User inventory
- Group inventory
- UID 0 review
- sudo review
- SSH review
- Authentication activity
- Filesystem permissions
- World-writable resources
- SUID files
- SGID files
- Process inventory
- Service inventory
- Package inventory
- Listening ports
- Active connections
- Firewall
- SELinux/AppArmor
- Logging
- Auditing
- Time synchronization
- Scheduled tasks
- Persistence review
- Security findings
Part 80 — Final Practical Challenge
Section titled “Part 80 — Final Practical Challenge”Without relying on a graphical security scanner, determine:
01 Which Linux system is this?
02 Which users can log in?
03 Who has administrative access?
04 Which service accounts exist?
05 Which sensitive permissions are excessive?
06 Which SUID/SGID files exist?
07 Which processes are running?
08 Which services start automatically?
09 Which ports are listening?
10 Which processes own those ports?
11 Which external connections exist?
12 Which firewall controls are active?
13 Is SELinux/AppArmor enforcing?
14 Are important events logged?
15 Is auditing available?
16 Which scheduled tasks exist?
17 Are there unexpected persistence mechanisms?
18 What changed recently?
19 What is the highest-risk issue?
20 What should be remediated first?Part 81 — Professional Security Assessment Workflow
Section titled “Part 81 — Professional Security Assessment Workflow”Use this workflow in real environments:
01 Authorization
02 Scope
03 Asset Identification
04 Baseline Collection
05 Identity Review
06 Privilege Review
07 Authentication Review
08 Filesystem Review
09 Process Review
10 Service Review
11 Software Review
12 Network Review
13 Security Control Review
14 Logging Review
15 Persistence Review
16 Evidence Correlation
17 Risk Assessment
18 Findings
19 Remediation
20 Validation
21 ReportingPart 82 — Common Linux Security Mistakes
Section titled “Part 82 — Common Linux Security Mistakes”Avoid:
Assuming Unknown Means Malicious
Deleting Evidence Immediately
Disabling SELinux to Fix Applications
Disabling Firewall to Troubleshoot
Using chmod 777
Ignoring Service Accounts
Ignoring SSH Keys
Ignoring Outbound Connections
Ignoring IPv6
Ignoring Scheduled Tasks
Ignoring systemd Persistence
Ignoring Cloud-Level Controls
Trusting Shell History as Complete Evidence
Applying Hardening Without Business Context
Reporting Findings Without EvidencePart 83 — Security Maturity Model
Section titled “Part 83 — Security Maturity Model”Level 1 — Reactive
Section titled “Level 1 — Reactive”Patch When Needed
Manual Account Management
Basic FirewallLevel 2 — Baseline
Section titled “Level 2 — Baseline”Documented Hardening
Access Reviews
Central Logging
Approved ServicesLevel 3 — Managed
Section titled “Level 3 — Managed”Configuration Management
SIEM
EDR
Automated Compliance
Central IAM
Vulnerability ManagementLevel 4 — Continuous
Section titled “Level 4 — Continuous”Continuous Monitoring
Configuration Drift Detection
Automated Response
Policy as Code
Just-in-Time Privilege
Threat DetectionPart 84 — Career Connection
Section titled “Part 84 — Career Connection”The skills practiced in this lab directly support:
Linux Security Engineer
SOC Analyst
Incident Responder
Cloud Security Engineer
Security Consultant
DevSecOps Engineer
Platform Security Engineer
Cybersecurity Analyst
Security ArchitectInterview Scenario 01
Section titled “Interview Scenario 01”You discover an unfamiliar Linux process. What should you do?
Identify PID ↓Identify User ↓Identify Parent ↓Identify Executable ↓Review Network Activity ↓Review Logs ↓Compare Baseline ↓Determine ContextInterview Scenario 02
Section titled “Interview Scenario 02”You find an unknown listening port. Is the server compromised?
Not necessarily.
Investigate:
Process
Service
User
Executable
Business Requirement
Network Exposure
Logsbefore concluding.
Interview Scenario 03
Section titled “Interview Scenario 03”You discover suspicious activity. Should you immediately delete the suspicious file?
Usually not before considering evidence requirements and the incident-response procedure.
First determine whether you need to preserve:
File Metadata
Process Information
Connections
Logs
TimelineInterview Scenario 04
Section titled “Interview Scenario 04”A server is fully patched. Is it secure?
Not necessarily.
It may still have:
Weak IAM
Excessive sudo
Bad SSH Configuration
Open Ports
Weak Permissions
Disabled Logging
Poor Security ControlsInterview Scenario 05
Section titled “Interview Scenario 05”Why is Linux administration knowledge essential for security analysts?
Because you cannot reliably identify:
Abnormal Linux Behaviorwithout understanding:
Normal Linux Behavior50 Linux Security Interview Questions
Section titled “50 Linux Security Interview Questions”- What is Linux security?
- What is a Linux security baseline?
- Why is asset identification important?
- How do you identify the Linux distribution?
- Why is kernel information security-relevant?
- What is UID 0?
- How would you identify privileged accounts?
- Why are service accounts security-sensitive?
- What is least privilege?
- Why are shared administrator accounts risky?
- How would you review sudo access?
- Why should SSH keys be reviewed?
- What is a stale SSH key?
- What are world-writable files?
- What is the sticky bit?
- What is SUID?
- What is SGID?
- Why should special-permission files be baselined?
- What are orphaned files?
- Why are file timestamps useful?
- What is file integrity monitoring?
- How would you investigate an unknown process?
- Why is PPID useful during investigation?
- What information does
/procexpose? - How would you identify running services?
- What is the difference between running and enabled services?
- How do you identify listening ports?
- Why is a wildcard bind security-relevant?
- How would you investigate an outbound connection?
- Why should egress traffic be monitored?
- What is host firewalling?
- Why is IPv6 part of a security assessment?
- Why should installed packages be inventoried?
- Why are software repositories security-relevant?
- What is SELinux?
- Why should SELinux not simply be disabled?
- What is AppArmor?
- Why is centralized logging important?
- What is Linux auditing?
- Why is time synchronization important?
- Why should scheduled tasks be reviewed?
- How can systemd relate to persistence?
- Why are authorized SSH keys part of persistence review?
- What is evidence preservation?
- Why is timeline reconstruction important?
- What is event correlation?
- What is defense in depth?
- How does Linux security relate to cloud security?
- How does Linux security support Kubernetes security?
- What is your Linux security assessment methodology?
Lab Completion Checklist
Section titled “Lab Completion Checklist”System
Section titled “System”- Identified OS
- Identified kernel
- Reviewed uptime
- Created baseline
- Recorded assessment scope
Identity
Section titled “Identity”- Reviewed users
- Reviewed UID 0
- Reviewed groups
- Reviewed service accounts
- Reviewed logged-in users
- Reviewed login history
Privilege
Section titled “Privilege”- Reviewed sudo
- Reviewed administrative groups
- Reviewed shared privileged identities
- Reviewed SUID
- Reviewed SGID
Authentication
Section titled “Authentication”- Reviewed SSH
- Reviewed authorized keys
- Reviewed authentication events
- Reviewed remote access exposure
Filesystem
Section titled “Filesystem”- Reviewed sensitive permissions
- Reviewed home directories
- Reviewed world-writable files
- Reviewed shared directories
- Reviewed recent changes
- Understood file integrity monitoring
Processes
Section titled “Processes”- Reviewed running processes
- Reviewed CPU usage
- Reviewed memory usage
- Investigated process ancestry
- Correlated processes with identities
Services
Section titled “Services”- Reviewed running services
- Reviewed enabled services
- Mapped services to business requirements
- Investigated unexpected services
Network
Section titled “Network”- Reviewed listening ports
- Mapped ports to processes
- Reviewed active connections
- Reviewed unexpected destinations
- Reviewed firewall controls
Software
Section titled “Software”- Reviewed installed packages
- Reviewed patch status
- Reviewed repositories
- Identified unnecessary software
Security Controls
Section titled “Security Controls”- Reviewed SELinux/AppArmor
- Reviewed logging
- Reviewed auditing
- Reviewed time synchronization
Persistence
Section titled “Persistence”- Reviewed scheduled tasks
- Reviewed systemd timers
- Reviewed enabled services
- Reviewed SSH keys
- Reviewed startup mechanisms
Investigation
Section titled “Investigation”- Built an event timeline
- Correlated multiple evidence sources
- Preserved relevant evidence
- Identified suspicious activity
- Distinguished evidence from assumptions
Reporting
Section titled “Reporting”- Created findings
- Assigned risk
- Created recommendations
- Documented evidence
- Produced final assessment report
Final Linux Security Mental Model
Section titled “Final Linux Security Mental Model”When approaching any Linux server, think:
WHAT IS THIS SYSTEM? ↓WHO CAN ACCESS IT? ↓WHO HAS PRIVILEGE? ↓WHAT SOFTWARE EXISTS? ↓WHAT IS RUNNING? ↓WHAT IS EXPOSED? ↓WHO IS IT TALKING TO? ↓WHAT HAS CHANGED? ↓WHAT STARTS AUTOMATICALLY? ↓WHAT SECURITY CONTROLS EXIST? ↓CAN WE SEE WHAT HAPPENED? ↓WHAT IS DIFFERENT FROM BASELINE?Mission Accomplished
Section titled “Mission Accomplished”You have completed the five core Linux labs:
Lab 01 — Linux Administration ↓Lab 02 — Linux Hardening ↓Lab 03 — Linux IAM ↓Lab 04 — Linux Networking ↓Lab 05 — Linux SecurityYou have progressed from:
Linux Userto:
Linux Administrator ↓Linux Security Administrator ↓Linux Security AnalystMore importantly, you now understand the security workflow:
BASELINE ↓OBSERVE ↓COLLECT ↓CORRELATE ↓ANALYZE ↓PRIORITIZE ↓REMEDIATE ↓VALIDATE ↓DOCUMENTWhat’s Next?
Section titled “What’s Next?”➡️ Runbook 01 — Linux Incident Investigation
The labs taught you individual technical skills.
The runbooks will now teach you how to apply those skills as repeatable professional operational procedures.
In the next runbook, you will learn how to respond when:
SOC Alert ↓Suspicious Linux Host ↓Initial Triage ↓Identity Investigation ↓Process Investigation ↓Network Investigation ↓Persistence Review ↓Log Correlation ↓Timeline Reconstruction ↓Scope Assessment ↓Containment Decision ↓Evidence and ReportingYour Linux journey now moves into operational security:
Linux Certifications ↓Linux Labs ↓Runbook 01 — Linux Incident Investigation ↓Runbook 02 — Linux Security Assessment ↓Runbook 03 — Linux Server Hardening