Runbook 02 — Security Data Ingestion and Normalization

Runbook Information

Runbook Type: Security Data Engineering / Security Automation Operations
Difficulty: Intermediate → Advanced
Primary Audience: SOC Analysts / Security Engineers / Detection Engineers / Cloud Security Engineers / Security Automation Engineers
Security Domains: SOC / SIEM / SOAR / Cloud Security / Threat Intelligence / Vulnerability Management
Execution Model: Repeatable ingestion and normalization workflow
Primary Goal: Turn inconsistent raw security data into trustworthy, structured, traceable security records

Purpose

Security automation depends on data.

But security data arrives from many different systems:

SIEM

EDR

FIREWALLS

IDENTITY PLATFORMS

CLOUD PROVIDERS

VULNERABILITY SCANNERS

THREAT INTELLIGENCE

EMAIL SECURITY

APPLICATION LOGS

LINUX SYSTEMS

WINDOWS SYSTEMS

Each system may represent similar information differently.

For example:

HOSTNAME

may appear as:

WEB01

web01

web01.example.local

Web01

Severity may appear as:

Critical

CRITICAL

critical

5

P1

Authentication outcomes may appear as:

Success

SUCCESS

allowed

accepted

0

Before security data can be:

CORRELATED

SCORED

SEARCHED

AUTOMATED

REPORTED

it needs to be:

VALIDATED

NORMALIZED

DEDUPLICATED

ENRICHED

TRACEABLE

Core Principle

The central principle of this runbook is:

BAD SECURITY DATA
IN
=
BAD SECURITY DECISIONS
OUT

Security Data Pipeline

SOURCE
  ↓
COLLECT
  ↓
PRESERVE RAW
  ↓
VALIDATE
  ↓
PARSE
  ↓
NORMALIZE
  ↓
DEDUPLICATE
  ↓
ENRICH
  ↓
QUALITY CHECK
  ↓
STORE
  ↓
CORRELATE
  ↓
SECURITY DECISION

01 — Identify the Security Use Case

Do not start by collecting everything.

Start with:

WHAT SECURITY QUESTION
ARE WE TRYING TO ANSWER?

Examples:

WHICH USERS ARE EXPERIENCING
REPEATED LOGIN FAILURES?

WHICH INTERNET-FACING ASSETS
HAVE CRITICAL VULNERABILITIES?

WHICH CLOUD RESOURCES
ARE PUBLIC?

WHICH ALERTS INVOLVE
PRIVILEGED USERS?

WHICH IOCS APPEAR
IN SECURITY EVENTS?

02 — Define Required Data

For each security question, determine:

WHICH DATA
IS ACTUALLY REQUIRED?

Example:

Question:
Which privileged accounts have suspicious authentication activity?

Required data:

AUTHENTICATION EVENTS

IDENTITY INVENTORY

PRIVILEGE STATUS

TIMESTAMPS

SOURCE IP

TARGET SYSTEM

03 — Avoid Collecting Everything

More data does not automatically mean better security.

Excessive collection increases:

COST

STORAGE

PRIVACY RISK

PROCESSING TIME

NOISE

ATTACK SURFACE

Use:

DATA MINIMIZATION

04 — Inventory Data Sources

Create a source inventory.

Example:

Source	Data	Method	Owner
SIEM	Alerts	API	SOC
Entra ID	Identity	API	IAM
CMDB	Assets	API	IT
Vulnerability tool	Findings	API	VM Team
Linux hosts	Auth logs	File/Agent	Platform
Cloud platform	Config	API	Cloud Team

05 — Record Source Ownership

Every data source should have an owner.

Document:

SOURCE NAME

TECHNICAL OWNER

BUSINESS OWNER

CONTACT

SUPPORT PROCESS

06 — Understand Source Authority

Not every system is authoritative for every field.

Example:

SIEM

may know:

OBSERVED USERNAME

but:

IDENTITY DIRECTORY

may be authoritative for:

ACCOUNT STATUS

DEPARTMENT

PRIVILEGE

07 — Define Authoritative Sources

Example:

IDENTITY
→ DIRECTORY

ASSET OWNER
→ CMDB

VULNERABILITY STATUS
→ VULNERABILITY PLATFORM

CLOUD RESOURCE STATE
→ CLOUD API

08 — Document Source Trust

For each source, record:

RELIABILITY

FRESHNESS

KNOWN LIMITATIONS

COLLECTION METHOD

FAILURE MODES

09 — Choose Collection Method

Common collection methods include:

API

LOG FORWARDER

AGENT

FILE EXPORT

DATABASE QUERY

MESSAGE QUEUE

WEBHOOK

CLOUD EVENT STREAM

10 — Prefer Structured Data

Where possible prefer:

JSON

CSV

DATABASE RECORDS

STRUCTURED API RESPONSES

over unstructured:

FREE-FORM TEXT

Structured data is usually easier to:

VALIDATE

PARSE

NORMALIZE

TEST

11 — Raw Logs Still Matter

Many useful security sources are log-based.

Examples:

LINUX AUTH LOG

WINDOWS EVENT LOG

WEB SERVER LOG

FIREWALL LOG

APPLICATION LOG

These may require:

REGEX

PARSER

FIELD EXTRACTION

before normalization.

12 — Define Ingestion Frequency

Decide whether data arrives:

REAL-TIME

NEAR REAL-TIME

HOURLY

DAILY

WEEKLY

ON DEMAND

13 — Match Frequency to Security Need

Example:

SOC ALERTS
→ NEAR REAL-TIME

ASSET INVENTORY
→ DAILY MAY BE ACCEPTABLE

QUARTERLY COMPLIANCE EVIDENCE
→ PERIODIC

14 — Define Collection Time

Every collection should record:

WHEN WAS THIS DATA COLLECTED?

Use:

UTC

where possible.

Example:

2026-08-29T06:30:00Z

15 — Event Time vs Collection Time

Do not confuse:

EVENT TIME

with:

COLLECTION TIME

Example:

EVENT OCCURRED:
10:00

COLLECTOR RECEIVED:
10:02

PIPELINE PROCESSED:
10:05

These are different.

16 — Preserve Original Timestamp

Never discard the original timestamp if it is useful for investigation.

Store:

original_timestamp

and normalized:

event_time_utc

17 — Preserve Raw Data

Before transforming important security data:

PRESERVE
THE ORIGINAL

Architecture:

SOURCE
 ↓
RAW COPY
 ↓
NORMALIZED COPY

18 — Why Preserve Raw Data?

If normalization produces the wrong result, you need:

ORIGINAL EVIDENCE

to determine:

WAS THE SOURCE WRONG?

WAS THE PARSER WRONG?

WAS THE NORMALIZER WRONG?

19 — Raw Data Storage

Possible structure:

data/
|
+-- raw/
|   +-- siem/
|   +-- identity/
|   +-- cloud/
|   +-- vulnerability/
|
+-- normalized/

20 — Do Not Modify Raw Evidence

Treat raw data as:

IMMUTABLE

where practical.

Do transformations on:

COPIES

21 — Hash Important Raw Files

Use:

SHA-256

to support integrity checking.

Mental model:

RAW FILE
  ↓
SHA-256
  ↓
INTEGRITY REFERENCE

22 — Record Provenance

Every normalized record should ideally answer:

WHERE DID THIS COME FROM?

Add fields such as:

source_system

source_record_id

collection_time

collector_version

23 — Example Provenance

{
  "source_system": "identity-monitor",
  "source_record_id": "ALT-1002",
  "collection_time": "2026-08-29T06:20:00Z",
  "collector_version": "1.3.0"
}

24 — Validate File Presence

Before processing files:

DOES FILE EXIST?

If not:

LOG FAILURE

MARK SOURCE UNAVAILABLE

DO NOT PRETEND DATA IS EMPTY

25 — Missing File vs Empty File

These are different:

FILE MISSING

may mean:

COLLECTION FAILURE

whereas:

EMPTY VALID FILE

may mean:

NO EVENTS

26 — Validate File Type

Confirm expected formats such as:

JSON

CSV

LOG

NDJSON

Do not assume extensions guarantee content validity.

27 — Validate JSON

A JSON source can fail because of:

MALFORMED JSON

TRUNCATED FILE

ENCODING ISSUE

UNEXPECTED STRUCTURE

28 — Validate CSV Headers

Before parsing CSV:

EXPECTED:
alert_id
timestamp
severity
asset

If the source provides:

id
time
risk
hostname

do not silently assume mappings.

29 — Define Required Fields

Example alert schema:

alert_id

timestamp

source

alert_type

severity

Optional:

user

asset

source_ip

description

30 — Reject or Quarantine Invalid Records

Invalid records should be:

QUARANTINED

rather than silently discarded.

Example:

invalid-records.json

31 — Record Rejection Reason

Example:

{
  "record_id": "ALT-20",
  "reason": "Invalid timestamp"
}

32 — Do Not Crash on One Bad Record

Preferred behavior:

1000 RECORDS
 ↓
999 VALID
1 INVALID
 ↓
PROCESS 999
QUARANTINE 1

rather than:

1 BAD RECORD
 ↓
ENTIRE PIPELINE FAILS

unless integrity requirements demand the entire batch be rejected.

33 — Schema Versioning

Data sources change.

Add:

schema_version

where practical.

Example:

1.0

34 — Why Schema Versions Matter

Version 1 may use:

hostname

Version 2 may use:

device_name

Your parser needs to know which schema it received.

35 — Normalize Field Names

Create consistent internal names.

Example:

host

hostname

device

computer

machine

becomes:

asset

36 — Common Security Schema

Example normalized alert:

{
  "event_id": "ALT-1001",
  "event_time": "2026-08-29T08:15:00Z",
  "event_type": "authentication_failure",
  "severity": "medium",
  "user": "admin01",
  "asset": "JUMP01",
  "source_ip": "203.0.113.25"
}

37 — Normalize Severity

Choose one internal severity model.

For example:

critical

high

medium

low

informational

unknown

38 — Map Vendor Severity

Example:

Vendor:
5

Internal:
critical

or:

Vendor:
P2

Internal:
high

Document mappings explicitly.

39 — Never Guess Ambiguous Mapping

If a vendor field says:

risk = 7

you need documentation before deciding whether:

7
=
high

40 — Preserve Original Severity

Store:

source_severity

and:

normalized_severity

when useful.

41 — Normalize Status

Different systems may use:

Open

OPEN

active

new

in_progress

Define a common model.

Example:

open

in_progress

closed

unknown

42 — Preserve Source Status

Do not lose vendor-specific meaning.

Store:

source_status

where necessary.

43 — Normalize Boolean Fields

Source values may include:

True

TRUE

yes

1

enabled

Normalize to:

true / false / unknown

44 — Avoid False Defaults

Do not convert:

MISSING

to:

false

automatically.

Use:

unknown

where appropriate.

45 — Normalize Hostnames

Example:

web01

WEB01

Web01

to:

WEB01

46 — FQDN vs Hostname

Example:

WEB01.EXAMPLE.LOCAL

may need:

asset_fqdn

and:

asset_short_name

Do not blindly remove domains if they matter.

47 — Asset Identity Is Hard

An asset may be identified by:

HOSTNAME

IP

CLOUD RESOURCE ID

AGENT ID

SERIAL NUMBER

INSTANCE ID

48 — Prefer Stable Asset IDs

Where possible correlate on:

ASSET ID

rather than only:

HOSTNAME

because hostnames can change.

49 — Normalize Usernames

Examples:

ADMIN01

admin01

DOMAIN\admin01

admin01@example.com

may refer to:

SAME OR DIFFERENT IDENTITIES

Do not normalize blindly.

50 — Preserve Identity Namespace

Store fields such as:

username

domain

upn

identity_id

51 — Stable Identity IDs

Prefer:

DIRECTORY OBJECT ID

EMPLOYEE ID

ACCOUNT ID

for correlation where available.

52 — Normalize IP Addresses

Use standard representations.

For IPv4:

010.001.001.001

should be handled carefully.

For IPv6:

multiple textual forms

may represent the same address.

Use proper IP parsing libraries.

53 — Validate IP Addresses

Do not accept:

999.999.999.999

as a valid IP.

54 — Classify IP Context

Potential categories:

PRIVATE

PUBLIC

LOOPBACK

LINK-LOCAL

DOCUMENTATION

UNKNOWN

55 — Normalize Domains

Convert:

Example.COM

to:

example.com

when appropriate.

56 — Handle Trailing Dot

DNS names may appear as:

example.com.

Document your normalization approach.

57 — Normalize URLs

Avoid over-normalizing URLs.

These may differ:

https://example.com/login

https://example.com/Login

depending on application behavior.

58 — Normalize Hashes

File hashes should be:

LOWERCASE HEX

for consistent comparison.

Common lengths:

MD5
32 hex

SHA-1
40 hex

SHA-256
64 hex

59 — Validate Hash Format

Do not classify:

randomtext

as:

SHA-256

60 — Normalize CVE IDs

Example:

cve-2026-1234

to:

CVE-2026-1234

where valid.

61 — Validate CVSS Scores

Expected:

0.0–10.0

Reject or mark unknown values such as:

15

high

N/A

62 — Normalize Dates

Convert:

into a consistent internal representation.

Preferred:

ISO 8601

63 — Timezone Normalization

Convert event timestamps to:

UTC

while preserving original time if useful.

64 — Do Not Assume Local Time

A log without timezone is dangerous.

Example:

2026-08-29 10:00:00

Ask:

WHICH TIMEZONE?

65 — Daylight Saving Considerations

Timezones can change because of:

DST

Prefer timezone-aware timestamps.

66 — Timestamp Parsing Failure

If a timestamp cannot be parsed:

DO NOT
MAKE ONE UP

Mark:

timestamp_status = invalid

67 — Normalize Event Types

Sources may report:

4625

login_failed

failed_authentication

authentication_failure

Map them to a common type such as:

authentication_failure

68 — Preserve Source Event Code

Keep:

source_event_id = 4625

because source-specific context still matters.

69 — Normalize Protocols

Example:

TCP

tcp

6

could normalize to:

tcp

when mapping is clear.

70 — Normalize Ports

Convert:

"443"

to integer:

after validation.

71 — Validate Port Range

Expected:

0–65535

depending on your model.

72 — Normalize Cloud Providers

Use:

aws

azure

gcp

instead of:

Amazon Web Services

Microsoft Azure

Google Cloud Platform

internally.

73 — Normalize Cloud Scope

Create standard fields:

cloud_provider

cloud_scope_id

cloud_scope_name

resource_id

resource_type

74 — Preserve Native Resource ID

Cloud resource names can be duplicated.

Prefer:

FULL RESOURCE IDENTIFIER

where possible.

75 — Normalize Environment

Possible model:

production

staging

development

test

sandbox

unknown

76 — Normalize Criticality

Example internal model:

critical

high

medium

low

unknown

77 — Do Not Guess Criticality

If asset inventory has no criticality:

unknown

is safer than:

low

78 — Normalize Owner

Owner may represent:

TEAM

APPLICATION OWNER

BUSINESS OWNER

TECHNICAL OWNER

Separate these when required.

79 — Validate Owner Against Directory

Avoid free-text owner fields when possible.

Use stable identifiers such as:

TEAM ID

GROUP ID

80 — Deduplication

Security systems often generate duplicate records.

Causes:

RETRIES

MULTIPLE COLLECTORS

MULTIPLE SCANNERS

MESSAGE REDELIVERY

REIMPORT

81 — Define Deduplication Key

Examples:

ALERT ID

or:

ASSET
+
CVE
+
PORT
+
SCANNER

or:

USER
+
SOURCE IP
+
TIMESTAMP BUCKET
+
EVENT TYPE

82 — Avoid Weak Deduplication

This:

same hostname

is not enough to declare:

duplicate vulnerability

83 — Preserve Duplicate Evidence

Do not necessarily delete duplicates permanently.

Store:

DUPLICATE COUNT

SOURCE IDS

FIRST SEEN

LAST SEEN

84 — Deduplication vs Correlation

These are different.

DEDUPLICATION

asks:

IS THIS THE SAME RECORD?

CORRELATION

asks:

ARE THESE DIFFERENT RECORDS
RELATED?

85 — Example

5 identical alerts

may be:

DUPLICATES

but:

FAILED LOGIN
+
SUCCESSFUL LOGIN

are:

RELATED

not duplicates.

86 — Handle Late-Arriving Data

Events may arrive after newer events.

Do not assume ingestion order equals:

EVENT ORDER

Use:

EVENT TIMESTAMP

for timelines.

87 — Handle Out-of-Order Data

Example:

EVENT A
10:02

EVENT B
10:01

may be collected:

B AFTER A

because of pipeline delay.

88 — Event ID Design

Normalized records should have a stable identifier.

Options:

SOURCE ID

UUID

HASH-BASED FINGERPRINT

89 — Build Record Fingerprints

Possible fingerprint:

source
+
event_type
+
asset
+
user
+
timestamp

Hash the combination for tracking.

90 — Fingerprint Limitations

Do not use fingerprinting blindly where timestamps or fields change slightly between copies.

91 — Data Enrichment

After normalization, add context.

Examples:

ASSET OWNER

ASSET CRITICALITY

IDENTITY PRIVILEGE

DEPARTMENT

CLOUD PROVIDER

IOC CONTEXT

VULNERABILITY CONTEXT

92 — Enrichment Order

Recommended:

NORMALIZE FIRST

ENRICH SECOND

because matching normalized values is more reliable.

93 — Identity Enrichment

Example:

user = admin01

becomes:

user = admin01

department = Cloud Operations

privileged = true

account_status = enabled

94 — Asset Enrichment

Example:

asset = WEB01

becomes:

criticality = high

environment = production

internet_facing = true

owner = Application Team

95 — IOC Enrichment

Example:

203.0.113.25

may gain:

source

classification

confidence

last_seen

96 — Vulnerability Enrichment

An alert on:

WEB01

may gain:

critical_open_vulnerabilities = 1

97 — Enrichment Failure

If an enrichment source is unavailable:

KEEP THE RECORD

Mark:

enrichment_status = partial

98 — Graceful Degradation

Example:

IDENTITY
✓

ASSET
✓

IOC
✗

VULNERABILITY
✓

Result:

PARTIAL CONTEXT

not:

DROP ALERT

99 — Data Quality Dimensions

Measure:

COMPLETENESS

VALIDITY

CONSISTENCY

UNIQUENESS

FRESHNESS

ACCURACY

100 — Completeness

Question:

ARE REQUIRED FIELDS PRESENT?

Example:

98% OF ALERTS
HAVE ASSET ID

101 — Validity

Question:

DO VALUES MATCH EXPECTED FORMAT?

Example:

99.9% OF IP ADDRESSES
ARE VALID

102 — Consistency

Question:

IS THE SAME CONCEPT
REPRESENTED THE SAME WAY?

Example:

critical

instead of:

critical / CRIT / 5 / urgent

103 — Uniqueness

Question:

HOW MANY RECORDS
ARE DUPLICATES?

104 — Freshness

Question:

HOW OLD IS THE DATA?

105 — Accuracy

Harder question:

DOES THE DATA
REFLECT REALITY?

This often requires comparison with authoritative systems.

106 — Data Quality Score

A future model might track:

completeness

validity

freshness

enrichment coverage

Do not compress everything into one opaque score without explanation.

107 — Context Completeness

Example:

IDENTITY FOUND

ASSET FOUND

IOC CHECK COMPLETE

VULNERABILITY CONTEXT AVAILABLE

108 — Unknown Context Should Be Visible

Analysts should see:

ASSET CONTEXT:
UNKNOWN

rather than the field silently disappearing.

109 — Build a Data Quality Report

Example:

TOTAL RECORDS

VALID RECORDS

INVALID RECORDS

DUPLICATES

UNKNOWN ASSETS

UNKNOWN USERS

INVALID IPS

INVALID TIMESTAMPS

110 — Example Data Quality JSON

{
  "total_records": 1000,
  "valid_records": 960,
  "invalid_records": 20,
  "duplicates": 20,
  "unknown_assets": 15,
  "unknown_users": 8
}

111 — Set Quality Thresholds

Example:

INVALID RATE > 5%
→ INVESTIGATE PIPELINE

Thresholds should reflect your environment.

112 — Detect Sudden Data Changes

Example:

NORMAL:
10,000 EVENTS/HOUR

CURRENT:
50 EVENTS/HOUR

Possible causes:

COLLECTOR FAILURE

SOURCE FAILURE

NETWORK ISSUE

113 — Detect Volume Spikes

Example:

NORMAL:
10,000

CURRENT:
2,000,000

Possible:

EVENT STORM

LOOP

ATTACK

PARSER ERROR

114 — Source Health

Track each data source:

HEALTHY

DEGRADED

FAILED

UNKNOWN

115 — Collection Health Example

{
  "siem": "healthy",
  "identity": "healthy",
  "cmdb": "degraded",
  "threat_intel": "failed"
}

116 — Do Not Hide Collection Gaps

A report should state:

THREAT INTELLIGENCE SOURCE
UNAVAILABLE

rather than pretending:

NO IOC MATCHES FOUND

117 — Storage Model

Normalized security data may be stored in:

FILES

SQLITE

POSTGRESQL

DATA LAKE

SIEM INDEX

SEARCH PLATFORM

118 — Choose Storage Based on Need

For labs:

JSON

CSV

SQLITE

are often enough.

For enterprise scale:

CENTRAL DATABASE

SEARCH PLATFORM

DATA LAKE

may be more appropriate.

119 — Separate Raw and Normalized Storage

Example:

RAW
→ immutable source copy

NORMALIZED
→ query-ready data

120 — Store Schema Version

Each normalized dataset should record:

schema_version

121 — Store Pipeline Version

Record:

pipeline_version

so you know which code produced the output.

122 — Store Normalization Version

A change such as:

P2 → high

can alter analytics.

Track:

normalization_version

123 — Store Collection ID

Generate one:

collection_id

per pipeline run.

124 — Why Collection IDs Matter

They allow:

TRACEABILITY

ROLLBACK

REPROCESSING

AUDIT

125 — Reprocessing

If normalization logic changes, you may need to:

REPROCESS RAW DATA

using the new pipeline.

126 — Never Require Source Recollection if Avoidable

Preserving raw data allows:

REPROCESSING

without requesting the source system again.

127 — Database Constraints

Use:

PRIMARY KEYS

UNIQUE KEYS

NOT NULL

CHECK CONSTRAINTS

where appropriate.

128 — Example Constraints

risk_score
0–100

cvss
0–10

port
0–65535

129 — Index Frequently Queried Fields

Examples:

event_time

asset

user

source_ip

severity

130 — Do Not Over-Index

Indexes:

IMPROVE READ PERFORMANCE

but increase:

STORAGE

WRITE COST

MAINTENANCE

131 — Normalize Before Correlation

Do not correlate:

WEB01

with:

web01

before normalization if they should represent the same asset.

132 — Correlation Keys

Common:

ASSET ID

USER ID

SOURCE IP

SESSION ID

ALERT ID

CLOUD RESOURCE ID

133 — Prefer Stable Keys

Avoid relying only on:

DISPLAY NAME

if a stable identifier exists.

134 — Correlation Time Windows

Correlation should usually consider:

TIME

Example:

FAILED LOGIN
08:15

SUCCESS
08:18

is more meaningful than:

FAILED LOGIN
JANUARY

SUCCESS
AUGUST

135 — Build Normalized Security Event Schema

Recommended core fields:

event_id

event_time

collection_time

source_system

event_type

severity

user_id

username

asset_id

asset_name

source_ip

destination_ip

status

raw_reference

schema_version

136 — Optional Security Context Fields

privileged_user

asset_criticality

environment

internet_facing

ioc_confidence

open_vulnerability_count

cloud_provider

137 — Keep Context Separate Where Needed

Do not duplicate massive enrichment objects into every record if a relational model is more efficient.

Use:

EVENT TABLE

ASSET TABLE

IDENTITY TABLE

with joins.

138 — Security Data Model Example

EVENTS
   ↓
asset_id
   ↓
ASSETS

EVENTS
   ↓
user_id
   ↓
IDENTITIES

EVENTS
   ↓
indicator
   ↓
IOC_CONTEXT

139 — Data Lineage

For important records, maintain:

RAW SOURCE
      ↓
PARSER
      ↓
NORMALIZER
      ↓
ENRICHMENT
      ↓
FINAL RECORD

140 — Why Lineage Matters

If an analyst challenges a value:

WHY DOES THIS SAY
USER IS PRIVILEGED?

you should be able to answer:

SOURCE:
Identity Directory

COLLECTED:
06:00 UTC

141 — Protect Sensitive Data

Security datasets may contain:

USERNAMES

INTERNAL IPS

HOSTNAMES

VULNERABILITIES

EMAIL ADDRESSES

INCIDENT INFORMATION

Treat them as sensitive.

142 — Apply Access Control

Not every user needs access to:

RAW SECURITY TELEMETRY

Define:

VIEWER

ANALYST

ENGINEER

ADMIN

where appropriate.

143 — Encryption

Protect sensitive security data:

IN TRANSIT

AT REST

according to organizational requirements.

144 — Logging the Pipeline

Pipeline logs should include:

RUN ID

SOURCE

START TIME

RECORD COUNT

VALID COUNT

INVALID COUNT

DUPLICATE COUNT

FAILURE COUNT

END TIME

145 — Avoid Logging Raw Sensitive Payloads

Do not log entire:

AUTHENTICATION TOKEN

EMAIL BODY

API RESPONSE

unless necessary and approved.

146 — Operational Metrics

Track:

INGESTION RATE

VALIDATION FAILURE RATE

NORMALIZATION FAILURE RATE

DUPLICATE RATE

ENRICHMENT SUCCESS RATE

PROCESSING LATENCY

147 — Processing Latency

Calculate:

PROCESSING TIME
-
EVENT TIME

where appropriate.

148 — Detect Pipeline Delay

If normal delay is:

1 MINUTE

and current delay is:

45 MINUTES

investigate.

149 — Queue Backlog

If using message queues, track:

QUEUE DEPTH

OLDEST MESSAGE AGE

150 — Dead-Letter Queue

Invalid or repeatedly failing records can move to:

DEAD-LETTER QUEUE

for review.

151 — Do Not Lose Failed Records

Preferred:

FAILED RECORD
      ↓
QUARANTINE
      ↓
REVIEW

not:

FAILED RECORD
      ↓
DELETE

152 — Retry Strategy

Retries may help for:

TEMPORARY NETWORK ERROR

429

5XX

153 — No Infinite Retries

Use:

MAX RETRIES

BACKOFF

DEAD-LETTER

154 — Idempotent Ingestion

Running ingestion twice should not unintentionally create:

DOUBLE RECORDS

155 — Idempotency Keys

Use:

source_record_id

event_id

fingerprint

156 — Partial Batch Failure

Example:

10,000 RECORDS

9,950 SUCCESS

50 FAILED

Report:

PARTIAL SUCCESS

157 — Do Not Hide Partial Success

Avoid:

STATUS = SUCCESS

without:

FAILED RECORD COUNT

158 — Data Retention

Define retention separately for:

RAW DATA

NORMALIZED DATA

QUARANTINED DATA

PIPELINE LOGS

159 — Retention Should Have Purpose

Do not keep data:

FOREVER

without a documented need.

160 — Privacy and Regulatory Requirements

Security telemetry can overlap with:

EMPLOYEE DATA

CUSTOMER DATA

PERSONAL DATA

Consult applicable organizational requirements.

161 — Normalize Without Destroying Evidence

The rule is:

NORMALIZE FOR ANALYSIS

PRESERVE FOR INVESTIGATION

162 — Example

Source:

DOMAIN\Admin01

Normalized:

username = admin01
domain = DOMAIN

Do not simply discard:

DOMAIN

if it is meaningful.

163 — Parser Testing

Every parser should be tested against:

VALID SAMPLE

MISSING FIELD

EXTRA FIELD

INVALID VALUE

EMPTY RECORD

UNEXPECTED TYPE

164 — Regression Testing

When source formats change, ensure:

OLD VALID DATA

still processes as expected.

165 — Test Dataset

Maintain synthetic samples:

tests/data/
|
+-- valid/
|
+-- invalid/
|
+-- edge-cases/

166 — Source Format Change

If a vendor changes:

host_name

to:

device_name

the pipeline should:

DETECT SCHEMA CHANGE

rather than silently producing:

EMPTY ASSET VALUES

167 — Detect Unexpected Null Increase

Example:

ASSET NULL RATE

Yesterday:
1%

Today:
80%

This likely indicates a parser or source problem.

168 — Schema Drift Monitoring

Track:

NEW FIELD

REMOVED FIELD

TYPE CHANGE

ENUM CHANGE

169 — Type Validation

Example:

cvss

should be:

NUMBER

not:

OBJECT

170 — Enumeration Validation

Example severity should belong to:

critical

high

medium

low

informational

Anything else:

UNKNOWN

or quarantined according to policy.

171 — Baseline Data Volumes

Document expected:

EVENTS / HOUR

ALERTS / DAY

VULNERABILITIES / SCAN

ASSETS / INVENTORY

172 — Data Volume Anomaly

A sudden change may indicate:

ATTACK ACTIVITY

COLLECTION FAILURE

DUPLICATION LOOP

CONFIGURATION CHANGE

173 — Multi-Source Normalization

Example:

AWS CloudTrail

Azure Activity Log

Google Cloud Audit Logs

all normalize into:

CLOUD ACTIVITY EVENT

174 — Preserve Cloud-Native Fields

Common model does not mean:

DELETE PROVIDER-SPECIFIC DATA

Keep useful native fields in:

source_details

or related storage.

175 — Normalization Layer

Architecture:

AWS
 ↓
AWS ADAPTER
 ↓
COMMON MODEL

AZURE
 ↓
AZURE ADAPTER
 ↓
COMMON MODEL

GCP
 ↓
GCP ADAPTER
 ↓
COMMON MODEL

176 — Adapter Principle

Provider-specific parsing belongs in:

ADAPTER

Common analytics should consume:

NORMALIZED RECORD

177 — SIEM Normalization

For SIEM data, normalize concepts such as:

ALERT ID

RULE ID

EVENT TIME

USER

ASSET

SOURCE IP

SEVERITY

178 — EDR Normalization

Common fields:

DEVICE ID

HOSTNAME

USER

PROCESS

PARENT PROCESS

HASH

ALERT SEVERITY

179 — Vulnerability Normalization

Common fields:

FINDING ID

ASSET ID

CVE

SEVERITY

CVSS

STATUS

FIRST SEEN

LAST SEEN

DUE DATE

180 — Threat Intelligence Normalization

Common:

INDICATOR

TYPE

SOURCE

CONFIDENCE

FIRST SEEN

LAST SEEN

EXPIRATION

181 — Identity Normalization

Common:

IDENTITY ID

USERNAME

ACCOUNT TYPE

ENABLED

PRIVILEGED

DEPARTMENT

182 — Asset Normalization

Common:

ASSET ID

HOSTNAME

IP

ENVIRONMENT

CRITICALITY

OWNER

INTERNET FACING

183 — Detection Normalization

Common:

RULE ID

RULE NAME

VERSION

SEVERITY

DATA SOURCE

184 — Data Quality Runbook Trigger

Use this runbook when:

NEW DATA SOURCE ADDED

SCHEMA CHANGED

ALERT FIELDS MISSING

CORRELATION FAILS

DUPLICATE RATE SPIKES

EVENT COUNTS DROP

UNKNOWN ASSETS INCREASE

UNKNOWN USERS INCREASE

185 — Initial Triage of Data Pipeline Issue

Check:

SOURCE AVAILABLE?

COLLECTOR RUNNING?

AUTHENTICATION WORKING?

EXPECTED FILE/API DATA PRESENT?

SCHEMA CHANGED?

PARSER ERRORS?

DATABASE WRITES WORKING?

186 — Troubleshooting Flow

NO DATA
  ↓
SOURCE?
  ↓
COLLECTOR?
  ↓
NETWORK?
  ↓
AUTH?
  ↓
PARSER?
  ↓
NORMALIZER?
  ↓
STORAGE?

187 — If Source Is Down

Action:

MARK SOURCE FAILED

PRESERVE OTHER PIPELINE OUTPUT

NOTIFY OWNER

DO NOT REPORT FULL COVERAGE

188 — If Authentication Fails

For:

review:

TOKEN

EXPIRATION

CLIENT CONFIGURATION

189 — If Authorization Fails

For:

review:

REQUIRED PERMISSION

ROLE ASSIGNMENT

SCOPE

Do not immediately grant broad admin access.

190 — If Rate Limited

For:

use:

BACKOFF

RETRY-AFTER

LOWER REQUEST RATE

CACHING

191 — If Schema Changed

Stop and review:

WHAT CHANGED?

WHICH FIELDS?

WHICH TYPE?

WHICH VERSION?

192 — Do Not Patch Schema Blindly

Understand vendor/source changes before changing mappings.

193 — If Duplicate Rate Spikes

Investigate:

RETRY LOOP

FORWARDER DUPLICATION

MULTIPLE COLLECTORS

PAGINATION BUG

194 — If Unknown Assets Increase

Investigate:

CMDB FRESHNESS

HOSTNAME FORMAT

NEW CLOUD ASSETS

ASSET INVENTORY GAP

195 — If Unknown Users Increase

Investigate:

IDENTITY SYNC

DOMAIN FORMAT

SERVICE ACCOUNTS

DELETED ACCOUNTS

NORMALIZATION BUG

196 — Data Quality Escalation

Escalate when:

CRITICAL DATA SOURCE UNAVAILABLE

SCHEMA CHANGE BREAKS INGESTION

LARGE PORTION OF DATA INVALID

NORMALIZATION PRODUCES WRONG IDENTITIES

CORRELATION RESULTS ARE UNTRUSTWORTHY

197 — Stop Downstream Automation if Necessary

If data integrity is questionable:

PAUSE
HIGH-IMPACT AUTOMATION

until data quality is restored.

198 — Why This Matters

A response system acting on:

WRONG USER

WRONG ASSET

WRONG SEVERITY

can create serious business impact.

199 — Build Ingestion Health Dashboard

Useful metrics:

SOURCE HEALTH

LAST SUCCESSFUL COLLECTION

RECORD COUNT

INVALID RATE

DUPLICATE RATE

PROCESSING LATENCY

UNKNOWN ASSET RATE

UNKNOWN USER RATE

200 — Example Dashboard

SIEM
Healthy
Last Collection: 06:30 UTC

Identity
Healthy

CMDB
Degraded
Unknown Assets: 18%

IOC
Failed

201 — Alert on Stale Sources

Example:

NO SUCCESSFUL
CMDB COLLECTION
FOR 24 HOURS

should trigger operational review.

202 — Prevent Silent Failure

The worst pipeline state is:

BROKEN
BUT APPEARS HEALTHY

Use:

HEARTBEATS

EXPECTED RECORD COUNTS

LAST-SUCCESS TIMESTAMP

203 — Pipeline Heartbeat

Record:

run_id

start_time

end_time

status

records_processed

204 — Pipeline Run Status

Use:

success

partial_success

failed

not only:

success / failure

205 — Operational Run Record

Example:

{
  "run_id": "RUN-20260829-001",
  "status": "partial_success",
  "sources_expected": 6,
  "sources_successful": 5,
  "invalid_records": 7
}

206 — Runbook Execution Checklist

Before ingestion:

During ingestion:

After ingestion:

207 — Data Source Onboarding Template

For every new source document:

SOURCE NAME:

OWNER:

PURPOSE:

COLLECTION METHOD:

AUTHENTICATION:

REQUIRED PERMISSIONS:

FORMAT:

SCHEMA VERSION:

COLLECTION FREQUENCY:

EXPECTED RECORD VOLUME:

AUTHORITATIVE FIELDS:

NORMALIZATION RULES:

DEDUPLICATION KEY:

RETENTION:

FAILURE BEHAVIOR:

208 — Field Mapping Template

Example:

Source Field	Normalized Field	Rule
host_name	asset_name	Uppercase
userPrincipalName	username	Lowercase
risk	severity	Mapping table
eventTime	event_time	Convert to UTC
srcIp	source_ip	IP validation

209 — Normalization Mapping Governance

Mapping changes should be:

DOCUMENTED

VERSIONED

TESTED

REVIEWED

210 — Example Severity Mapping

SOURCE VALUE     INTERNAL

5                critical

4                high

3                medium

2                low

1                informational

Only use if supported by the source’s official definition.

211 — Example Status Mapping

SOURCE:

ACTIVE
NEW

→

open

SOURCE:

RESOLVED
CLOSED

→

closed

212 — Mapping Change Risk

Changing:

P2
FROM MEDIUM
TO HIGH

can alter:

DASHBOARDS

SCORES

ALERT QUEUES

AUTOMATION

Treat mapping changes as security engineering changes.

213 — Build Data Quality Investigation Template

ISSUE:

SOURCE:

FIRST OBSERVED:

EXPECTED:

ACTUAL:

INVALID RATE:

DUPLICATE RATE:

AFFECTED FIELDS:

DOWNSTREAM IMPACT:

ROOT CAUSE:

REMEDIATION:

VALIDATION:

214 — Example Issue

Issue:
Asset enrichment failure

Expected:
<5% unknown assets

Actual:
68% unknown assets

Cause:
EDR started reporting FQDN instead of short hostname

215 — Example Resolution

PRESERVE FQDN

EXTRACT SHORT NAME

USE ASSET ID WHERE AVAILABLE

UPDATE NORMALIZATION TESTS

REPROCESS AFFECTED DATA

216 — Data Pipeline Mental Model

When data arrives, ask:

WHERE DID IT COME FROM?
        ↓
CAN I TRUST THE SOURCE?
        ↓
IS THE RECORD COMPLETE?
        ↓
IS THE FORMAT VALID?
        ↓
WHAT DOES EACH FIELD MEAN?
        ↓
HOW SHOULD IT BE NORMALIZED?
        ↓
IS IT A DUPLICATE?
        ↓
WHAT CONTEXT CAN BE ADDED?
        ↓
HOW FRESH IS THE CONTEXT?
        ↓
WHAT IS UNKNOWN?
        ↓
CAN DOWNSTREAM AUTOMATION
SAFELY USE THIS DATA?

217 — Key Security Lesson

The most important lesson is:

NORMALIZATION
IS NOT JUST
FORMATTING

It is the process of ensuring that:

SECURITY DATA
HAS CONSISTENT
MEANING

218 — Final Security Equation

RAW DATA
   +
VALIDATION
   +
NORMALIZATION
   +
PROVENANCE
   +
DATA QUALITY
   +
CONTEXT
   =
TRUSTWORTHY SECURITY DATA

Runbook Outcome

After completing this runbook you should have:

DOCUMENTED DATA SOURCES

RAW DATA PRESERVATION

SCHEMA VALIDATION

NORMALIZED SECURITY RECORDS

DEDUPLICATION CONTROLS

DATA QUALITY METRICS

SOURCE HEALTH VISIBILITY

TRACEABLE PROVENANCE

Your security data is now ready for:

CORRELATION

ANALYTICS

RISK SCORING

REPORTING

SECURITY AUTOMATION

What’s Next?

➡️ Runbook 03 — Security API Integration and Failure Handling

The next runbook focuses on safely operating security API integrations.

You will build a repeatable operational procedure covering:

API DISCOVERY
      ↓
AUTHENTICATION
      ↓
LEAST PRIVILEGE
      ↓
REQUEST VALIDATION
      ↓
TIMEOUTS
      ↓
HTTP STATUS HANDLING
      ↓
RATE LIMITS
      ↓
PAGINATION
      ↓
RETRIES
      ↓
BACKOFF
      ↓
PARTIAL FAILURE
      ↓
API HEALTH
      ↓
RECOVERY

The goal will be to ensure that security automation remains reliable when external APIs are slow, unavailable, rate-limited, malformed, or partially failing.