Runbook 03 — Security API Integration and Failure Handling

Runbook Information

Runbook Type: Security API Operations / Security Automation
Difficulty: Intermediate → Advanced
Primary Audience: Security Engineers / SOC Engineers / Detection Engineers / Cloud Security Engineers / Security Automation Engineers
Security Domains: SOC / SOAR / Cloud Security / Vulnerability Management / Threat Intelligence / Identity Security
Execution Model: Secure and resilient API integration workflow
Primary Goal: Build and operate reliable security integrations without allowing API failures to create unsafe security decisions

Purpose

Modern security environments depend heavily on APIs.

Security automation may integrate with:

SIEM

SOAR

EDR / XDR

IDENTITY PROVIDERS

CLOUD PLATFORMS

VULNERABILITY MANAGEMENT

THREAT INTELLIGENCE

EMAIL SECURITY

ASSET INVENTORY

CASE MANAGEMENT

A typical workflow might look like:

SIEM ALERT
    ↓
IDENTITY API
    ↓
ASSET API
    ↓
VULNERABILITY API
    ↓
THREAT INTELLIGENCE API
    ↓
NORMALIZE
    ↓
CORRELATE
    ↓
ANALYST REVIEW

The workflow is only reliable if every API dependency is handled safely.

Core Principle

A successful API call does not automatically mean:

SUCCESSFUL
SECURITY OUTCOME

Likewise:

API FAILURE

does not mean:

NO SECURITY RISK

The integration must distinguish between:

NO RESULT

and:

COULD NOT CHECK

API Reliability Model

DISCOVER API
      ↓
DEFINE PURPOSE
      ↓
AUTHENTICATE
      ↓
AUTHORIZE
      ↓
BUILD REQUEST
      ↓
SEND REQUEST
      ↓
VALIDATE HTTP RESPONSE
      ↓
VALIDATE CONTENT
      ↓
HANDLE PAGINATION
      ↓
NORMALIZE DATA
      ↓
HANDLE PARTIAL FAILURE
      ↓
RETRY WHEN SAFE
      ↓
MONITOR
      ↓
AUDIT
      ↓
RECOVER

01 — Define the Security Use Case

Before integrating an API, define:

WHY ARE WE CALLING IT?

Examples:

Retrieve security alerts

Retrieve asset information

Check vulnerability findings

Collect cloud configuration

Enrich indicators

Retrieve identity context

Create approved security cases

02 — Define Required Operations

Document whether the integration needs:

READ

SEARCH

CREATE

UPDATE

DELETE

Prefer:

READ-ONLY

unless writes are genuinely required.

03 — Separate Read and Write Workflows

Recommended:

COLLECTION WORKFLOW
       ↓
READ-ONLY IDENTITY

and separately:

APPROVED RESPONSE
       ↓
CONTROLLED WRITE IDENTITY

04 — Identify API Owner

Document:

API NAME

SERVICE OWNER

SECURITY OWNER

SUPPORT CONTACT

DOCUMENTATION LOCATION

05 — Identify API Type

Common types:

REST

GRAPHQL

SOAP

RPC

WEBHOOK

EVENT API

This runbook primarily focuses on REST-style APIs, but the operational principles apply more broadly.

06 — Understand the Base Endpoint

Example training endpoint:

http://127.0.0.1:8080/api

Production integrations should use approved secure endpoints.

07 — Identify API Version

Examples:

/api/v1/

/api/v2/

Record the expected version.

08 — Why Version Matters

An API upgrade may change:

FIELD NAMES

DATA TYPES

PAGINATION

AUTHENTICATION

STATUS CODES

ENDPOINTS

09 — Review API Documentation

Before implementation understand:

AUTHENTICATION

ENDPOINTS

METHODS

PARAMETERS

RATE LIMITS

PAGINATION

ERROR RESPONSES

VERSIONING

10 — Authentication

Common authentication models include:

API TOKEN

OAUTH 2.0

SIGNED REQUEST

CLIENT CERTIFICATE

WORKLOAD IDENTITY

MANAGED IDENTITY

11 — Prefer Short-Lived Authentication

Where supported prefer:

SHORT-LIVED TOKEN

WORKLOAD IDENTITY

FEDERATED IDENTITY

MANAGED IDENTITY

over:

STATIC LONG-LIVED SECRET

12 — Never Hardcode Production Credentials

Avoid:

API_TOKEN = "real-production-secret"

13 — Use Secure Credential Sources

Depending on environment:

SECRET MANAGER

WORKLOAD IDENTITY

MANAGED IDENTITY

PROTECTED ENVIRONMENT VARIABLE

14 — Environment Variable Example

For a local training lab:

export SECURITY_API_TOKEN="training-token"

PowerShell:

$env:SECURITY_API_TOKEN = "training-token"

15 — Validate Credential Presence

Before sending requests:

TOKEN AVAILABLE?
       ↓
YES → CONTINUE

NO → STOP

Do not silently send unauthenticated requests unless the endpoint is intentionally public.

16 — Never Log Credentials

Do not log:

API TOKEN

PASSWORD

PRIVATE KEY

SESSION TOKEN

AUTHORIZATION HEADER

17 — Redact Sensitive Headers

Instead of:

Authorization:
Bearer eyJ...

log:

Authorization:
[REDACTED]

18 — Apply Least Privilege

Ask:

WHAT IS THE MINIMUM
API PERMISSION REQUIRED?

Example:

READ FINDINGS

does not require:

ADMINISTRATOR

19 — Scope Credentials

Where supported restrict credentials by:

RESOURCE

ACCOUNT

SUBSCRIPTION

PROJECT

API ENDPOINT

ACTION

20 — Separate Credentials by Purpose

Example:

SECURITY INVENTORY CLIENT
→ READ ONLY

CASE CREATION CLIENT
→ LIMITED WRITE

ADMINISTRATIVE CLIENT
→ SEPARATE / RESTRICTED

21 — Credential Rotation

Document:

OWNER

EXPIRATION

ROTATION PROCESS

REVOCATION PROCESS

22 — Credential Failure

Common authentication response:

401 Unauthorized

Treat it as:

AUTHENTICATION FAILURE

not:

NO SECURITY FINDINGS

23 — Authorization Failure

Common:

403 Forbidden

Usually means:

AUTHENTICATED
BUT NOT AUTHORIZED

24 — Do Not Solve 403 with Admin Access

Determine the exact missing permission.

Apply:

LEAST PRIVILEGE

25 — Build Requests Explicitly

Every request should define:

METHOD

URL

HEADERS

PARAMETERS

BODY

TIMEOUT

26 — Common HTTP Methods

GET
→ retrieve

POST
→ create/process

PUT
→ replace

PATCH
→ modify

DELETE
→ delete

27 — Treat Write Methods Carefully

Methods such as:

POST

PUT

PATCH

DELETE

may modify state.

Require stronger validation.

28 — Validate Destination

Before calling an API verify:

EXPECTED HOST?

EXPECTED PROTOCOL?

EXPECTED PATH?

29 — Use HTTPS for Production APIs

Preferred:

HTTPS

not:

HTTP

except controlled local training environments.

30 — TLS Validation

Do not casually disable:

CERTIFICATE VALIDATION

to make an integration work.

Investigate the trust problem.

31 — Request Headers

Typical headers:

Authorization

Accept

Content-Type

User-Agent

32 — User-Agent

Identify the automation.

Example:

GoHackersCloud-Security-Automation/1.0

This can help API owners identify traffic.

33 — Correlation ID

Generate:

request_id

or:

correlation_id

for each operation.

34 — Why Correlation IDs Matter

They connect:

AUTOMATION LOG
      ↓
API REQUEST
      ↓
API RESPONSE
      ↓
DOWNSTREAM ACTION

35 — Validate Query Parameters

Do not blindly pass uncontrolled values into requests.

Validate:

DATES

SEVERITY

STATUS

PAGE NUMBER

PAGE SIZE

36 — Encode Query Parameters Safely

Use your language’s URL/query-building functions rather than manually concatenating untrusted input.

37 — Request Body Validation

Before sending JSON verify:

REQUIRED FIELDS

DATA TYPES

ALLOWED VALUES

RESOURCE ID

38 — Preview Write Requests

For write operations:

BUILD REQUEST
      ↓
DISPLAY PROPOSED CHANGE
      ↓
APPROVAL
      ↓
SEND

39 — Use Dry Run Where Available

If the target API supports:

VALIDATE

DRY RUN

WHAT-IF

use it before impactful changes.

40 — Define Timeout

Every external request should have a timeout.

Avoid:

REQUEST
   ↓
WAIT FOREVER

41 — Timeout Types

Depending on client/library:

CONNECTION TIMEOUT

READ TIMEOUT

TOTAL TIMEOUT

42 — Choose Timeout Based on Service

A local API might respond in:

<1 SECOND

while a large analytics query may legitimately require longer.

Do not use one arbitrary timeout for every API.

43 — Timeout Is Not “No Result”

This:

THREAT INTELLIGENCE REQUEST
TIMED OUT

does not mean:

INDICATOR IS SAFE

It means:

RESULT UNKNOWN

44 — HTTP Status Classes

Mental model:

1XX
INFORMATIONAL

2XX
SUCCESS

3XX
REDIRECTION

4XX
CLIENT-SIDE CONDITION

5XX
SERVER-SIDE FAILURE

45 — 200 OK

Usually:

REQUEST SUCCEEDED

but you must still validate:

BODY

SCHEMA

CONTENT

46 — 201 Created

Usually indicates:

RESOURCE CREATED

For write operations, verify the resulting resource.

47 — 204 No Content

Can represent successful operation without response content.

Do not attempt JSON parsing automatically.

48 — 400 Bad Request

Usually indicates:

INVALID REQUEST

Review:

PARAMETERS

BODY

SCHEMA

49 — 401 Unauthorized

Review:

TOKEN PRESENT?

TOKEN EXPIRED?

TOKEN VALID?

AUTH FORMAT CORRECT?

50 — 403 Forbidden

Review:

ROLE

SCOPE

PERMISSIONS

RESOURCE ACCESS

51 — 404 Not Found

Could mean:

BAD ENDPOINT

WRONG RESOURCE ID

RESOURCE REMOVED

API VERSION CHANGED

52 — 409 Conflict

May indicate:

RESOURCE ALREADY EXISTS

STATE CONFLICT

DUPLICATE OPERATION

This is particularly relevant to idempotency.

53 — 429 Too Many Requests

Means:

RATE LIMIT

Do not continuously hammer the API.

54 — 500 Internal Server Error

Usually:

SERVER-SIDE FAILURE

May be retryable.

55 — 502 / 503 / 504

Often represent temporary service or gateway conditions.

These may be candidates for controlled retry.

56 — Build Status Handling Matrix

Status	Meaning	Typical Action
200	Success	Validate body
201	Created	Verify resource
204	Success/no body	Continue
400	Bad request	Fix request
401	Authentication	Stop/review credential
403	Authorization	Review permissions
404	Not found	Validate endpoint/resource
409	Conflict	Check idempotency/state
429	Rate limited	Backoff
500	Server error	Controlled retry
503	Unavailable	Retry/escalate

57 — Do Not Treat All Errors the Same

Bad design:

ANY ERROR
   ↓
RETRY

Better:

ERROR
 ↓
CLASSIFY
 ↓
RETRY?
STOP?
ESCALATE?

58 — Response Validation

After receiving:

HTTP 200

ask:

IS BODY PRESENT?

IS IT EXPECTED FORMAT?

IS JSON VALID?

IS SCHEMA VALID?

ARE REQUIRED FIELDS PRESENT?

59 — Successful HTTP ≠ Valid Data

Example:

200 OK

with:

{
  "error": "backend unavailable"
}

may still represent an unusable application response.

60 — Validate Content-Type

Expected:

application/json

but server returns:

text/html

This may indicate:

PROXY ERROR

LOGIN PAGE

SERVICE FAILURE

61 — JSON Parsing

Handle malformed JSON safely.

Example:

HTTP 200
      ↓
INVALID JSON
      ↓
RECORD FAILURE
      ↓
DO NOT CRASH ENTIRE WORKFLOW

62 — Validate Top-Level Structure

Expected:

{
  "results": []
}

Do not assume every JSON object is valid.

63 — Validate Required Fields

Example finding requires:

id

asset

severity

status

64 — Handle Missing Fields

Possible response:

QUARANTINE RECORD

MARK FIELD UNKNOWN

CONTINUE WITH WARNING

depending on field importance.

65 — Validate Types

Example:

score

expected:

NUMBER

but received:

"critical"

Treat as schema/data problem.

66 — Validate Enumerations

If expected severity is:

critical

high

medium

low

and API returns:

urgent

do not silently map it unless documented.

67 — Preserve Raw Response

For troubleshooting, preserve an approved raw response where appropriate.

Example:

raw-api-response.json

Be careful not to persist:

TOKENS

SENSITIVE HEADERS

UNNECESSARY SENSITIVE DATA

68 — Record API Provenance

Normalized data should contain:

source_api

endpoint

collection_time

api_version

69 — Pagination

Many APIs return only part of a dataset.

Example:

PAGE 1
100 RECORDS

PAGE 2
100 RECORDS

PAGE 3
45 RECORDS

70 — Common Pagination Models

PAGE NUMBER

OFFSET/LIMIT

CURSOR

CONTINUATION TOKEN

NEXT LINK

71 — Detect Pagination Model

Read the API documentation.

Do not assume:

page=1

works for every API.

72 — Page Number Model

Example:

?page=1&page_size=100

73 — Offset Model

Example:

?offset=0&limit=100

74 — Cursor Model

Example response:

{
  "results": [],
  "next_cursor": "abc123"
}

75 — Next-Link Model

Some APIs provide:

next

or:

nextLink

Follow only validated links expected for the trusted service.

76 — Pagination Safety

Define:

MAXIMUM PAGES

MAXIMUM RECORDS

TIME LIMIT

77 — Prevent Infinite Pagination

A broken API might return:

same next cursor
forever

Track previously seen cursors.

78 — Detect Duplicate Pages

Fingerprint page content or track stable record IDs where appropriate.

79 — Record Pagination Metrics

Track:

pages_processed

records_received

duplicate_records

80 — Validate Total Count

If API reports:

total = 1,000

but you receive:

mark:

INCOMPLETE COLLECTION

81 — Rate Limiting

APIs often restrict:

REQUESTS / SECOND

REQUESTS / MINUTE

REQUESTS / DAY

82 — Respect Rate Limits

Do not attempt to bypass legitimate service rate controls.

Instead:

SLOW DOWN

CACHE

BATCH

BACKOFF

83 — Retry-After

A 429 response may include:

Retry-After

Honor it where appropriate.

84 — Rate-Limit Headers

Some services expose:

X-RateLimit-Limit

X-RateLimit-Remaining

X-RateLimit-Reset

Names vary by provider.

85 — Monitor Remaining Quota

If available, record:

REMAINING REQUESTS

to prevent unexpected service disruption.

86 — Caching

Cache data when:

SOURCE CHANGES SLOWLY

MULTIPLE ALERTS NEED SAME DATA

Example:

ASSET CRITICALITY

may not need an API request for every alert.

87 — Define Cache TTL

Every cache needs:

EXPIRATION

Example:

IDENTITY CONTEXT
→ 15 minutes

ASSET INVENTORY
→ 1 hour

These are examples only.

88 — Stale Cache Risk

Caching reduces:

API LOAD

but can increase:

DATA STALENESS

Balance both.

89 — Retry Strategy

Retry only when failure is likely temporary.

Typical candidates:

TIMEOUT

429

500

502

503

504

90 — Usually Do Not Retry

Without a specific reason, avoid retrying:

91 — Retry Limit

Always define:

MAX RETRIES

92 — Exponential Backoff

Example:

ATTEMPT 1
   ↓
WAIT 1 SECOND

ATTEMPT 2
   ↓
WAIT 2 SECONDS

ATTEMPT 3
   ↓
WAIT 4 SECONDS

STOP

93 — Add Jitter

At scale, many workers retrying simultaneously can create:

THUNDERING HERD

Use randomized:

JITTER

where appropriate.

94 — Retry Budget

Define how much total time the workflow may spend retrying.

Example:

MAX RETRIES:
3

MAX TOTAL RETRY TIME:
30 seconds

95 — Retry Only Safe Operations

A GET is generally easier to retry than a write operation.

96 — Write Retry Risk

Example:

POST /tickets

times out.

Did the server:

CREATE THE TICKET

before the timeout?

Unknown.

Blind retry could create:

DUPLICATE TICKET

97 — Idempotency

Design writes so repeated requests do not create unintended duplicate effects.

98 — Idempotency Key

Where supported:

Idempotency-Key:
ALERT-1001

99 — Application-Level Deduplication

Before creating a resource:

SEARCH EXISTING
      ↓
EXISTS?
   ↙      ↘
 YES      NO
 ↓         ↓
USE       CREATE
EXISTING

100 — Stable Security IDs

Useful identifiers:

ALERT ID

FINDING ID

INCIDENT ID

EVENT ID

ASSET ID

101 — Write Verification

After:

POST

PATCH

PUT

verify the resulting state.

102 — Example

Request:

CREATE CASE

Response:

201 Created

Then retrieve:

CASE ID

and verify:

EXPECTED FIELDS

103 — Never Assume Response Means Outcome

Security automation should use:

REQUEST
   ↓
RESPONSE
   ↓
VERIFICATION

104 — Partial Failure

Multi-API workflows frequently partially fail.

Example:

IDENTITY API
✓

CMDB API
✓

VULNERABILITY API
✗

THREAT INTEL API
✓

105 — Do Not Discard Good Data

Result:

PARTIAL CONTEXT

rather than:

TOTAL FAILURE

when the use case permits.

106 — Mark Missing Context

Example:

{
  "identity_context": "available",
  "asset_context": "available",
  "vulnerability_context": "unavailable",
  "threat_intel_context": "available"
}

107 — Decision Safety

If a missing dependency is critical for the decision:

STOP AUTOMATED ACTION

and:

REQUIRE HUMAN REVIEW

108 — Example

If identity verification fails during proposed account containment:

DO NOT ASSUME
IDENTITY IS MALICIOUS

Escalate.

109 — Graceful Degradation

Design:

FULL CONTEXT
      ↓
NORMAL WORKFLOW

PARTIAL CONTEXT
      ↓
LIMITED ANALYSIS
      ↓
ANALYST REVIEW

110 — Critical Dependency Classification

Classify API dependencies:

OPTIONAL

IMPORTANT

CRITICAL

111 — Optional Dependency

Example:

GEOGRAPHIC IP ENRICHMENT

may not prevent basic alert triage.

112 — Critical Dependency

Example:

IDENTITY VERIFICATION

may be required before an identity response action.

113 — Dependency Matrix

API	Purpose	Criticality	Failure Behavior
Identity	User context	Critical	Stop write action
CMDB	Asset context	Important	Mark unknown
Threat Intel	IOC context	Important	Mark unavailable
Reporting	Dashboard	Optional	Queue report

114 — Circuit Breaker Concept

If an API repeatedly fails:

REQUEST
 ↓
FAIL
 ↓
FAIL
 ↓
FAIL
 ↓
STOP TEMPORARY CALLS

This prevents repeatedly overwhelming a failing service.

115 — Circuit Breaker States

Conceptually:

CLOSED
→ requests allowed

OPEN
→ requests blocked temporarily

HALF-OPEN
→ limited test requests

116 — Circuit Breaker Benefit

It can reduce:

UNNECESSARY LOAD

LONG TIMEOUT CHAINS

CASCADE FAILURE

117 — Dependency Isolation

One failing API should not unnecessarily crash:

ALL SECURITY AUTOMATION

Separate dependencies where possible.

118 — Bulkhead Concept

Architecture:

IDENTITY WORKERS
      │
      ├── isolated
      │
VULNERABILITY WORKERS
      │
      ├── isolated
      │
THREAT INTEL WORKERS

Failure in one pool should not consume all resources.

119 — Concurrency

Parallel API requests can improve performance but increase:

RATE-LIMIT RISK

RESOURCE USAGE

COMPLEXITY

120 — Limit Concurrency

Define:

MAX WORKERS

rather than unlimited parallel requests.

121 — Backpressure

If downstream systems cannot keep up:

SLOW INPUT

rather than building an unlimited queue.

122 — Queue-Based Integration

For larger systems:

EVENT
 ↓
QUEUE
 ↓
WORKER
 ↓
API
 ↓
RESULT

can improve resilience.

123 — Dead-Letter Queue

Repeated failures can move to:

DLQ

for review.

124 — DLQ Record

Include:

request_id

operation

attempt_count

last_error

timestamp

Do not include unnecessary secrets.

125 — Webhooks

Some APIs send data to you.

Flow:

SECURITY PLATFORM
       ↓
WEBHOOK
       ↓
YOUR RECEIVER
       ↓
VALIDATE
       ↓
QUEUE
       ↓
PROCESS

126 — Never Trust Webhook Input Automatically

Validate:

SOURCE

AUTHENTICITY

SIGNATURE

SCHEMA

TIMESTAMP

according to provider documentation.

127 — Replay Protection

Where supported, validate:

TIMESTAMP

EVENT ID

NONCE

to reduce duplicate/replayed processing.

128 — Webhook Acknowledgement

Do not perform long analysis before acknowledging if the provider expects a fast response.

Possible design:

WEBHOOK
 ↓
VALIDATE
 ↓
QUEUE
 ↓
ACKNOWLEDGE
 ↓
PROCESS ASYNC

129 — API Schema Drift

APIs change.

Possible changes:

NEW FIELD

REMOVED FIELD

RENAMED FIELD

TYPE CHANGE

ENUM CHANGE

130 — Detect Schema Drift

Track:

EXPECTED FIELDS

UNEXPECTED FIELDS

TYPE MISMATCHES

131 — Do Not Ignore Missing Critical Fields

Example:

asset_id

suddenly missing from 80% of records.

This should trigger:

PIPELINE INVESTIGATION

132 — Adapter Pattern

Use:

PROVIDER API
     ↓
PROVIDER ADAPTER
     ↓
COMMON SECURITY MODEL

133 — Why Adapters Help

If the API changes:

UPDATE ADAPTER

instead of rewriting:

ALL DOWNSTREAM ANALYTICS

134 — API Version Upgrade

Before changing versions:

READ CHANGELOG

TEST NEW VERSION

COMPARE OUTPUT

UPDATE MAPPINGS

RUN REGRESSION TESTS

135 — Parallel Version Testing

Where practical:

V1
 ↓
NORMALIZED OUTPUT A

V2
 ↓
NORMALIZED OUTPUT B

COMPARE

136 — API Deprecation

Track:

DEPRECATION DATE

MIGRATION DEADLINE

OWNER

REPLACEMENT VERSION

137 — Observability

Monitor:

REQUEST COUNT

SUCCESS RATE

ERROR RATE

LATENCY

TIMEOUT RATE

RETRY RATE

429 RATE

5XX RATE

138 — API Health Dashboard

Example:

Identity API
Success: 99.9%
Latency: 180 ms
429: 0

Threat Intel API
Success: 82%
Latency: 3.2 s
Timeouts: Elevated

139 — Availability Is Not Enough

An API may be:

UP

but returning:

STALE

INCOMPLETE

INVALID

data.

Monitor:

DATA QUALITY

as well.

140 — Data Freshness

Track:

LAST SUCCESSFUL COLLECTION

LATEST SOURCE TIMESTAMP

141 — Latency Percentiles

At larger scale monitor:

P50

P95

P99

rather than only average latency.

142 — Error Budget Concept

Define acceptable reliability for the integration.

Example:

TARGET:
99.5% successful enrichment

Use organizational requirements rather than arbitrary numbers.

143 — Structured API Logging

Example:

{
  "request_id": "REQ-1001",
  "service": "asset-api",
  "method": "GET",
  "status": 200,
  "duration_ms": 184
}

144 — Do Not Log Full URLs Blindly

URLs can contain:

TOKENS

USER DATA

QUERY SECRETS

Redact sensitive parameters.

145 — Log Failure Category

Use:

authentication

authorization

timeout

rate_limit

server_error

schema_error

validation_error

146 — Avoid Only Logging Exception Text

Structured categories make it easier to:

SEARCH

TREND

ALERT

147 — Request Metrics

Track:

api_requests_total

api_requests_success

api_requests_failed

api_retries_total

api_timeouts_total

148 — Dependency Metrics

Track separately per:

SERVICE

ENDPOINT

OPERATION

149 — Alert Thresholds

Examples:

ERROR RATE SPIKE

NO SUCCESSFUL REQUEST

HIGH LATENCY

REPEATED 401

REPEATED 429

SCHEMA FAILURE

150 — Repeated 401

May indicate:

TOKEN EXPIRED

ROTATION FAILURE

AUTH CONFIGURATION CHANGE

151 — Repeated 403

May indicate:

PERMISSION CHANGE

ROLE REMOVAL

SCOPE CHANGE

152 — Repeated 404

May indicate:

API VERSION REMOVED

RESOURCE PATH CHANGED

153 — Repeated 429

May indicate:

REQUEST VOLUME INCREASE

BAD CACHE

CONCURRENCY TOO HIGH

PAGINATION LOOP

154 — Repeated 5XX

Possible:

SERVICE OUTAGE

PROVIDER INCIDENT

DEPENDENCY FAILURE

155 — Escalation

Escalate when:

CRITICAL API UNAVAILABLE

AUTHENTICATION BROKEN

SCHEMA CHANGE BREAKS PROCESSING

DATA QUALITY UNTRUSTWORTHY

HIGH-IMPACT ACTION CANNOT BE VERIFIED

156 — API Outage Procedure

When an important API is unavailable:

CONFIRM FAILURE
      ↓
CLASSIFY DEPENDENCY
      ↓
STOP UNSAFE ACTIONS
      ↓
ENABLE DEGRADED MODE
      ↓
RECORD COVERAGE GAP
      ↓
NOTIFY OWNER
      ↓
MONITOR RECOVERY

157 — Confirm the Failure

Check:

ONE REQUEST?

MULTIPLE REQUESTS?

MULTIPLE WORKERS?

HEALTH ENDPOINT?

SERVICE STATUS?

Avoid declaring an outage from one transient timeout.

158 — Degraded Mode

Example:

THREAT INTEL API DOWN

Workflow may continue:

ALERT TRIAGE

but mark:

THREAT INTELLIGENCE
UNAVAILABLE

159 — Disable Unsafe Response

If a required verification API is unavailable:

PAUSE
AUTOMATED RESPONSE

while allowing:

READ-ONLY COLLECTION

where safe.

160 — Recovery

When service returns:

HEALTH CHECK
      ↓
TEST REQUEST
      ↓
VALIDATE SCHEMA
      ↓
VALIDATE DATA
      ↓
RESUME LIMITED TRAFFIC
      ↓
MONITOR
      ↓
FULL RECOVERY

161 — Do Not Resume Full Load Immediately

Prefer:

CANARY REQUESTS

before restoring large request volume.

162 — Process Backlog Carefully

After an outage you may have:

10,000 QUEUED REQUESTS

Do not send them all simultaneously.

Use:

RATE CONTROL

BATCHING

PRIORITIZATION

163 — Backlog Prioritization

Process:

CRITICAL CURRENT EVENTS

before:

LOW-PRIORITY STALE ENRICHMENT

when appropriate.

164 — Stale Request Review

Some queued actions may no longer be relevant after a long outage.

Revalidate before processing.

165 — Recovery Verification

Confirm:

REQUEST SUCCESS

DATA VALID

LATENCY NORMAL

ERROR RATE NORMAL

BACKLOG DECREASING

166 — API Incident Record

Document:

START TIME

SERVICE

IMPACT

FAILURE MODE

AFFECTED WORKFLOWS

SAFETY ACTIONS

RECOVERY TIME

ROOT CAUSE

FOLLOW-UP

167 — API Failure Matrix

Maintain:

Failure	Retry?	Workflow
Timeout	Usually limited	Degraded
400	No	Fix request
401	No	Auth review
403	No	Permission review
404	Usually no	Endpoint review
429	Yes, controlled	Backoff
500	Limited	Retry/degraded
Invalid JSON	Usually no	Schema/data review
Missing critical field	No	Quarantine

168 — Safe Write Workflow

For any API that modifies security state:

DETECT
   ↓
VALIDATE
   ↓
ENRICH
   ↓
PROPOSE ACTION
   ↓
PREVIEW
   ↓
HUMAN APPROVAL
   ↓
WRITE
   ↓
VERIFY
   ↓
AUDIT

169 — Example Case Creation

Low-impact example:

CRITICAL VULNERABILITY
       ↓
VALIDATE FINDING
       ↓
MAP OWNER
       ↓
CHECK EXISTING CASE
       ↓
CREATE CASE
       ↓
VERIFY CASE

170 — Example Identity Response

Higher-impact:

SUSPICIOUS IDENTITY ALERT
        ↓
VERIFY IDENTITY
        ↓
COLLECT EVIDENCE
        ↓
ANALYST REVIEW
        ↓
APPROVED RESPONSE
        ↓
VERIFY STATE

171 — Avoid Autonomous Destructive API Workflows

Do not design generic automation that automatically:

DELETES CLOUD RESOURCES

DISABLES SECURITY LOGGING

REMOVES SECURITY CONTROLS

MASS-CHANGES IAM

MODIFIES FIREWALLS

without appropriate authorization, governance, testing, and safeguards.

172 — Security API Threat Model

Consider threats against:

API CREDENTIAL

API CLIENT

CONFIGURATION

REQUEST DATA

RESPONSE DATA

CACHE

QUEUE

LOGS

173 — Credential Theft

If a token is compromised:

REVOKE

ROTATE

INVESTIGATE USE

REVIEW PERMISSIONS

174 — Excessive Permissions

If integration has unnecessary privileges:

REDUCE SCOPE

Do not wait for an incident.

175 — Malicious or Unexpected Response Data

Treat API response data as:

UNTRUSTED INPUT

Validate before:

DATABASE INSERT

REPORTING

DOWNSTREAM ACTION

176 — API Supply Chain Consideration

External services introduce:

DEPENDENCY RISK

Understand:

WHAT DATA IS SENT?

WHAT ACCESS EXISTS?

WHAT HAPPENS IF SERVICE FAILS?

177 — Data Minimization

Send only required data.

Do not send an entire incident record when the API only requires:

ONE HASH

178 — Sensitive Data

Be cautious with:

USER INFORMATION

INTERNAL HOSTNAMES

PRIVATE IPS

INCIDENT DETAILS

CUSTOMER DATA

179 — API Response Storage

Do not retain full responses forever without a reason.

Define:

RETENTION

ACCESS CONTROL

ENCRYPTION

180 — API Integration Testing

Test:

SUCCESS

AUTH FAILURE

PERMISSION FAILURE

NOT FOUND

RATE LIMIT

TIMEOUT

SERVER ERROR

INVALID JSON

MISSING FIELD

WRONG TYPE

EMPTY RESPONSE

181 — Test Pagination

Include:

ONE PAGE

MULTIPLE PAGES

EMPTY PAGE

REPEATED CURSOR

MISSING NEXT CURSOR

182 — Test Retry Logic

Verify:

RETRYABLE ERROR
→ RETRIES

NON-RETRYABLE ERROR
→ DOES NOT RETRY

183 — Test Backoff

Confirm retries are not:

IMMEDIATE

and do not overload the service.

184 — Test Write Idempotency

Run the same synthetic write request twice.

Expected:

ONE LOGICAL RESULT

not:

TWO DUPLICATE RESULTS

185 — Test Partial Failure

Example:

3 APIS

2 SUCCESS

1 FAILURE

Verify output clearly states:

PARTIAL CONTEXT

186 — Test Recovery

Simulate:

API DOWN
      ↓
API RECOVERS

Verify:

CIRCUIT RECOVERS

BACKLOG PROCESSES

NO REQUEST STORM

187 — Test Schema Drift

Change synthetic response:

severity

to:

risk_level

The integration should:

DETECT FAILURE

rather than silently produce bad data.

188 — Test Unknown Enumeration

Return:

severity = urgent

Verify:

UNKNOWN / QUARANTINE

according to policy.

189 — Test Missing Token

Expected:

FAIL BEFORE REQUEST

190 — Test Wrong Token

Expected:

401
      ↓
NO RETRY LOOP
      ↓
AUTHENTICATION ERROR

191 — Test Service Offline

Expected:

CONNECTION FAILURE
      ↓
BOUNDED RETRY
      ↓
DEGRADED MODE

192 — Test Slow API

Simulate response exceeding timeout.

Expected:

TIMEOUT
      ↓
CONTROLLED RETRY
      ↓
FAILURE STATUS

193 — Test Invalid Endpoint

Expected:

404
      ↓
NO BLIND RETRY

194 — Test 429

Expected:

429
 ↓
READ RETRY-AFTER
 ↓
WAIT
 ↓
RETRY WITHIN BUDGET

195 — Test 500

Expected:

500
 ↓
CONTROLLED RETRY
 ↓
BACKOFF
 ↓
DEGRADED MODE

if still failing.

196 — API Readiness Checklist

Before production:

197 — Operational API Checklist

During each integration run:

198 — API Failure Investigation Template

Use:

INCIDENT ID:

API:

ENDPOINT:

FIRST FAILURE:

LAST SUCCESS:

HTTP STATUS:

ERROR CATEGORY:

REQUEST ID:

AFFECTED WORKFLOW:

AFFECTED DATA:

DEPENDENCY CRITICALITY:

RETRY ATTEMPTS:

DEGRADED MODE:

SECURITY IMPACT:

OWNER:

RECOVERY STATUS:

199 — API Change Review Template

Before API migration:

CURRENT VERSION:

NEW VERSION:

DEPRECATION DATE:

AUTHENTICATION CHANGES:

ENDPOINT CHANGES:

SCHEMA CHANGES:

PAGINATION CHANGES:

RATE LIMIT CHANGES:

TEST RESULTS:

ROLLBACK PLAN:

APPROVER:

200 — Integration Health Template

SERVICE:

STATUS:

LAST SUCCESS:

SUCCESS RATE:

ERROR RATE:

AVERAGE LATENCY:

P95 LATENCY:

TIMEOUT RATE:

429 RATE:

5XX RATE:

DATA FRESHNESS:

QUEUE DEPTH:

OWNER:

201 — API Failure Decision Tree

API REQUEST FAILED
       ↓
WHAT TYPE?
       ↓
 ┌─────┼─────────┬─────────┐
 ↓     ↓         ↓         ↓
401   403       429       5XX
 ↓     ↓         ↓         ↓
AUTH  PERMISSION BACKOFF   RETRY
REVIEW REVIEW     ↓         ↓
                 LIMIT     BOUNDED
                 RETRY     RETRY

Then:

STILL FAILED?
      ↓
IS DEPENDENCY CRITICAL?
      ↓
 ┌────┴────┐
 ↓         ↓
YES        NO
 ↓          ↓
STOP       DEGRADED
UNSAFE     MODE
ACTION
 ↓
HUMAN
REVIEW

202 — Security Decision Model

Never design:

API RETURNED NOTHING
        ↓
NO RISK

Use:

API RETURNED
VALID EMPTY RESULT
        ↓
NO MATCH FROM THIS SOURCE

versus:

API FAILED
        ↓
UNKNOWN

203 — Reliability Mental Model

REQUEST
  ↓
DID IT CONNECT?
  ↓
DID AUTH WORK?
  ↓
WAS ACCESS ALLOWED?
  ↓
DID HTTP SUCCEED?
  ↓
IS CONTENT VALID?
  ↓
IS SCHEMA VALID?
  ↓
IS DATA COMPLETE?
  ↓
IS PAGINATION COMPLETE?
  ↓
IS DATA FRESH?
  ↓
CAN I TRUST IT
FOR THIS DECISION?

204 — Security API Safety Equation

SECURE AUTHENTICATION
        +
LEAST PRIVILEGE
        +
VALIDATION
        +
TIMEOUTS
        +
CONTROLLED RETRIES
        +
RATE-LIMIT HANDLING
        +
IDEMPOTENCY
        +
OBSERVABILITY
        +
SAFE FAILURE
        =
RELIABLE SECURITY API
INTEGRATION

205 — Key Lesson

Remember:

NO MATCH
IS NOT THE SAME AS
NO DATA

and:

NO DATA
IS NOT THE SAME AS
API FAILURE

206 — Final Mental Model

Whenever your security automation calls an API, ask:

WHY AM I CALLING IT?
        ↓
WHAT ACCESS DO I NEED?
        ↓
HOW DO I AUTHENTICATE?
        ↓
IS THE CONNECTION TRUSTED?
        ↓
WHAT IS MY TIMEOUT?
        ↓
WHAT HTTP STATUS DID I GET?
        ↓
IS THE RESPONSE VALID?
        ↓
IS THE SCHEMA EXPECTED?
        ↓
IS THERE PAGINATION?
        ↓
AM I RATE LIMITED?
        ↓
SHOULD I RETRY?
        ↓
IS THE OPERATION IDEMPOTENT?
        ↓
IS ANY CONTEXT MISSING?
        ↓
CAN I SAFELY MAKE
THE SECURITY DECISION?

Runbook Outcome

After completing this runbook, your security API integrations should be designed to:

AUTHENTICATE SECURELY

USE LEAST PRIVILEGE

VALIDATE REQUESTS

VALIDATE RESPONSES

HANDLE TIMEOUTS

CLASSIFY HTTP FAILURES

RESPECT RATE LIMITS

PROCESS PAGINATION

RETRY SAFELY

PREVENT DUPLICATE WRITES

HANDLE PARTIAL FAILURE

OPERATE IN DEGRADED MODE

MONITOR DEPENDENCIES

RECOVER SAFELY

The objective is not simply:

MAKE THE API CALL WORK

The objective is:

MAKE THE SECURITY WORKFLOW
RELIABLE WHEN THE API
DOES NOT WORK

What’s Next?

➡️ Runbook 04 — SOC Alert Enrichment, Correlation and Triage

The next runbook moves from reliable data collection and API integration into operational SOC decision support.

You will build a repeatable process for:

SECURITY ALERT
      ↓
VALIDATE
      ↓
NORMALIZE
      ↓
DEDUPLICATE
      ↓
IDENTITY CONTEXT
      ↓
ASSET CONTEXT
      ↓
IOC CONTEXT
      ↓
VULNERABILITY CONTEXT
      ↓
CORRELATE
      ↓
PRIORITIZE
      ↓
TRIAGE RECOMMENDATION
      ↓
ANALYST REVIEW
      ↓
ESCALATE / CLOSE

The goal will be to transform raw security alerts into context-rich, explainable, analyst-ready investigations while keeping final security decisions and disruptive response actions under appropriate human control.