OneNym.idDiscuss deployment
English

THE HARDENED STACK / Chapter 14

Incident Response Playbooks

An incident response plan says contain the threat; someone on the service desk has to know what that means on a keyboard. This chapter covers the runbooks a small IT team follows in the first hours, from triage to containment and evidence.

This page summarises the chapter. The full text is in the book.

Phases, and why triage comes first

The chapter opens on a Tuesday morning when Meridian's service desk receives eleven reports of a fake invoice email whose link captured passwords. Lucy, on the desk, works a one-page runbook in order, and within about an hour the team knows that only one account is truly compromised; the other ten are a training opportunity. The book places these runbooks alongside the board-approved incident plan from Book 1: the plan decides who decides and who speaks, and the runbooks tell IT staff what to do.

The lifecycle, adapted from NIST, runs prepare, detect, triage, contain, eradicate, recover and learn. Preparation decides most of the quality of the response. Triage is the phase most often skipped and the one that prevents both over- and under-reaction. The most common failure is collapsing triage, containment and eradication into one motion, wiping and rebuilding a server before anyone recorded what was on it. Phases can loop, but evidence capture always comes before change. The author also recommends writing down who owns each phase so that tasks do not fall between people.

A small severity scale and a triage runbook

The book proposes four levels. SEV-1 is active, spreading compromise or an outage of a critical service, such as ransomware encrypting shares or confirmed data exfiltration, and engages leadership, legal and the insurer. SEV-2 is confirmed compromise contained to one account or host, or a serious vulnerability under active exploitation, worked within hours. SEV-3 is a suspicious event without confirmed spread, such as one phishing click where MFA held. SEV-4 covers policy violations and single blocked detections. Each level should list who is notified, how fast and who may declare it. At Meridian anyone may raise severity, but lowering it needs a second person, and responders unsure between two levels pick the higher one. The chapter also separates impact from urgency: urgency drives speed, impact drives how far you escalate.

For triage, the author recommends looking for evidence that the incident is worse than it seems: new inbox rules, new MFA methods, OAuth consents, access to unusual resources, sign-ins from hosting ranges, and the identity's effective permissions. The sample runbook for a suspected account compromise opens an incident record, identifies accounts and hosts, reviews 24 hours of sign-ins, checks inbox rules and OAuth grants, enumerates roles (raising to SEV-2 if any are privileged), and records the severity decision and who made it. It explicitly forbids resets, deletions and reimaging during triage. At Meridian, phishing-resistant keys stopped ten of the eleven users at the look-alike site; the eleventh showed a sign-in from a foreign hosting provider and, minutes later, an inbox rule hiding messages about invoices and payments. That case was raised to SEV-2 and captured before anyone touched it.

Containment without destruction

For an account, the three moves are blocking sign-in, revoking active sessions and refresh tokens, and resetting the credential. Revocation is the step most often forgotten, because a stolen refresh token can keep producing access after the account is blocked. The book's PowerShell sketch requires an explicit account name and incident reference, refuses to act on break-glass accounts, and asks the operator to confirm, because bulk scripts without those guards have locked teams out of their own tenants. The sweep then removes the recorded inbox rules, revokes OAuth grants made during the incident window, and removes any newly registered MFA methods. If the account held privileged roles, everything those roles could reach is in scope.

For a host, EDR network isolation plays the same role: it cuts the attacker off while the management console can still reach the machine and memory is preserved. Powering off is reserved for cases where running is actively dangerous. Containment should be reversible wherever possible.

Evidence and the common playbooks

Evidence handling for a small team is a set of habits: capture before you change, keep logs in a central store out of the attacker's reach and extend retention when an incident opens, and record for each item what it is, where and when (in UTC) it was captured, by whom, and a SHA-256 hash for files. Originals stay untouched in a pre-provisioned, access-controlled evidence store, and legal contacts are called early if crime or litigation is possible.

Four playbooks cover most incidents. A phishing report searches all mailboxes, finds who clicked and who entered credentials, runs the account playbook for those, removes the message and blocks the sender and domain. A compromised account adds a final review of everything the account could touch during the window. Ransomware is SEV-1 by default: isolate hosts without powering them off, find patient zero, escalate rather than paying or deciding alone, and avoid restoring into a dirty environment. A lost device is decided largely by whether disk encryption and MDM were in place, followed by remote lock or wipe and session revocation. The author notes the first fifteen minutes barely change between types; only the containment lever does.

Communication and an honest on-call model

Runbooks should contain explicit escalation triggers, so the responder recognises a condition and makes the call while decisions about statements, regulators and ransom stay with the people the plan names. Keep one incident channel, one incident record and one named scribe, agree an out-of-band channel in case email and chat are compromised, and keep the first external message simple and honest. Finally, the book asks small teams to be realistic about nights: a sustainable rota, a short list of conditions that justify waking someone, and a managed detection and response service or retained incident-response firm for the hours the team cannot cover.

KEY POINTS

Key points

  • Write a one-page SEV-1 to SEV-4 scale with observable criteria and a named owner who may declare each level.
  • Draft numbered runbooks for phishing, compromised account, ransomware and lost device, and keep a copy off-system.
  • Build a guarded disable-and-revoke procedure with a break-glass exclusion list and a confirmation step.
  • Adopt an evidence-note template and make capture before change a required step.
  • Agree escalation triggers with leadership and an out-of-band communication channel.
  • Decide who covers nights and whether a retained incident-response firm or MDR service is needed.

QUESTIONS

Common questions

What are the phases of incident response?
A common version adapted from NIST is prepare, detect, triage, contain, eradicate, recover and learn. Preparation decides most of the quality of the response, and triage, deciding how serious an event really is, is the phase teams most often skip. The phases can loop, but evidence should always be captured before anything is changed.
How should incident severity levels be defined for a small IT team?
The Hardened Stack suggests four levels: SEV-1 for active, spreading compromise or a critical outage such as ransomware; SEV-2 for confirmed compromise contained to one account or host, or active exploitation of a serious flaw; SEV-3 for a suspicious event with no confirmed spread; and SEV-4 for policy violations and single blocked detections. Each level should state who is notified, how quickly, and who may declare it. When unsure between two levels, choose the higher one.
How do you safely disable a compromised Microsoft 365 or Entra ID account?
Block sign-in, revoke all active sessions and refresh tokens, and reset the credential; blocking alone is not enough because a stolen refresh token can keep issuing access. Then remove malicious inbox rules after recording them, revoke OAuth grants made during the incident, and remove any MFA methods the attacker registered. Any script used for this should require an explicit account and incident reference, exclude break-glass accounts and ask for confirmation.
Why shouldn't you reset passwords or delete inbox rules during triage?
Resetting, deleting or reimaging during triage destroys evidence you will need to understand how the attacker got in and what they did, and it can alert an attacker who is watching. Triage should only observe, record and decide severity. Capture screenshots, exports and logs first, then move to the containment runbook.
What evidence should IT capture during a security incident?
For an account compromise, typical items are the account's sign-in and audit logs, an export and screenshots of inbox rules, OAuth consent records, MFA registration changes, and relevant host logs, with a disk image for SEV-1 or SEV-2 incidents. For each item record what it is, where and when in UTC it was captured, who captured it, and a SHA-256 hash for files. Work on copies and keep originals in an access-controlled evidence store.
What should IT do first when ransomware is detected?
Treat it as SEV-1 and isolate affected hosts from the network without powering them off, since powering off can destroy memory evidence. Identify the first infected system and how it spread, preserve evidence, and escalate to leadership, legal and the insurer under the incident plan rather than paying or deciding alone. Do not restore data into an environment that may still be compromised.