9.3 Containment, Eradication, Recovery, and Root Cause Analysis

Tactical Containment Strategies: Short-Term vs. Long-Term Actions

Once an incident's blast radius is scoped, the incident response team executes containment. The objective of containment is to stop the adversary's lateral movement, halt active data exfiltration, and prevent further operational degradation while preserving evidentiary integrity and minimizing collateral business impact. Containment is partitioned into short-term tactical actions and long-term architectural controls.

Short-Term Containment (Immediate Blast Radius Containment)

Short-term containment comprises immediate, decisive interventions designed to isolate compromised assets within minutes of validation:

• Host Network Isolation via EDR: The gold standard for endpoint containment. Modern EDR agents (e.g., CrowdStrike Falcon, Microsoft Defender for Endpoint, SentinelOne) inject kernel-level filtering rules that sever all inbound and outbound TCP/IP traffic on the host, while maintaining a single encrypted management tunnel to the EDR cloud console. This allows analysts to interact live with the operating system, run forensic commands, and pull memory dumps while blocking all adversary command-and-control (C2) and East-West lateral movement.
• Network Port and VLAN Isolation: For legacy endpoints or network appliances lacking EDR agents, network engineers execute switch-port quarantine (RFC 3580 802.1X dynamic VLAN assignment) to drop the host into an isolated sandbox VLAN with zero routing to enterprise subnets. Alternatively, perimeter firewalls apply egress drop rules to null-route (blackhole) adversary C2 IP addresses.
• Identity and Session Invalidation: Threat actors maintain access via stolen session tokens and Kerberos tickets even if passwords are changed. Short-term identity containment requires:
  • Disabling the Active Directory user account: Set-ADUser -Identity <username> -Enabled $false.
  • Purging active Kerberos Ticket Granting Tickets (TGT): Executing klist purge across active sessions.
  • Revoking cloud sessions: Microsoft Graph POST /users/{id}/revokeSignInSessions invalidates the user's Entra refresh tokens and Entra browser session cookies after propagation. Already issued access tokens and application-owned sessions may require additional controls, so responders verify the affected services and do not describe revocation as instantaneous or universal.

Long-Term Containment (Sustainable Operational Containment)

Long-term containment establishes sustainable architectural barriers that allow business units to continue operating while the CSIRT conducts exhaustive root-cause investigations:

• Network Microsegmentation: Deploying temporary firewall access control lists (ACLs) or software-defined networking (SDN) rules that restrict traffic between critical business tiers (e.g., completely isolating the ERP database from corporate workstation subnets).
• Virtual Patching and Web Application Firewalls (WAF): If the intrusion exploited a zero-day or unpatched web application vulnerability that cannot be immediately remediated without breaking production code, security engineers deploy targeted WAF regex rules or modsecurity signatures to drop exploit payloads at the network edge.
• Honeypot Diversion & Canary Infrastructure: In advanced persistent threat (APT) scenarios, the CSIRT may intentionally divert adversary traffic to deceptive shadow infrastructure or sandboxes to observe adversary tradecraft, collect secondary C2 infrastructure indicators, and assess intent before executing final eradication.

Containment Options Decision Matrix

---

Evidence Preservation Prior to Containment: The Golden Rule

Powering off, rebooting, or re-imaging a system can destroy volatile evidence and should be an explicit, documented response decision. Evidence value is one factor alongside safety, ongoing harm, legal authority, service impact, and the feasibility and trustworthiness of live collection.

Balance Evidence Preservation with Containment

When safe and authorized, preserve the volatile artifacts needed to answer the investigation before a destructive action. This is not an absolute rule: the incident commander may prioritize shutdown, isolation, or process termination when continued operation threatens people, critical services, data, or the wider environment.

Adversaries increasingly rely on fileless malware, in-memory DLL injection (e.g., reflective DLL injection, process hollowing into explorer.exe or svchost.exe), and memory-only Cobalt Strike beacons. A hard power-off immediately flushes the CPU registers, cache, and system RAM, permanently destroying:

• In-memory decrypted malware payloads, shellcode, and C2 configuration blocks.
• Active network connection state tables (revealing external C2 sockets and internal lateral movement).
• Uncommitted process execution trees and command-line arguments.
• In-memory decrypted encryption keys (critical for potential ransomware recovery).
• Cached user credentials in lsass.exe memory.

Volatile Memory Acquisition Workflow

When live collection is safe, authorized, and proportionate, analysts can capture physical memory (RAM) with a validated tool and document the changes that acquisition introduces:

1. Tool Selection: WinPmem, DumpIt, FTK Imager Lite CLI, or LiME (Linux Memory Extractor). When using modern EDR platforms, analysts execute remote memory acquisition scripts via the live response console.
2. Execution Syntax (CLI): Running winpmem.exe -o C:\Temp\memdump.raw outputs a complete, uncompressed bit-stream image of physical RAM.
3. Volatile State Snapshot: Concurrently extract live volatile state into text logs:
   • Network sockets: netstat -ano > C:\Temp\sockets.txt
   • Running processes & handles: tasklist /v > C:\Temp\processes.txt and Sysinternals handle.exe > C:\Temp\handles.txt
   • Active user sessions: qwinsta > C:\Temp\sessions.txt
4. Cryptographic Verification: Compute the SHA-256 hash of the memory image immediately upon capture and record it in the chain of custody log.

---

Systematic Eradication Procedures and Persistence Neutralization

Eradication is the phase where all components of the intrusion are completely removed from the environment. Eradication cannot succeed if containment was incomplete; if an adversary retains a single persistent backdoor, the environment will be re-infected within hours of recovery.

Enterprise Eradication Checklist

• Active Process and Thread Termination: Terminate all malicious process trees using EDR console tools or administrative scripts: Stop-Process -Id <PID> -Force. For injected system processes, the entire host must be quarantined for subsequent clean rebuilding.
• Malware and Payload Deletion: Delete all staging folders, downloaded dropper executables, scripts, and temporary files identified across the file system (e.g., C:\ProgramData\*, C:\Windows\Temp\*, /tmp/*).
• Persistence Mechanism Sanitization:
  • Scheduled Tasks: Inspect and delete rogue tasks via schtasks.exe /delete /tn "<TaskName>" /f (correlated with Windows Security Event ID 4699).
  • Registry Run Keys: Clean malicious values from HKLM\Software\Microsoft\Windows\CurrentVersion\Run, RunOnce, Winlogon\Userinit, and Image File Execution Options (IFEO) debugger hijacks.
  • Windows Services: Stop and completely delete unauthorized services installed by the adversary via sc.exe delete <ServiceName> (correlated with System Event ID 7045).
  • WMI Event Subscriptions: Threat actors establish stealthy persistence using Windows Management Instrumentation. Analysts must query and purge rogue instances of __EventFilter, __EventConsumer, and __FilterToConsumerBinding using PowerShell: Get-CimInstance -Namespace root\subscription -ClassName __EventConsumer | Remove-CimInstance.
• Active Directory Identity Sanitization:
  • Delete all unauthorized user accounts created by the threat actor during the intrusion.
  • Remove compromised accounts from privileged security groups (Domain Admins, Enterprise Admins, Schema Admins, Account Operators).
  • The Double KRBTGT Password Reset: In Active Directory, the krbtgt service account encrypts all Kerberos Ticket Granting Tickets (TGTs). If an adversary extracted the krbtgt NTLM password hash, they can forge Kerberos Golden Tickets, granting persistent, undetectable administrative access even after all user passwords are changed. Active Directory stores two password hashes for krbtgt: the current password and the previous password (to prevent authentication failures during scheduled rotation). Therefore, to completely invalidate Golden Tickets, the krbtgt password must be reset TWICE, with sufficient delay between resets (typically 10 to 24 hours) to ensure domain-wide Active Directory replication has completed.
• Vulnerability Remediation: Patch the initial access vector (e.g., applying security updates to an unpatched VPN gateway, patching an Apache Struts vulnerability, or modifying vulnerable firewall rules).

---

Phased Recovery, Restoration, and Enhanced Monitoring

Recovery is the process of safely restoring affected systems, services, and data back to operational production status. Recovery must be executed in measured, validated phases to ensure that threat actors cannot exploit latent backdoors.

Golden Rule of System Restoration: Rebuild vs. Restore

• Rebuilding from Known-Good Images: The industry-standard best practice for compromised operating systems. Rather than attempting to manually scrub a heavily compromised server, the operating system is completely wiped and rebuilt from a validated, pre-hardened golden image or deployed automatically via Infrastructure-as-Code (Terraform, Ansible, SCCM).
• Restoring from Backup: If critical data must be restored from backups, the CSIRT must verify that the backup point-in-time precedes the adversary's Initial Access timestamp (Patient Zero compromise date). Restoring a backup taken after the attacker gained initial access simply re-introduces malware and backdoors into production.

The Phased Restoration Workflow

1. Restoration into Sandboxed Quarantine VLAN: Rebuilt systems are initially booted into an isolated network segment with restricted outbound and inbound access. System administrators verify application functionality, database integrity, and operating system patch levels.
2. Enterprise-Wide Credential Reset: Prior to reintroducing systems to the corporate network, execute a synchronized credential rotation across all affected domains: reset all user passwords, reset all service account credentials (migrating to Group Managed Service Accounts / gMSAs where possible), rotate all database connection strings, and regenerate enterprise SSH keys and API access tokens.
3. Gradual Production Re-Entry: Systems are reconnected to the production network in a phased rollout (e.g., restoring core infrastructure first, followed by internal application servers, and finally public-facing services).
4. Enhanced Post-Restoration Monitoring: For 30 to 90 days following recovery, the SOC applies heightened surveillance protocols. Detection engineers create dedicated high-sensitivity SIEM correlation rules specifically monitoring recovered assets, audit logs are retained at higher frequencies, and EDR agents are configured to maximum behavioral blocking sensitivity.

---

Post-Incident Review, Lessons Learned, and Root Cause Analysis (RCA)

The final phase of incident response—Post-Incident Activity / Lessons Learned—is the most critical for long-term organizational defense. Within one to two weeks following incident closure, the CSIRT convenes a formal Post-Mortem Meeting with cross-functional stakeholders (technical responders, IT management, legal counsel, and business owners). The meeting must maintain a blameless, objective culture focused on systemic failure analysis rather than individual scapegoating.

Root Cause Analysis (RCA) Methodologies

• The 5 Whys Methodology: An iterative interrogative technique used to explore the cause-and-effect relationships underlying a security incident. Responders repeatedly ask "Why?" (typically five times) to peel away superficial technical symptoms and expose the fundamental process, governance, or architectural breakdown.
• Ishikawa (Fishbone) Diagram: A structured visual diagram categorizing potential contributing causes of a breach across six core domains: People (training, staffing), Process (policies, patch management cycles), Technology (EDR coverage, legacy hardware), Environment (cloud configuration, physical security), Management (budget allocation, risk acceptance), and Measurement (monitoring metrics, logging gaps).

5 Whys Worked Case Study: Enterprise Ransomware Infiltration

Root Cause Identified: Failure of identity governance to mandate and enforce Multi-Factor Authentication (MFA) across 100% of external remote access endpoints, compounded by the absence of local administrator password randomization (LAPS).

Enterprise Incident Report Structure Template

Every major incident culminates in an executive-level Incident Report. The report serves as a historical record, regulatory artifact, and operational roadmap for capital expenditure:

================================================================================
                      ENTERPRISE INCIDENT REPORT TEMPLATE                      
================================================================================
1. EXECUTIVE SUMMARY
   - High-level non-technical summary of incident scope, duration, and business impact
   - Direct financial, operational, and regulatory consequences
   - Core root cause summary and key strategic recommendations

2. INCIDENT OVERVIEW & CHRONOLOGICAL TIMELINE
   - Incident discovery date, detection source, and confirmation timestamp (UTC)
   - Step-by-step chronological narrative from Initial Access to Final Recovery
   - Key milestone metrics: MTTD (Detection), MTTA (Acknowledge), MTTC (Containment)

3. TECHNICAL ANALYSIS & ATT&CK MAPPING
   - Detailed breakdown of attack vectors, malware families, and exploitation mechanics
   - MITRE ATT&CK Matrix mapping of all observed adversary TTPs
   - Defanged Indicators of Compromise (IoCs): SHA-256 hashes, C2 domains, IP addresses

4. ROOT CAUSE ANALYSIS (RCA)
   - Formal 5 Whys and Ishikawa diagram breakdown
   - Primary technical, architectural, and organizational governance failures

5. CONTAINMENT, ERADICATION & RECOVERY ACTIONS
   - Specific tactical interventions executed (EDR isolation, KRBTGT double reset)
   - Validation metrics confirming clean production recovery

6. CORRECTIVE ACTION PLAN (CAP)
   - Prioritized remediation table with assigned owners, resource costs, and deadlines:
     * Action Item 1: Enforce mandatory FIDO2 hardware MFA on all remote access (P1)
     * Action Item 2: Deploy Microsoft LAPS across 100% of domain endpoints (P1)
     * Action Item 3: Implement Kerberos credential guard and LSASS run-as-PPL (P2)
================================================================================