3.3 Automated Incident Remediation with EventBridge & Step Functions

Automated Incident Remediation with EventBridge & Step Functions

When an adversary breaches a cloud workload, automated script execution operates in seconds, while human security analysts require minutes or hours to detect, investigate, and manually contain the threat. This disparity in operational velocity is known as adversary dwell time. To eliminate dwell time and mitigate potential damage, cloud security architectures must incorporate event-driven automated remediation. By coupling real-time security telemetry with serverless orchestration engines, organizations can detect an intrusion, quarantine compromised resources, revoke leaked credentials, and capture forensic evidence in sub-minute timeframes.

---

Event-Driven Security Remediation Architecture

In AWS, automated security remediation follows an event-driven architecture centered on Amazon EventBridge, which acts as the universal serverless event bus:

+-------------------------------------------------------------+
| 1. Threat Detection Engine:                                 |
|    • Amazon GuardDuty (e.g., UnauthorizedAccess:EC2)        |
|    • AWS Security Hub (ASFF finding: Compliance FAILED)     |
+-------------------------------------------------------------+
                               |
                               v
+-------------------------------------------------------------+
| 2. Event Routing (Amazon EventBridge):                      |
|    Matches JSON Event Pattern (Severity >= 7.0)             |
+-------------------------------------------------------------+
                               |
                               v
+-------------------------------------------------------------+
| 3. Workflow Orchestration (AWS Step Functions):             |
|    Evaluates Resource Type -> Enforces Safety Guardrails    |
+-------------------------------------------------------------+
                               |
        +----------------------+----------------------+
        |                                             |
        v                                             v
+-------------------------------+   +---------------------------------+
| 4a. AWS SSM Automation:       |   | 4b. AWS Lambda:                 |
| • AWS-StopEC2Instance         |   | • Revoke Active STS Sessions    |
| • AWS-CreateSnapshot          |   | • Disable IAM Access Keys       |
| • S3 Block Public Access      |   | • Quarantine Network Interface  |
+-------------------------------+   +---------------------------------+

Ingesting Normalized Findings: The AWS Security Finding Format (ASFF)

Different security services historically produced divergent alert formats. AWS Security Hub resolves this by normalizing all findings—whether originating from native services like GuardDuty, Macie, and Inspector, or from third-party firewalls—into the standardized AWS Security Finding Format (ASFF). Key ASFF JSON fields evaluated by automated remediation rules include:

• Id: The unique identifier for the specific finding.
• ProductArn: Identifies the emitting tool (e.g., arn:aws:securityhub:us-east-1::product/aws/guardduty).
• AwsAccountId: The account where the incident occurred.
• Types: An array classifying the threat (e.g., ["TTPs/Initial Access/UnauthorizedAccess:EC2-TorIP"]).
• Severity.Label: The normalized severity string (INFORMATIONAL, LOW, MEDIUM, HIGH, CRITICAL).
• Severity.Normalized: A numerical score from 0 to 100, where scores 70.0 to 89.9 represent HIGH and 90.0 to 100.0 represent CRITICAL.
• Resources: Detailed metadata describing affected EC2 instances, S3 buckets, IAM roles, or Lambda functions.
• Compliance.Status: In compliance checks, indicates PASSED, WARNING, FAILED, or NOT_AVAILABLE.

Constructing EventBridge Rule Event Patterns

Amazon EventBridge evaluates incoming JSON events against declarative filter patterns. To trigger remediation only for high-priority threats and avoid flooding automation runners with informational noise, security teams construct specific event patterns:

Example: EventBridge Pattern Matching High-Severity GuardDuty Findings

{
  "source": ["aws.guardduty"],
  "detail-type": ["GuardDuty Finding"],
  "detail": {
    "severity": [
      { "numeric": [">=", 7.0] }
    ],
    "type": [
      { "prefix": "UnauthorizedAccess:EC2" },
      { "prefix": "Recon:IAMUser" },
      { "prefix": "CryptoCurrency:EC2" }
    ]
  }
}

Example: EventBridge Pattern Matching Failed Security Hub S3 Compliance Findings

{
  "source": ["aws.securityhub"],
  "detail-type": ["Security Hub Findings - Imported"],
  "detail": {
    "findings": {
      "Compliance": {
        "Status": ["FAILED"]
      },
      "GeneratorId": [
        { "prefix": "aws-foundational-security-best-practices/v/1.0.0/S3." }
      ],
      "Severity": {
        "Label": ["CRITICAL", "HIGH"]
      }
    }
  }
}

Cross-Account Event Aggregation

In multi-account environments, findings emitted in spoke workload accounts should not trigger uncoordinated local remediation. Instead, spoke accounts configure EventBridge rules that forward all security events to a centralized Security Event Bus hosted in the Security Tooling account. Remediation workflows execute centrally, maintaining unified audit logging and centralized control.

---

AWS Step Functions: Multi-Step Remediation Orchestration

While a single AWS Lambda function can execute a basic script, real-world incident containment requires multi-step, sequential, and parallel coordination across multiple systems. Relying on a single monolithic Lambda script introduces severe fragility: if one API call times out or fails, the entire script may crash mid-execution, leaving resources partially modified in an undefined state.

AWS Step Functions state machines provide the ideal enterprise remediation orchestrator:

• Visual Workflow Management: Every state transition, execution path, and error is recorded graphically in real time.
• Built-in Error Handling & Retries: Native Retry blocks support exponential backoff, jitter, and error-specific handling without writing custom code.
• Catch Blocks & Fallback States: If a primary remediation step fails, a Catch block routes execution to an alert state that pages the on-call engineer.
• Choice States: The state machine dynamically evaluates the finding metadata to branch between production and non-production accounts or between compute, storage, and identity remediation paths.

Standard vs. Express Workflows for Incident Response

Step Functions provides two distinct workflow types:

For security remediation involving human approval gates, EBS snapshot creation, and deep forensic capture, Standard Workflows are required.

---

AWS Systems Manager Automation Documents

AWS Systems Manager (SSM) Automation allows engineers to safely execute standardized operational and security runbooks at scale. Instead of reinventing custom remediation scripts, security teams leverage pre-built, AWS-managed SSM documents:

1. AWSConfigRemediation-ConfigureS3BucketPublicAccessBlock

When an S3 bucket is inadvertently made public, this runbook turns on the bucket-level S3 Block Public Access settings (BlockPublicAcls, IgnorePublicAcls, BlockPublicPolicy, RestrictPublicBuckets). The older AWS-DisableS3BucketPublicReadWrite runbook serves the same purpose for a single bucket.

2. AWS-StopEC2Instance

Gracefully stops a target EC2 instance. In an automated containment workflow, this action is typically sequenced after EBS snapshot creation and volatile memory capture to freeze persistent block storage.

3. AWS-CreateSnapshot

Creates a point-in-time snapshot of one EBS volume (you pass the volume ID). Loop over each attached volume, or call the EC2 CreateSnapshots API to capture all of an instance's volumes at one crash-consistent point. These snapshots are tagged with incident identifiers and can be automatically shared with the isolated Forensics account.

4. AWS-QuarantineEC2Instance and AWSSupport-ContainEC2Instance

AWS-QuarantineEC2Instance assigns a security group that allows no inbound or outbound traffic; AWSSupport-ContainEC2Instance is a broader containment runbook for an instance. Detaching the IAM instance profile (the EC2 DisassociateIamInstanceProfile API, typically called from Lambda or a custom Automation step) stops IMDS from serving new role credentials, but credentials already stolen stay valid until you revoke the role's active sessions.

Cross-Account SSM Automation Execution

To execute SSM runbooks across member accounts from the centralized Security Tooling account:

1. The centralized Step Functions state machine calls the SSM StartAutomationExecution API.
2. The call specifies the AutomationAssumeRole parameter pointing to a pre-provisioned role in the spoke account (e.g., arn:aws:iam::444455556666:role/SSMAutomationSpokeExecutionRole).
3. The spoke role executes the document locally, ensuring the security account does not require broad direct permissions inside the spoke account.

---

AWS Lambda for High-Speed Targeted Remediation

For microsecond-level execution speeds and custom containment logic, AWS Lambda complements SSM Automation. A prime example is instantly revoking compromised IAM user credentials.

The Critical Nuance of Active STS Session Invalidation

When an IAM user's long-term access keys are leaked, junior engineers often make the mistake of simply calling iam:DeleteAccessKey or changing the user's password. This does not terminate an active attack.

If the adversary used the compromised access key to generate temporary security credentials via AWS STS (such as calling sts:GetSessionToken or sts:AssumeRole), those temporary session tokens remain valid until their expiration time (which can be up to 12 or 36 hours). The attacker continues executing API calls unimpeded.

To immediately and completely invalidate all existing temporary STS sessions, the remediation Lambda function must attach an inline session revocation policy to the user or role. The policy uses an explicit Deny with an aws:TokenIssueTime condition:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ImmediatelyRevokeAllActiveSessions",
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "DateLessThan": {
          "aws:TokenIssueTime": "2026-09-29T12:00:00Z"
        }
      }
    }
  ]
}

How it works: aws:TokenIssueTime is the time the temporary credential was issued. Any request signed with a credential issued before 2026-09-29T12:00:00Z satisfies DateLessThan, so the explicit Deny blocks it, while credentials issued afterward work normally. Do not use aws:CurrentTime here: that key is the time of the request, so the deny would block everyone until the timestamp passes and then stop protecting you. Requests signed directly with long-term access keys carry no aws:TokenIssueTime, so deactivate or delete those keys as well. For roles, the IAM console's Revoke active sessions action attaches this exact pattern as the AWSRevokeOlderSessions inline policy.

---

Safety Guardrails, Approval Gates & Rollback Mechanisms

Uncontrolled automated remediation is dangerous. If an automated script encounters a false-positive alert and aggressively shuts down a core production database or revokes the credentials of an enterprise application role, the remediation script causes more damage than the threat it was designed to contain. Enterprise remediation workflows must incorporate strict safety guardrails:

+-------------------------------------------------------------+
| EventBridge Trigger: High-Severity Finding Received         |
+-------------------------------------------------------------+
                               |
                               v
+-------------------------------------------------------------+
| Step Functions Choice State: Is this a Production Workload? |
+-------------------------------------------------------------+
              |                                 |
     [YES]    |                                 |   [NO / Staging]
              v                                 v
+---------------------------+     +---------------------------+
| Human Approval Gate:      |     | Immediate Execution:      |
| • Pause via Task Token    |     | • Isolate Instance        |
| • Send Email / Slack Link |     | • Revoke Credentials      |
+---------------------------+     +---------------------------+
              |                                 |
     [Approved by SecOps]                       |
              v                                 |
+-----------------------------------------------+             |
| Execute Containment with Dry-Run Validation   | <-----------+
+-----------------------------------------------+
                               |
                               v
+-------------------------------------------------------------+
| Verify Finding Status -> Update Security Hub to RESOLVED    |
+-------------------------------------------------------------+

1. Human-in-the-Loop Approval Gates (.waitForTaskToken)

For production environments or destructive remediation actions (such as terminating compute or modifying public databases), Step Functions provides the Task Token Callback pattern:

1. The state machine reaches a Task state that publishes an approval request to an Amazon SNS topic or Slack channel using arn:aws:states:::sns:publish.waitForTaskToken.
2. Step Functions pauses execution and generates a unique, cryptographically signed TaskToken.
3. The notification contains two URLs (e.g., pointing to an Amazon API Gateway endpoint with query parameters): one for Approve and one for Reject, embedding the task token.
4. When a verified security engineer clicks "Approve", API Gateway triggers a Lambda function that calls sfn:SendTaskSuccess(taskToken). If clicked "Reject", it calls sfn:SendTaskFailure(taskToken).
5. Upon receiving SendTaskSuccess, Step Functions unpauses and proceeds with the automated remediation steps.

2. Idempotency and Dry-Run Verification

Automated remediation scripts must be idempotent: executing the script multiple times against the same resource must produce the exact same final state without generating errors or corrupting infrastructure. Furthermore, scripts interacting with EC2 should execute API calls with --dry-run to verify that IAM execution roles have the necessary permissions and that target resource IDs exist before executing state alterations.

3. Automated Rollback and Finding Closure

If a containment step fails midway through execution, Step Functions Catch blocks trigger rollback actions (such as restoring original security groups or alerting the primary on-call engineer through the paging tool). Upon successful containment, the workflow calls securityhub:BatchUpdateFindings to update the finding's Workflow.Status to RESOLVED, documenting the automated remediation audit trail in Security Hub.

---

Restoring Availability with Amazon Application Recovery Controller (ARC)

Skill 2.1.4 also names Amazon Application Recovery Controller (ARC), because recovery is part of remediation. ARC provides:

• Zonal shift: temporarily moves traffic for a supported resource (for example, an Application Load Balancer or Network Load Balancer with cross-zone settings) away from one Availability Zone. A shift is manual, temporary, and carries an expiration of up to three days that you can extend. Zonal autoshift lets AWS start the shift automatically when it detects an AZ impairment.
• Routing control and Region switch: multi-Region recovery controls that move traffic between Regional replicas of an application, backed by readiness checks that confirm the standby is actually able to take load.

In an incident, zonal shift is a fast containment-plus-recovery lever: if every compromised instance sits in one AZ, shifting traffic away keeps customers served while responders snapshot and rebuild that capacity. Automate it the same way as other remediations: an EventBridge rule or Step Functions state calls the ARC API, and CloudTrail records who started the shift.

Comparison of Incident Remediation Technologies

---

Exam Tips & Common Traps