2.4 Automated Remediation with Systems Manager & Lambda

2.4 Automated Remediation with Systems Manager & Lambda

Automated remediation transforms static monitoring into self-healing infrastructure. When operational incidents, resource failures, or compliance violations occur, relying on human engineers to diagnose and execute corrective actions introduces latency, operational overhead, and human error. AWS provides two primary compute engines for automated remediation: AWS Systems Manager (SSM) Automation and AWS Lambda. For the CloudOps Associate exam, engineers must know how to choose between these engines, design multi-step runbooks, implement approval gates, integrate with AWS Config rules, and execute production incident response workflows.

Remediation Workflow Architecture: Choosing SSM Automation vs. Lambda

An automated remediation loop follows three distinct operational phases:

1. Detection: A CloudWatch Alarm trips into the ALARM state, an EventBridge rule detects a state-change or API call, or an AWS Config rule flags a resource as non-compliant.
2. Orchestration & Execution: The detection mechanism triggers either an SSM Automation runbook or an AWS Lambda function.
3. Audit & Verification: Every remediation action must be logged in AWS CloudTrail, Systems Manager execution history, and CloudWatch Logs to ensure traceability and verify successful resolution.

Choosing the appropriate execution engine is a common exam theme:

As a general CloudOps rule of thumb: prefer AWS-managed SSM Automation runbooks first. Only introduce AWS Lambda when custom parsing, external third-party API integration, or sub-second latency is strictly necessary.

SSM Automation Runbooks: Structure & Execution

An SSM Automation runbook is a declarative document (defined in YAML or JSON) that specifies the tasks executed by Systems Manager. AWS provides dozens of predefined, battle-tested runbooks, including:

• AWS-RestartEC2Instance: Safely reboots an EC2 instance.
• AWS-StopEC2Instance: Stops an instance to halt runaway costs or isolate workloads.
• AWS-CreateSnapshot: Generates point-in-time EBS volume snapshots prior to maintenance.
• AWS-PublishSNSNotification: Broadcasts incident updates to operational channels.
• AWS-AttachEBSVolume: Automatically attaches replacement storage volumes.

Document Structure & The Automation Assume Role

An Automation document consists of several core sections:

• schemaVersion: Must be specified as 0.3 to support modern branching, looping, and multi-step workflows.
• assumeRole: The ARN of the IAM role that Systems Manager assumes to perform actions (AutomationAssumeRole). This role must have a trust policy allowing ssm.amazonaws.com and identity policies granting the specific AWS API actions executed in the runbook.
• parameters: Defines inputs passed at runtime (e.g., InstanceId, SnapshotId), along with types, descriptions, and defaults.
• mainSteps: An ordered sequence of steps using SSM action plugins.
• outputs: Values exported from steps for subsequent runbooks or parent workflows.

A production-grade custom runbook leverages action plugins such as aws:executeAwsApi (making direct AWS API calls), aws:waitForAwsResourceProperty (polling until resources enter a desired state), and conditional branching (onFailure, nextStep):

description: "Diagnose and reboot unresponsive EC2 instance"
schemaVersion: "0.3"
assumeRole: "arn:aws:iam::123456789012:role/SSMAutomationRole"
parameters:
  InstanceId:
    type: "String"
    description: "Target EC2 Instance to remediate"
mainSteps:
  - name: "CreateSafetySnapshot"
    action: "aws:executeAwsApi"
    inputs:
      Service: "ec2"
      Api: "CreateSnapshot"
      VolumeId: "{{GetVolume.VolumeId}}"
      Description: "Pre-remediation snapshot"
    nextStep: "RestartInstance"
    onFailure: "Abort"
  - name: "RestartInstance"
    action: "aws:changeInstanceState"
    inputs:
      InstanceIds:
        - "{{InstanceId}}"
      CheckStateOnly: false
      DesiredState: "reboot"
    nextStep: "WaitForHealthy"
  - name: "WaitForHealthy"
    action: "aws:waitForAwsResourceProperty"
    timeoutSeconds: 300
    inputs:
      Service: "ec2"
      Api: "DescribeInstanceStatus"
      InstanceIds:
        - "{{InstanceId}}"
      PropertySelector: "$.InstanceStatuses[0].InstanceStatus.Status"
      DesiredValues:
        - "ok"
    isEnd: true

Manual Approval Steps (aws:approve)

In mission-critical or high-risk environments, full automation may be restricted by compliance policies. SSM Automation supports hybrid workflows using the aws:approve action plugin. When reached, execution pauses, dispatches an approval notification via Amazon SNS to designated Approvers (IAM users or roles), and waits for a human operator to issue an Approve or Reject API call before proceeding to potentially destructive steps (such as instance termination or database failover).

Integrating Remediation with AWS Config Rules

AWS Config continuously evaluates AWS resource configurations against desired baselines. When a managed or custom Config rule detects non-compliance, it can initiate automatic remediation via SSM Automation.

To configure automatic remediation:

1. Select the AWS Config rule (e.g., s3-bucket-server-side-encryption-enabled or restricted-ssh).
2. Attach an SSM Automation remediation action (e.g., AWS-EnableS3BucketEncryption).
3. Configure the Resource ID parameter to map the non-compliant resource ID emitted by Config directly to the runbook's target parameter (e.g., BucketName).
4. Specify the AutomationAssumeRole ARN with permissions to modify the target resource.
5. Set the execution mode to Automatic and define retry attempts (e.g., retry 5 times every 60 seconds).

As soon as a developer creates an unencrypted S3 bucket, AWS Config detects non-compliance within seconds and triggers the runbook, applying default SSE-S3 or KMS encryption without human intervention.

Operational Scenarios: Self-Healing & Incident Response

1. Application Daemon Recovery via SSM Run Command

When a CloudWatch metric alarm detects that a critical daemon (e.g., httpd or nginx) has stopped, an EventBridge rule invokes an SSM Automation runbook. The runbook uses the aws:runCommand plugin to execute systemctl restart nginx via the SSM Agent on the instance, avoiding SSH key management and private network access restrictions.

2. Compromised Instance Quarantine & Digital Forensics

When AWS GuardDuty detects that an EC2 instance is communicating with a known command-and-control server:

1. EventBridge intercepts the GuardDuty finding and invokes an incident response SSM Automation runbook.
2. The runbook immediately detaches the instance from its Auto Scaling group (or puts it in standby) to prevent the ASG from terminating it.
3. The runbook replaces the instance's security groups with an Isolation Security Group that denies all inbound and outbound traffic, severing network access.
4. The runbook takes point-in-time EBS volume snapshots of all attached disks for forensic analysis.
5. It triggers volatile memory capture via the SSM Agent before stopping the instance.
6. All actions are logged immutably in CloudTrail for legal and compliance auditability.