6.2 Systems Manager Run Command, State Manager & Associations

Systems Manager Run Command: Fleet-Wide Execution at Scale

AWS Systems Manager Run Command enables administrators and automated CI/CD pipelines to execute commands, shell scripts, and configuration bundles across thousands of managed instances simultaneously. Because Run Command leverages the outbound SSM Agent communication channel, operators can administer fleets without opening inbound management ports, managing SSH bastion keys, or orchestrating complex VPN topologies.

Standard Command Documents

Run Command operations are defined by SSM Command Documents (JSON or YAML manifests) that specify the execution plugins, parameters, and operating system targets:

• AWS-RunShellScript: Executes Linux, macOS, or Unix shell scripts (/bin/sh or /bin/bash). Commands execute under the local root account (or ssm-user if configured).
• AWS-RunPowerShellScript: Executes Windows PowerShell scripts and cmdlets on Windows Server instances under the NT AUTHORITY\SYSTEM context.
• AWS-RunRemoteScript: Downloads a script from an Amazon S3 bucket or public/private GitHub/CodeCommit repository and executes it locally on the managed node.
• AWS-ConfigureAWSPackage: Installs, uninstalls, or updates AWS-curated or custom software packages (such as the unified Amazon CloudWatch Agent or AWS CLI) across targeted instances.

Fleet Targeting Strategies

Run Command allows flexible target specification to prevent operational drift and ensure precise command delivery:

1. Explicit Instance IDs: Targeting specific instances (e.g., i-0123456789abcdef0, mi-0123456789abcdef0). Suitable for emergency break-glass triage on individual hosts.
2. Tag-Based Targeting: Selecting nodes dynamically using resource tag keys and values (e.g., tag:Environment=Production AND tag:Role=WebServer). Newly launched Auto Scaling instances matching the tag criteria automatically become eligible for subsequent invocations.
3. Resource Groups: Targeting AWS Resource Groups that aggregate diverse resources across tags and CloudFormation stacks.

Concurrency and Blast Radius Management: maxConcurrency & maxErrors

When deploying updates or executing disruptive commands across large production fleets, cascading failures can cause catastrophic outages. Run Command provides built-in rate control via two critical parameters:

• maxConcurrency: The maximum number or percentage of managed instances that can execute the command simultaneously. For example, setting maxConcurrency: 10 limits parallel execution to 10 instances; setting maxConcurrency: 20% on a 50-instance fleet processes instances in batches of 10.
• maxErrors: The maximum number or percentage of instance execution failures permitted before AWS Systems Manager immediately halts the command execution across the rest of the fleet. For example, setting maxErrors: 0 ensures that if a single instance returns a non-zero exit code, the command is aborted immediately for all remaining un-executed targets.

[ Fleet of 100 Managed Instances ]
  │
  ├─ Batch 1: 20 Instances (maxConcurrency: 20%)
  │    ├── 19 Succeeded
  │    └── 1 Failed! ──> Evaluates against maxErrors (0% / 0 count)
  ▼
[ Circuit Breaker Tripped! ]
  │
  └─ Status transitions to FAILED
     Remaining 80 instances are CANCELLED without running the command

Command Output Capture & The 2,500 Character Truncation Limit

[!IMPORTANT] DOP-C02 Exam Trap: The Systems Manager API calls (GetCommandInvocation and ListCommandInvocations) and the AWS Management Console truncate stdout and stderr output to the first 2,500 characters! If a script outputs a stack trace, verbose installation log, or compilation result that exceeds 2,500 characters, the trailing output is permanently lost unless external logging is configured.

To preserve comprehensive execution transcripts:

• Configure OutputS3BucketName and OutputS3KeyPrefix: Systems Manager streams full stdout and stderr logs as discrete text files directly to the designated S3 bucket.
• Configure CloudWatchOutputConfig: Systems Manager streams command logs into an Amazon CloudWatch Logs log group, allowing real-time log analysis and subscription filtering.
• Use Amazon EventBridge: Systems Manager emits status events on command state changes (InProgress, Success, Failed, TimedOut, Cancelled), enabling automated event-driven alerting or automated rollback pipelines.

Systems Manager State Manager: Continuous Desired State Configuration

While Run Command is designed for imperative, ad-hoc execution, Systems Manager State Manager provides declarative, continuous configuration management. State Manager ensures that your target fleet conforms to a defined "desired state" and continuously monitors and remediates configuration drift.

The State Manager Association

A State Manager Association is the foundational binding between a configuration document, a set of target instances, parameters, and an execution schedule:

1. Association Document: The SSM Document (command or automation) specifying the configuration steps. Examples include updating the SSM Agent (AWS-UpdateSSMAgent), executing an Ansible Playbook (AWS-ApplyAnsiblePlaybooks), or enforcing security benchmark settings.
2. Targeting: Defines which instances the association applies to. Targets can be dynamic (using tag queries such as tag:OS=Linux), explicit instance IDs, or account-wide (*).
3. Schedule: Associations run on a recurring schedule defined by a cron expression (e.g., cron(0 0 ? * SUN *) for every Sunday at midnight) or a rate expression (e.g., rate(30 minutes)).
4. Compliance Status: Every time an association executes, State Manager evaluates the execution outcome and assigns a compliance status: Compliant or Non-Compliant. This compliance state is published to the Systems Manager Compliance dashboard and emitted to AWS Config.

Third-Party Framework Integration: Ansible, Chef, and PowerShell DSC

State Manager eliminates the traditional operational burden of maintaining separate, long-running configuration management master servers (such as Ansible Tower/AWX, Chef Infra Server, or Puppet Master):

• AWS-ApplyAnsiblePlaybooks: An AWS-managed SSM document that downloads an Ansible playbook directly from Amazon S3, GitHub, or AWS CodeCommit, installs the Ansible engine locally if not present, and executes ansible-playbook against localhost.
• AWS-ApplyChefRecipes: Downloads Chef cookbooks and executes Chef Solo locally.
• AWS-InstallPowerShellModule: Installs PowerShell Desired State Configuration (DSC) resources and compiles configurations locally.

This "masterless" architecture allows infrastructure teams to store declarative playbooks in Git, update the S3 source bucket via CI/CD pipelines, and let State Manager automatically enforce the playbook across thousands of instances without SSH access.

---

SSM Automation Documents: Multi-Step Operational Workflows

SSM Automation Documents (defined under schemaVersion: '0.3') orchestrate complex, multi-step cloud workflows that span multiple AWS services, managed nodes, and third-party systems. Automation documents are frequently used for automated golden AMI baking, disaster recovery failover, forensic isolation of compromised EC2 instances, and automated patch remediation.

Core Automation Action Plugins

Error Handling, Rollback Steps, and Branching

Enterprise automation requires deterministic failure recovery. In an SSM Automation document, each step can explicitly define failure behavior:

• onFailure: Defines what occurs when the step fails. Options include:
  • Abort (default): Immediately stops workflow execution and marks the execution as Failed.
  • Continue: Ignores the error and proceeds to the next sequential step.
  • step:<step_name>: Immediately diverts execution to a dedicated recovery, cleanup, or rollback step.
• isEnd: true: Marks the step as the terminal step of the workflow upon successful completion.
• maxAttempts: Configures automatic retries for transient API rate-limiting or network issues.

Production Automation Document Example

The following document illustrates dynamic branching, inline Python scripting, step retries, and automated rollback upon validation failure:

description: "Automated EC2 Maintenance and Health Verification with Rollback"
schemaVersion: '0.3'
assumeRole: "arn:aws:iam::123456789012:role/SSMAutomationExecutionRole"
parameters:
  InstanceId:
    type: String
    description: "Target EC2 Instance ID"
mainSteps:
  - name: InspectInstanceHealth
    action: aws:executeScript
    inputs:
      Runtime: python3.10
      Handler: check_health
      InputPayload:
        instance_id: "{{ InstanceId }}"
      Script: |
        import boto3
        def check_health(events, context):
            ec2 = boto3.client('ec2')
            resp = ec2.describe_instance_status(InstanceIds=[events['instance_id']])
            status = resp['InstanceStatuses'][0]['SystemStatus']['Status'] if resp['InstanceStatuses'] else 'impaired'
            return {'status': status}
    outputs:
      - Name: SystemStatus
        Selector: $.Payload.status
        Type: String

  - name: EvaluateBranch
    action: aws:branch
    inputs:
      Choices:
        - NextStep: ExecutePackageUpgrade
          Variable: "{{ InspectInstanceHealth.SystemStatus }}"
          StringEquals: ok
      Default: TerminateWithRollback

  - name: ExecutePackageUpgrade
    action: aws:runCommand
    onFailure: step:RollbackPackageUpgrade
    inputs:
      DocumentName: AWS-RunShellScript
      InstanceIds:
        - "{{ InstanceId }}"
      Parameters:
        commands:
          - "yum update -y security"
          - "systemctl restart my-critical-service"

  - name: VerifyServiceHealth
    action: aws:runCommand
    onFailure: step:RollbackPackageUpgrade
    inputs:
      DocumentName: AWS-RunShellScript
      InstanceIds:
        - "{{ InstanceId }}"
      Parameters:
        commands:
          - "curl -f http://localhost:8080/health || exit 1"
    isEnd: true

  - name: RollbackPackageUpgrade
    action: aws:runCommand
    inputs:
      DocumentName: AWS-RunShellScript
      InstanceIds:
        - "{{ InstanceId }}"
      Parameters:
        commands:
          - "echo 'Rolling back package upgrades...'"
          - "yum history undo last -y"
          - "systemctl restart my-critical-service"
    isEnd: true

  - name: TerminateWithRollback
    action: aws:invokeLambdaFunction
    inputs:
      FunctionName: "AlertOperationsCenter"
      InputPayload:
        message: "Instance {{ InstanceId }} failed initial health inspection. Aborting."
    isEnd: true

---

Comparison: Run Command vs. State Manager vs. Automation