7.2 Managing Stacks, Drift Detection & Change Sets

7.2 Managing Stacks, Drift Detection & Change Sets

CloudOps Blueprint Focus: Managing production infrastructure requires rigorous change governance and drift remediation. For the AWS Certified CloudOps Engineer – Associate (SOA-C03) exam, you must understand the operational hazards of direct stack updates, execute Change Sets to preview resource replacements, detect and resolve infrastructure drift caused by out-of-band changes, enforce Stack Termination Protection, and author JSON Stack Policies to lock down stateful resources.

Stack Lifecycle Operations: Direct Updates vs. Change Sets

CloudFormation manages collections of related resources through the stack lifecycle. Throughout this lifecycle, stacks transition through standardized status states (CREATE_IN_PROGRESS, CREATE_COMPLETE, UPDATE_IN_PROGRESS, UPDATE_COMPLETE, ROLLBACK_IN_PROGRESS, ROLLBACK_COMPLETE, DELETE_IN_PROGRESS, DELETE_COMPLETE).

When updating an existing stack with a modified template or parameter set, CloudOps engineers can choose between two operational approaches:

Direct Stack Updates

In a direct update (aws cloudformation update-stack), CloudFormation compares the submitted template directly against the currently deployed template and immediately begins executing modifications. While fast, direct updates carry severe operational risks in production environments:

• Unforeseen resource replacements can terminate active database instances or network interfaces.
• Property changes may cause service restarts or unexpected outages without warning.
• Operators cannot review the precise execution sequence before the update begins.

Change Sets Deployment Pattern

To eliminate operational risk, enterprise governance mandates the use of Change Sets (aws cloudformation create-change-set). Change Sets decouple change evaluation from execution:

1. CloudFormation calculates the delta between the active stack state and the proposed template.
2. CloudFormation generates a detailed, queryable preview report without modifying live resources.
3. The CloudOps engineer inspects the planned changes, assesses downtime impact, and either executes (execute-change-set) or discards (delete-change-set) the Change Set.

---

Change Sets Workflow & Replacement Impact Analysis

When evaluating a Change Set via the AWS Management Console or describe-change-set CLI command, operators must analyze four key dimensions for every affected resource:

The Operational Hazards of Replacement: True

When Replacement is marked as True, CloudFormation cannot update the resource in place because an immutable property was modified (such as changing an EC2 instance's AMI, modifying an RDS DB Subnet Group, or renaming a DynamoDB partition key). CloudFormation executes a replacement workflow:

1. CloudFormation provisions a brand-new resource alongside the existing resource.
2. CloudFormation updates dependent resources to reference the new physical ID.
3. CloudFormation terminates and deletes the original resource.

[!WARNING] Replacement: True results in a new physical ID, new private/public IP addresses, and new DNS hostnames. For compute instances, all ephemeral storage is lost. For unversioned storage or databases lacking retained snapshots, replacement can trigger catastrophic data loss and extended application downtime.

When Replacement is marked as Conditional, replacement depends on property values evaluated at execution time (for example, whether an RDS engine version update allows in-place upgrade or requires a complete instance replacement).

---

CloudFormation Drift Detection Architecture & Operational Realities

In modern cloud operations, team members occasionally bypass IaC pipelines and execute out-of-band modifications directly via the AWS Management Console, AWS CLI, or direct SDK calls (e.g., manually opening a security group port during an incident or resizing an EBS volume).

CloudFormation Drift Detection identifies these discrepancies by comparing the live runtime configuration of supported AWS resources against the expected state recorded in the CloudFormation template.

Drift Detection Statuses

Drift detection reports status at both stack and resource levels:

• Stack Drift Status: IN_SYNC (all resources match), DRIFTED (one or more resources have modified or deleted properties), or NOT_CHECKED.
• Resource Drift Status: IN_SYNC, MODIFIED (specific properties differ from template), or DELETED (the resource was terminated or deleted out-of-band).

Critical Operational Rule: CloudFormation Does Not Auto-Reconcile Drift

CloudFormation Drift Detection is purely diagnostic. CloudFormation cannot automatically reverse out-of-band modifications or roll back live resources to match the template without intervention. There is no automated "synchronize" or "revert drift" button.

To resolve drift, CloudOps engineers must choose one of two remediation paths:

1. Revert the Live Resource: Manually reconfigure the live AWS resource (via AWS Console or CLI) to match the template's specified properties, then re-run drift detection to confirm the status is IN_SYNC.
2. Update the Template: Modify the CloudFormation template to reflect the drifted out-of-band changes, then execute a standard stack update (via Change Set) so the template and live resource align perfectly.

---

Safeguarding Infrastructure: Termination Protection & Stack Policies

Production operations require multiple defensive layers to prevent accidental or malicious infrastructure destruction:

Stack Termination Protection

Stack Termination Protection is a boolean attribute configured on a stack (aws cloudformation update-termination-protection --enable-termination-protection). When enabled, CloudFormation rejects all stack deletion calls made via the AWS Management Console, CLI, or API. An administrator must explicitly disable termination protection before the stack can be deleted.

[!IMPORTANT] Termination protection only prevents stack deletion. It does not prevent stack updates, nor does it prevent resource replacement or resource deletion occurring as part of a stack update.

Stack Policies: Enforcing Granular Update Governance

A Stack Policy is a JSON policy document attached directly to a CloudFormation stack that governs which update actions can be executed on specific resources during stack updates.

By default, all update actions are permitted on all resources in a stack. When a Stack Policy is applied, CloudFormation shifts to an explicit default-deny security model: all update actions are denied unless explicitly allowed.

To protect mission-critical stateful infrastructure, engineers attach a stack policy that allows general updates while explicitly denying replacement or deletion on critical databases:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "Update:*",
      "Principal": "*",
      "Resource": "*"
    },
    {
      "Effect": "Deny",
      "Action": [
        "Update:Replace",
        "Update:Delete"
      ],
      "Principal": "*",
      "Resource": "LogicalResourceId/ProductionDatabase"
    }
  ]
}

If an update would cause ProductionDatabase to be replaced or deleted, CloudFormation halts the update immediately and rolls back. When an intentional database replacement is required during planned maintenance, an authorized operator can temporarily bypass the restriction by passing an override policy during the update call (--stack-policy-during-update-body).