3.3 In-Place, Rolling & Immutable Deployments on EC2

EC2 Deployment Paradigms

When deploying application code to Amazon EC2 fleets, DevOps engineers must select an appropriate deployment pattern based on availability requirements, rollback speed, cost tolerances, and sensitivity to configuration drift.

1. In-Place Deployment (Mutable)

In an In-Place Deployment, the deployment agent stops the application on existing running instances, installs the new revision bundle, updates configuration files, and restarts the service.

• Advantages: Fastest deployment cycle; zero additional infrastructure cost; preserves instance metadata, Elastic IP attachments, and local caches.
• Disadvantages: Fleet capacity is reduced while instances undergo updates (unless overprovisioned); instances can suffer from configuration drift over time; rollbacks require redeploying the previous revision bundle, which takes just as long as the original deployment.

2. Rolling Deployment

A Rolling Deployment updates an Auto Scaling group or fleet in discrete, sequential batches (e.g., 20% or 1 instance at a time). As each batch successfully completes health checks, the orchestrator moves to the next batch.

• Advantages: Avoids complete service downtime; limits blast radius of defects to the active batch size.
• Disadvantages: Deployment duration scales linearly with fleet size; fleet capacity temporarily drops during the update unless an additional batch is launched ahead of time; transient mixed-version states exist where both version A and version B handle live user traffic simultaneously.

3. Immutable Deployment

An Immutable Deployment never modifies running instances. Instead, an entirely new set of EC2 instances is launched (either via an updated AMI or clean user-data bootstrapping). The new instances are placed behind the load balancer, validated with health checks, and old instances are terminated once the new fleet is healthy.

• Advantages: Absolute immunity against configuration drift; pristine runtime state; near-instant rollback (simply terminate the newly launched instances while leaving the old fleet untouched).
• Disadvantages: Requires temporary duplicate infrastructure capacity (doubling compute costs during rollout); longer launch times due to instance provisioning.

---

AWS CodeDeploy Architecture & Agent Operations

AWS CodeDeploy automates application deployments across Amazon EC2 instances, on-premises servers, Amazon ECS, and AWS Lambda.

The CodeDeploy Agent

The CodeDeploy Agent is an open-source software package (built in Ruby) that runs on each target EC2 instance:

• Network Communication: The agent uses an outbound-only polling model over HTTPS (port 443) to communicate with the CodeDeploy control plane. It does not require any inbound ports to be open in security groups or network ACLs.
• IAM Instance Profile: The EC2 instance profile must grant permissions to interact with CodeDeploy and retrieve deployment artifacts from Amazon S3 (s3:GetObject) or GitHub.
• Auto Scaling Group Integration: When integrated with an Auto Scaling group, CodeDeploy automatically suspends specific ASG scaling processes (ReplaceUnhealthy, AZRebalance, AlarmNotification, ScheduledActions) during deployment execution to prevent race conditions.

---

The appspec.yml Anatomy & Lifecycle Event Hooks

The appspec.yml file is placed in the root directory of the application revision bundle. For EC2/On-Premises deployments, it defines files to copy, permissions to set, and custom scripts to execute during specific Lifecycle Event Hooks.

Exact Lifecycle Hook Execution Sequence (In-Place)

During an in-place deployment to an EC2 instance, CodeDeploy executes hooks in an exact, deterministic order:

+-------------------------------------------------------------+
|        CodeDeploy EC2 In-Place Lifecycle Hook Sequence      |
|                                                             |
|   1. ApplicationStop     <-- Gracefully stop existing app   |
|   2. DownloadBundle      <-- Reserved (Agent downloads zip) |
|   3. BeforeInstall       <-- Pre-install tasks, decrypting  |
|   4. Install             <-- Reserved (Agent extracts files)|
|   5. AfterInstall        <-- Post-install config & symlinks |
|   6. ApplicationStart    <-- Start daemon / application     |
|   7. ValidateService     <-- Health checks & endpoint probe |
+-------------------------------------------------------------+

1. ApplicationStop: Executes scripts to gracefully drain connections and stop the running application version. On an instance's very first deployment, this hook is skipped.
2. DownloadBundle (Agent Reserved): The CodeDeploy agent downloads the revision bundle from the S3 bucket to a temporary directory on the host. Custom scripts cannot be mapped here.
3. BeforeInstall: Custom pre-installation scripts execute (e.g., pre-allocating directories, backing up databases, decrypting secrets from AWS Secrets Manager).
4. Install (Agent Reserved): The agent copies files from the extracted bundle to the target file system locations specified in the files section.
5. AfterInstall: Custom scripts execute to configure file permissions, establish symlinks, compile assets, or install dependencies.
6. ApplicationStart: Starts the application services or background daemons.
7. ValidateService: The most critical verification hook. Executes scripts (such as curl health probes against http://localhost:8080/health) to confirm the service is healthy and ready to accept traffic. If this script exits with a non-zero exit code or times out, the deployment fails.

Production appspec.yml Snippet

version: 0.0
os: linux
files:
  - source: /build/target/app.jar
    destination: /opt/production/app/
  - source: /config/application.yml
    destination: /etc/app/
hooks:
  ApplicationStop:
    - location: scripts/stop_service.sh
      timeout: 120
      runas: root
  BeforeInstall:
    - location: scripts/backup_and_clean.sh
      timeout: 60
      runas: root
  AfterInstall:
    - location: scripts/configure_environment.sh
      timeout: 180
      runas: appuser
  ApplicationStart:
    - location: scripts/start_service.sh
      timeout: 60
      runas: appuser
  ValidateService:
    - location: scripts/validate_health.sh
      timeout: 90
      runas: appuser

---

Deployment Configurations & Minimum Healthy Hosts

CodeDeploy uses Deployment Configurations to govern the speed of rollout and enforce fleet health guardrails.

Predefined Deployment Configurations

• CodeDeployDefault.AllAtOnce: Deploys to all instances simultaneously. Fastest rollout; maximum capacity degradation during deployment.
• CodeDeployDefault.HalfAtATime: Deploys to a maximum of 50% of instances at a time. Retains at least half capacity.
• CodeDeployDefault.OneAtATime: Deploys to one instance at a time. Lowest risk; slowest execution.

Custom Deployment Configurations: Minimum Healthy Hosts

You can create custom configurations using minimumHealthyHosts defined as either FLEET_PERCENT or HOST_COUNT:

• FLEET_PERCENT: Specifies that at least X percent of instances in the deployment group must remain in a healthy, running state throughout the deployment. For example, in a 10-instance fleet with minimumHealthyHosts: 80%, CodeDeploy deploys to at most 2 instances (20%) concurrently.
• HOST_COUNT: Specifies the exact absolute number of healthy instances required. In a 5-instance fleet with minimumHealthyHosts: 4, CodeDeploy deploys to exactly 1 instance at a time.

---

AWS Elastic Beanstalk Deployment Policies

AWS Elastic Beanstalk abstracts EC2 deployment orchestration through five distinct deployment policies:

+---------------------------------------------------------------------------------------------------------+
|                                Elastic Beanstalk Deployment Policies                                    |
|                                                                                                         |
|  Policy                 | Downtime | Capacity Impact | Cost Overhead | Rollback Speed | Drift Protection|
|  -----------------------+----------+-----------------+---------------+----------------+-----------------|
|  All at Once            | Yes      | 100% loss       | None (0%)     | Slow           | Poor            |
|  Rolling                | No       | Reduced by batch| None (0%)     | Slow           | Poor            |
|  Rolling + Addl Batch   | No       | None (100% kept)| 1 Batch temp  | Slow           | Moderate        |
|  Immutable              | No       | None (100% kept)| 100% temp     | Instant        | Absolute        |
|  Traffic Splitting      | No       | None (100% kept)| 100% temp     | Instant        | Absolute        |
+---------------------------------------------------------------------------------------------------------+

1. All at once: Deploys the new version to all instances simultaneously. Causes complete environment downtime. Suitable only for non-critical development environments.
2. Rolling: Deploys in batches. While a batch is updating, total fleet capacity drops by the batch size. Zero additional infrastructure cost, but capacity is compromised during deployment.
3. Rolling with additional batch: Launches an extra batch of new EC2 instances first to maintain 100% operational capacity throughout the deployment. Once the extra batch is healthy, it updates existing instances in batches, terminating the extra instances at the end.
4. Immutable: Creates an entirely new, temporary Auto Scaling group within the same environment. Launches a single instance to verify health; once validated, it launches the remaining capacity in the temporary ASG, shifts all traffic, merges instances into the main ASG, and terminates old instances.
5. Traffic splitting: Launches an immutable temporary ASG and splits a configurable percentage of live production traffic (e.g., 10% for 15 minutes) via Application Load Balancer routing rules. If CloudWatch health alarms breach, traffic immediately reverts to the old fleet.

---

Automated Rollbacks & Hook Failure Handling

CodeDeploy incorporates automated rollback safety triggers configured in the Deployment Group settings:

• Rollback on Deployment Failure: Automatically triggers a rollback if any instance fails a lifecycle hook (such as ValidateService) or exceeds timeout limits.
• Rollback on CloudWatch Alarms: Monitors CloudWatch alarms (e.g., ALB HTTP 5xx error rate or high latency). If an alarm transitions to ALARM state while a deployment is in progress, CodeDeploy halts progression and immediately redeploys the last known good revision.
• The ApplicationStop Caveat: If an application is corrupted or misconfigured, the ApplicationStop script on the instance may fail during a new deployment. By default, a failed hook halts deployment. However, you can pass the --ignore-application-stop-failures flag via the CLI or enable it in deployment settings to bypass ApplicationStop failures and proceed with deployment.