3.2 Automated Image Baking with EC2 Image Builder

Automated Image Baking Architecture

In high-velocity enterprise environments, deploying virtual machine workloads requires a balance between boot-time configuration (bootstrapping via user-data) and pre-baked configuration (Golden AMIs). While bootstrapping introduces runtime latency and vulnerability to external repository failures, traditional manual AMI creation produces configuration drift and undocumented dependencies.

AWS EC2 Image Builder provides a fully managed, serverless orchestration pipeline to automate the creation, management, testing, and distribution of secure, compliant Amazon Machine Images (AMIs) and Open Container Initiative (OCI) container images.

The Four Core Constructs of EC2 Image Builder

+-----------------------------------------------------------------------------+
|                           EC2 Image Builder Pipeline                        |
|                                                                             |
|  +---------------------+         +---------------------------------------+  |
|  |    Image Recipe     |         |      Infrastructure Configuration     |  |
|  | - Base AMI          |         | - IAM: EC2InstanceProfileForImage...  |  |
|  | - Build Components  |         | - Subnet, Security Groups, VPC        |  |
|  | - Test Components   |         | - Instance Type (e.g., m5.large)      |  |
|  | - Storage (EBS)     |         | - Terminate Instance on Failure       |  |
|  +----------+----------+         +-------------------+-------------------+  |
|             |                                        |                      |
|             +-------------------+--------------------+                      |
|                                 |                                           |
|                                 v                                           |
|               +-----------------------------------+                         |
|               |   Image Build & Validation Flow   |                         |
|               |   1. Launch temporary EC2 instance|                         |
|               |   2. Execute Build components     |                         |
|               |   3. Create snapshot & AMI        |                         |
|               |   4. Launch Test instance & test  |                         |
|               |   5. Terminate build instances    |                         |
|               +-----------------+-----------------+                         |
|                                 |                                           |
|                                 v                                           |
|                  +------------------------------+                           |
|                  |   Distribution Settings      |                           |
|                  | - Multi-Region AMI Copy      |                           |
|                  | - Cross-Account Permissions  |                           |
|                  | - Destination KMS Re-encrypt |                           |
|                  | - SSM Parameter / Launch Tmpl|                           |
|                  +------------------------------+                           |
+-----------------------------------------------------------------------------+

1. Image Recipe: The blueprint defining the source parent image (e.g., Amazon Linux 2023, Red Hat Enterprise Linux, Ubuntu, or Windows Server), additional EBS storage volume configurations, working directories, and the ordered sequence of Build Components and Test Components.
2. Infrastructure Configuration: Defines the ephemeral compute environment launched to execute the recipe. Specifies the build instance type (e.g., t3.medium, m5.large), the VPC subnet, security groups, key pair (optional), SNS alert topics, S3 logging bucket, and the critical IAM Instance Profile (EC2InstanceProfileForImageBuilder).
3. Distribution Configuration: Governs the output distribution rules after testing passes. Configures target AWS regions, cross-account sharing (via AWS Account IDs or AWS Organization / Organizational Unit ARNs), AMI launch permissions, target KMS CMK encryption keys per region, and automatic updates to Amazon EC2 Launch Templates or AWS Systems Manager Parameter Store parameters.
4. Image Pipeline: The automation engine that binds the Recipe, Infrastructure Configuration, and Distribution Configuration together. Governs trigger conditions, such as cron schedules, rate expressions, manual execution, or Amazon EventBridge events triggered when a new upstream parent AMI is released.

---

Component Document Authoring (YAML Specification)

Components are declarative documents written in YAML format using the Image Builder Component Document Schema Version 1.0. Image Builder connects to the ephemeral build and test instances through the AWS Systems Manager Agent, installs the AWS Task Orchestrator and Executor (AWSTOE) application on each instance (or inside the running container for container recipes), and AWSTOE executes the component phases and steps sequentially.

Lifecycle Phases of a Component

• build: Instructions executed to install software, apply kernel configurations, configure logging daemons, and set security baselines.
• validate: Commands executed immediately after the build phase to confirm that services are running, binaries exist, and files are configured properly before creating the intermediate AMI snapshot.
• test: Commands executed on a fresh instance launched from the newly baked AMI to verify bootability, application functionality, and end-to-end operational health.

Component Document Syntax Example

The following custom component document installs the CloudWatch agent, configures corporate hardening, and validates configuration syntax:

name: EnterpriseHardeningAndCloudWatch
description: Installs Unified CloudWatch Agent and configures CIS baseline
schemaVersion: 1.0

phases:
  - name: build
    steps:
      - name: InstallCloudWatchAgent
        action: ExecuteBash
        inputs:
          commands:
            - echo "Downloading Amazon CloudWatch Agent..."
            - wget -q https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
            - rpm -U ./amazon-cloudwatch-agent.rpm
            - rm -f ./amazon-cloudwatch-agent.rpm
      - name: ConfigureSecurityKernelParameters
        action: ExecuteBash
        inputs:
          commands:
            - sysctl -w net.ipv4.ip_forward=0
            - sysctl -w net.ipv4.conf.all.accept_redirects=0
            - echo "net.ipv4.ip_forward = 0" >> /etc/sysctl.d/99-cis.conf
            - echo "net.ipv4.conf.all.accept_redirects = 0" >> /etc/sysctl.d/99-cis.conf

  - name: validate
    steps:
      - name: ValidateAgentInstallation
        action: ExecuteBash
        inputs:
          commands:
            - /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status
            - test -f /etc/sysctl.d/99-cis.conf

  - name: test
    steps:
      - name: ExecuteHealthCheck
        action: ExecuteBash
        inputs:
          commands:
            - echo "Running post-AMI boot validation..."
            - systemctl is-enabled amazon-cloudwatch-agent

Common built-in actions include ExecuteBash, ExecutePowerShell, ExecuteBinary, S3Download, S3Upload, and SetRegistryKey.

---

Container Image Recipes & Output to Amazon ECR

In addition to virtual machine AMIs, EC2 Image Builder supports Container Image Recipes to bake Docker and OCI-compliant container images:

1. Base Image Source: Can be sourced from Amazon ECR Public, private Amazon ECR repositories, or Docker Hub.
2. Dockerfile or Components: You can supply inline Dockerfile directives or attach modular Image Builder components that execute container build commands.
3. Testing Phase: Image Builder launches the container inside an isolated environment, runs container health probes and test scripts.
4. Target Distribution: Upon passing validation, the pipeline automatically signs, tags, and pushes the final container image directly to target Amazon ECR repositories across multiple AWS accounts and regions.

---

CIS Benchmark Hardening & Amazon Inspector Integration

Enterprise security compliance mandates strict alignment with Center for Internet Security (CIS) Benchmarks (Level 1 and Level 2 profiles).

• AWS-Managed Compliance Components: AWS provides pre-built, versioned CIS benchmark components directly inside Image Builder for Amazon Linux 2, Amazon Linux 2023, RHEL, and Windows Server.
• Amazon Inspector Integration: EC2 Image Builder integrates natively with Amazon Inspector to conduct automated Common Vulnerabilities and Exposures (CVE) scanning during the image build pipeline. If Inspector detects high or critical vulnerabilities exceeding corporate risk thresholds, Image Builder fails the pipeline execution, preventing vulnerable AMIs from being published.

---

Multi-Account Golden AMI Distribution Architecture

A critical DevOps professional pattern involves distributing AMIs across hundreds of spoke AWS accounts within an Organization.

Cross-Account AMI Sharing and KMS Key Permissions

When sharing an AMI encrypted with an AWS KMS Customer Managed Key (CMK) across AWS accounts, direct sharing requires two coordinated permission layers:

1. AMI Launch Permission: In the distribution configuration, share the AMI with the target AWS Account IDs or Organization path (arn:aws:organizations::123456789012:organization/o-abc123def).
2. KMS Key Policy Permission: The KMS CMK used to encrypt the underlying EBS snapshot in the build account must explicitly grant access to the target account's root principal or roles via kms:DescribeKey, kms:ReEncrypt*, kms:CreateGrant, and kms:Decrypt.

Exam Trap: AWS-managed KMS keys (aws/ebs) cannot be shared across accounts. You MUST use a Customer Managed Key (CMK) to encrypt the Golden AMI snapshots if the image is distributed to external AWS accounts.

{
  "Sid": "AllowSpokeAccountsToDecryptGoldenAMI",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::222233334444:root"
  },
  "Action": [
    "kms:DescribeKey",
    "kms:ReEncrypt*",
    "kms:CreateGrant",
    "kms:Decrypt"
  ],
  "Resource": "*"
}

Once shared, target accounts typically copy the shared AMI into their local region and re-encrypt the snapshot using their own local KMS CMK, isolating encryption boundaries.

---

Image Baking Strategies: Bake vs. Fry vs. Half-Baked