6.1 Golden AMIs & Container Image Hardening Pipelines

6.1 Golden AMIs & Container Image Hardening Pipelines

Compute workloads represent the primary target for remote exploitation, unauthorized privilege escalation, and lateral movement in cloud environments. Securing virtual machines and containerized microservices requires an immutable infrastructure strategy: rather than patching and modifying running production instances, security teams construct pre-hardened, pre-configured machine images (Golden AMIs) and container images within automated continuous integration pipelines. Any update, vulnerability patch, or configuration adjustment produces a brand-new image version, replacing running workloads through rolling deployments.

This section examines the architecture and implementation of EC2 Image Builder pipelines, operating system hardening using Center for Internet Security (CIS) Benchmarks, secure multi-account and cross-Region KMS distribution patterns, container image hardening best practices, and Amazon Elastic Container Registry (Amazon ECR) supply-chain protections.

---

EC2 Image Builder Architecture & Pipelines

EC2 Image Builder is a fully managed AWS service that automates the creation, customization, security validation, and distribution of Amazon Machine Images (AMIs) and container images. It replaces legacy, unmanaged image-baking frameworks (such as custom Packer and Ansible scripts running on ad-hoc EC2 instances) with managed, auditable, and event-driven automation.

An EC2 Image Builder pipeline consists of four distinct architectural building blocks:

Loading diagram...

1. Image Recipes

An Image Recipe defines the source base image and the ordered sequence of components applied to construct and validate the image:

• Base Operating System: Selects an AWS-managed base image (e.g., Amazon Linux 2023, Red Hat Enterprise Linux 9, Windows Server 2022) or an existing custom AMI ARN.
• Working Directory & Storage Configuration: Defines root and secondary EBS volume configurations, including volume size, volume type (gp3, io2), IOPS, throughput, deletion on termination, and encryption using an AWS Key Management Service (AWS KMS) Customer Managed Key (CMK).
• Build Components: Step-by-step modular configuration documents (written in YAML) that execute software installations, OS parameter tuning, security daemon configuration, and CIS benchmark hardening.
• Test Components: Validation scripts that run against the finalized image inside an isolated test instance to assert compliance before the AMI is marked available for production distribution.

2. Component Documents (YAML Schema)

Components are declarative YAML documents conforming to the EC2 Image Builder component schema. A component defines phases (build, validate, and test) containing sequential execution steps.

Below is an example of an enterprise custom build component that disables legacy network protocols, configures kernel parameters via sysctl, and locks down SSH host configurations:

name: EnterpriseLinuxHardeningComponent
description: Disables unused protocols, configures sysctl, and hardens SSH configuration.
schemaVersion: 1.0
phases:
  - name: build
    steps:
      - name: DisableLegacyFilesystems
        action: ExecuteBash
        inputs:
          commands:
            - echo 'install cramfs /bin/true' > /etc/modprobe.d/cramfs.conf
            - echo 'install freevxfs /bin/true' >> /etc/modprobe.d/freevxfs.conf
            - echo 'install jffs2 /bin/true' >> /etc/modprobe.d/jffs2.conf
            - echo 'install hfs /bin/true' >> /etc/modprobe.d/hfs.conf
            - echo 'install hfsplus /bin/true' >> /etc/modprobe.d/hfsplus.conf
      - name: KernelSysctlHardening
        action: ExecuteBash
        inputs:
          commands:
            - sysctl -w net.ipv4.conf.all.accept_redirects=0
            - sysctl -w net.ipv4.conf.default.accept_redirects=0
            - sysctl -w net.ipv4.conf.all.secure_redirects=0
            - sysctl -w net.ipv4.conf.all.send_redirects=0
            - sysctl -w net.ipv4.conf.all.log_martians=1
            - sysctl -w net.ipv4.icmp_echo_ignore_broadcasts=1
            - sysctl -p
      - name: SSHDHardening
        action: ExecuteBash
        inputs:
          commands:
            - sed -i 's/^#PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
            - sed -i 's/^#MaxAuthTries.*/MaxAuthTries 3/' /etc/ssh/sshd_config
            - sed -i 's/^#PermitEmptyPasswords.*/PermitEmptyPasswords no/' /etc/ssh/sshd_config
            - sed -i 's/^#ClientAliveInterval.*/ClientAliveInterval 300/' /etc/ssh/sshd_config
            - sed -i 's/^#ClientAliveCountMax.*/ClientAliveCountMax 0/' /etc/ssh/sshd_config
            - systemctl restart sshd
  - name: validate
    steps:
      - name: VerifyRootSSHDisabled
        action: ExecuteBash
        inputs:
          commands:
            - grep -E '^PermitRootLogin no' /etc/ssh/sshd_config

3. Infrastructure Configuration

The Infrastructure Configuration defines the ephemeral EC2 environment provisioned to execute the recipe:

• Instance Profile: Specifies the IAM role required by the temporary build and test instances. The role must grant permissions for AWS Systems Manager (AmazonSSMManagedInstanceCore), S3 bucket logging (s3:PutObject), and KMS cryptographic operations (kms:Decrypt, kms:GenerateDataKey).
• Networking (VPC, Subnet, Security Groups): Defines where build instances launch. Best practice mandates deploying instances into a private subnet with egress access via a NAT Gateway or VPC Interface Endpoints, attached to a security group that allows no inbound traffic (0.0.0.0/0) whatsoever. Image Builder drives the build and test instances through the SSM Agent (Systems Manager), so no inbound access is required.
• Instance Type: Specifies the compute size (e.g., m5.large, c6g.large for Graviton) to match target hardware architectures.

4. Distribution Configuration

The Distribution Configuration defines the global release target for the built AMI:

• Cross-Region Replication: Distributes the AMI to multiple AWS Regions automatically (e.g., replicating from us-east-1 to eu-west-1 and ap-southeast-1).
• Target Accounts & Organizations: Grants launch permissions to specific AWS account IDs or an entire AWS Organizations Organizational Unit (OU).
• KMS Key Configuration: Designates the regional KMS CMKs used to re-encrypt the root and data volume snapshots in each destination Region.
• Launch Template & Parameter Store Publishing: Image Builder can automatically update Amazon EC2 Launch Templates with the new AMI ID and write the resulting AMI ARN to an AWS Systems Manager Parameter Store parameter (e.g., /golden-ami/linux/production/latest), triggering automated rolling deployments across consumer Auto Scaling groups.

---

CIS Benchmark Hardening: Level 1 vs Level 2 Profiles

The Center for Internet Security (CIS) publishes globally recognized consensus benchmarks for securing operating systems. EC2 Image Builder can apply CIS hardening through CIS-published hardening components (available with a CIS Hardened Images subscription in AWS Marketplace), apply DISA STIG settings through AWS-managed STIG components, or run your own components for either.

Understanding the architectural distinctions and operational trade-offs between CIS Level 1 and Level 2 is critical for the SCS-C03 exam:

Exam Tip: In exam questions involving mission-critical production web servers that handle real-time customer traffic, deploying a CIS Level 2 profile directly without prior staging often results in operational failures. For example, setting noexec on /tmp can prevent runtime compilation scripts from executing, and disabling SCTP or DCCP can break specialized telephony or streaming microservices. CIS Level 1 is the recommended standard baseline for general enterprise workloads, while Level 2 is reserved for regulated, high-security enclaves (e.g., payment processing, defense, and government systems).

---

Multi-Account & Cross-Region AMI Sharing Architecture

Enterprise cloud governance relies on a centralized Shared Services or Security Tooling AWS account where golden AMIs are baked and validated. Once validated, these AMIs must be shared with dozens or hundreds of spoke accounts (Production, Staging, Development) across multiple geographical Regions.

Loading diagram...

The KMS Key Policy Cross-Account Hurdle

When an AMI is created, its underlying root and data volumes are backed by Amazon EBS snapshots. If the AMI is encrypted, the snapshot is encrypted using an AWS KMS key. A foundational security principle frequently tested on the AWS Certified Security – Specialty exam governs this pattern:

1. AWS-Managed Key Limitation (aws/ebs): Snapshots encrypted with the default AWS-managed key (aws/ebs) cannot be shared across AWS accounts. AWS-managed key policies are immutable and cannot be edited to add external AWS account principals. Any attempt to share an AMI encrypted with aws/ebs will fail.
2. Customer Managed Key (CMK) Requirement: To share an encrypted AMI across accounts, the underlying snapshot must be encrypted with a Customer Managed Key (CMK).
3. Key Policy Permissions: The CMK key policy in the source account must explicitly grant the target account root or IAM principals the cryptographic permissions: kms:DescribeKey, kms:CreateGrant, kms:Decrypt, and kms:ReEncrypt*.

Below is the required KMS Key Policy statement applied to the CMK in the central tooling account:

{
  "Sid": "AllowCrossAccountSpokeAccess",
  "Effect": "Allow",
  "Principal": {
    "AWS": [
      "arn:aws:iam::111122223333:root",
      "arn:aws:iam::444455556666:root"
    ]
  },
  "Action": [
    "kms:DescribeKey",
    "kms:ReEncrypt*",
    "kms:CreateGrant",
    "kms:Decrypt"
  ],
  "Resource": "*"
}

Step-by-Step Cross-Account Launch Workflow

Granting launch permissions via ec2:ModifyImageAttribute (or sharing the AMI via AWS Resource Access Manager [AWS RAM]) allows the target account to see the AMI. Target accounts can launch directly from a shared encrypted AMI when the source key policy grants them access (Auto Scaling additionally needs a KMS grant for its service-linked role), but every launch then depends on the source account's key and AMI staying available.

The more resilient workflow is:

1. Share AMI: The central account shares the AMI with the target account and grants CMK decrypt permissions via the KMS key policy.
2. Copy & Re-encrypt: The target account invokes the ec2:CopyImage API, copying the shared AMI into its own local AWS account and Region while specifying its own local KMS Customer Managed Key.
3. Local Deployment: The target account launches EC2 instances or configures Auto Scaling Launch Templates using the locally copied, locally re-encrypted AMI.
4. Decouple: The central account can rotate or deprecate the source AMI without immediately breaking running workloads in the spoke accounts.

---

Container Image Hardening Principles

Containerization shifts the boundary of workload isolation from the hypervisor to the operating system kernel. While virtual machines isolate memory and CPU via hardware virtualization (Nitro / Xen), containers share the host Linux kernel (cgroups, namespaces). A vulnerability in an unhardened container can facilitate host privilege escalation, container breakouts, and compromise of adjacent containers on the same node.

Securing container images requires applying hardening principles across five key architectural domains:

Loading diagram...

1. Multi-Stage Docker Builds

Traditional single-stage Docker builds leave compilation toolchains (e.g., gcc, mvn, npm, golang-sdk), intermediate test dependencies, and temporary credentials baked into final production layers, vastly increasing the Common Vulnerabilities and Exposures (CVE) footprint. Multi-stage builds solve this by separating the build environment from the final execution runtime.

# Stage 1: Build environment (ephemeral, discarded)
FROM golang:1.23-alpine AS builder
WORKDIR /build
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /build/auth-service .

# Stage 2: Hardened runtime environment
FROM gcr.io/distroless/static-debian12:nonroot
WORKDIR /app
# Copy only the compiled binary from the builder stage
COPY --from=builder /build/auth-service /app/auth-service
# Enforce non-root execution context (distroless nonroot UID is 65532)
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/app/auth-service"]

2. Distroless and Minimal Base Images

Standard operating system base images (such as ubuntu:22.04 or debian:12) contain over 100 system utilities, package managers (apt, dpkg), network utilities (curl, wget, nc), and interactive command interpreters (/bin/sh, /bin/bash). If an attacker identifies a Remote Code Execution (RCE) flaw in an application running on such an image, they can immediately spawn a reverse shell or download malware.

Distroless base images contain exclusively application binaries and their direct runtime dependencies (e.g., glibc, SSL certificates, timezone data). They contain no package managers, no interactive shells, and no system utilities. An attacker who injects a shell command will encounter an execution failure because /bin/sh does not exist in the filesystem.

3. Non-Root Execution Context

By default, containers execute as the root user (UID 0) inside the container namespace. If an adversary achieves a container escape via a Linux kernel vulnerability (e.g., Dirty COW, Dirty Pipe), they inherit root privileges on the underlying host node.

• In the Dockerfile, explicitly declare a dedicated system user: USER 10001:10001.
• In Amazon ECS Task Definitions, enforce user: "10001:10001".
• In Kubernetes / Amazon EKS Pod Security Standards, set securityContext.runAsNonRoot: true and securityContext.runAsUser: 10001.

4. Read-Only Root Filesystems & Ephemeral tmpfs Mounts

Attackers who compromise an application typically attempt to write malicious payloads, scripts, or cryptominers to disk. Setting the container's root filesystem to read-only prevents unauthorized file modification, persistence, and execution.

• In Amazon ECS Task Definitions: set readonlyRootFilesystem: true within the containerDefinitions block.
• For applications requiring temporary scratch space (e.g., PID files, temporary caching), mount an in-memory tmpfs volume at /tmp or /run with the mount options noexec, nosuid, and nodev.

5. Linux Kernel Capabilities Lockdown

Linux divides root privileges into distinct units called capabilities. Standard Docker containers retain capabilities such as CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FOWNER, CAP_NET_RAW, and CAP_SYS_CHROOT. Hardened container specifications follow the principle of least privilege by dropping all capabilities and selectively adding back only strictly required capabilities.

Below is an Amazon ECS Task Definition snippet implementing capability dropping, read-only root filesystems, non-root execution, and tmpfs mounts:

{
  "family": "hardened-microservice",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "512",
  "memory": "1024",
  "containerDefinitions": [
    {
      "name": "api-worker",
      "image": "111122223333.dkr.ecr.us-east-1.amazonaws.com/payments:v2.1.0",
      "user": "10001:10001",
      "readonlyRootFilesystem": true,
      "linuxParameters": {
        "capabilities": {
          "drop": ["ALL"],
          "add": ["NET_BIND_SERVICE"]
        }
      },
      "mountPoints": [
        {
          "containerPath": "/tmp",
          "sourceVolume": "ephemeral-scratch"
        }
      ]
    }
  ],
  "volumes": [
    {
      "name": "ephemeral-scratch",
      "dockerVolumeConfiguration": {
        "scope": "task",
        "driver": "local",
        "driverOpts": {
          "type": "tmpfs",
          "device": "tmpfs",
          "o": "size=64m,noexec,nosuid,nodev"
        }
      }
    }
  ]
}

---

Amazon ECR Security Posture

Amazon Elastic Container Registry (Amazon ECR) acts as the central repository and supply-chain trust boundary for container images. Securing ECR involves three core mechanisms: tag immutability, KMS encryption, and vulnerability scanning.

1. Tag Immutability (imageTagMutability: IMMUTABLE)

In standard container registries, tags are mutable pointers. An automated CI/CD pipeline might build and push myapp:v1.0.0 or myapp:latest. In a mutable repository, an insider threat, compromised build server, or malicious actor can push a backdoored container image using an identical tag, silently overwriting the trusted production image.

Enabling Tag Immutability ensures that once an image tag is pushed, it cannot be overwritten, modified, or replaced under any circumstances. Any subsequent attempt to push an image using an existing tag results in an immediate ImageAlreadyExistsException API error. This guarantees reproducibility and prevents supply-chain tag tampering.

2. ECR Image Scanning: Basic vs Enhanced

ECR supports two scanning engines to identify software vulnerabilities in container layers:

• Basic Scanning: Uses AWS native scanning technology (it replaced the earlier Clair-based engine) to find operating system package CVEs. It supports scan on push and manual scans, at most once per image every 24 hours, so results are point-in-time.
• Enhanced Scanning: Powered by Amazon Inspector. Provides automated continuous scanning of both operating system packages and programming language package dependencies (e.g., Python, Node.js, Java, Go, Ruby). Enhanced scanning continuously re-evaluates stored container images whenever new CVEs are published in the National Vulnerability Database (NVD), alerting security teams to zero-day vulnerabilities in already-deployed images.

3. ECR Repository Policies & KMS Encryption

• KMS Envelope Encryption: ECR repositories can be encrypted at rest using server-side encryption with AWS KMS Customer Managed Keys (KMS) rather than default AWS-managed encryption (AES256). This allows granular control over which IAM principals can decrypt and pull container layers via KMS key policies.
• Repository Access Policies: Resource-based policies applied to individual repositories control cross-account pull/push permissions, restricting image consumption strictly to approved AWS Organizations accounts.

---

Specialty Exam Pitfalls & Architectural Traps

1. The aws/ebs Cross-Account Sharing Trap: Attempting to share an AMI encrypted with the default AWS-managed key aws/ebs across accounts will fail. You cannot modify the key policy of aws/ebs. To share an AMI across accounts, you must re-encrypt the snapshot with a Customer Managed Key (CMK) and grant cross-account permissions in the CMK key policy.
2. Launching Directly from Shared AMIs in Auto Scaling Groups: Even when a central account shares an encrypted AMI and grants KMS permissions, configuring an Auto Scaling group in a spoke account to launch instances directly from the external AMI creates operational fragility. If the central account modifies the KMS key policy or deprecates the AMI, downstream scaling events will fail with AccessDenied. The architecturally sound approach is for the target account to invoke ec2:CopyImage, re-encrypting the snapshot with its own local KMS CMK.
3. Read-Only Root Filesystems Breaking Legacy Daemons: Enabling readonlyRootFilesystem: true in ECS or Kubernetes without provisioning in-memory tmpfs mounts for /tmp, /var/run, or /var/log will cause web servers (like Nginx or Apache) and logging daemons to crash immediately on container launch as they attempt to create PID files or socket descriptors.
4. Mutable Tags in CI/CD Deployments: Relying on mutable tags such as latest or stable in ECS task definitions or Kubernetes manifests introduces non-deterministic deployments. If an image is updated or poisoned, rolling restarts will pull differing image digests under the same tag name. Always enforce Tag Immutability in ECR and reference specific version tags or cryptographic SHA256 image digests (@sha256:...).