13.3 Troubleshooting CI/CD Pipeline & Deployment Failures

AWS CodePipeline Failure Analysis & Cross-Account Permissions

AWS CodePipeline orchestrates continuous delivery workflows. When pipeline executions stall or fail, DevOps engineers must rapidly pinpoint whether the root cause stems from stage transitions, IAM role trust relationships, or cross-account artifact encryption.

Common CodePipeline Execution Failure Modes

1. Disabled Stage Transitions: A stage transition can be deliberately or programmatically disabled, preventing pipeline executions from advancing. The pipeline execution displays a Transition disabled indicator with a recorded reason.
2. Pipeline Concurrency Behavior: By default, CodePipeline uses Superseded mode: if a newer pipeline execution reaches a stage while an older execution is running, the newer execution overtakes the older execution, causing the older execution to be canceled.
3. Artifact Bucket Policy & Ownership Mismatches: When CodePipeline writes build or test output to an Amazon S3 artifact bucket, S3 bucket policies enforcing bucket owner full control (s3:PutObjectAcl with bucket-owner-full-control) will reject uploads if the writing role does not explicitly attach the required canned ACL.

Cross-Account Deployment Permissions & The KMS CMK Requirement

[!IMPORTANT] Critical DOP-C02 Concept: In cross-account CodePipeline architectures (e.g., Tooling Account A deploying to Production Account B), the default AWS-managed KMS key (aws/s3) CANNOT be used. AWS-managed keys do not permit cross-account key policies. An AWS KMS Customer Managed Key (CMK) is mandatory.

[ Tooling Account A (Pipeline Hub) ]
  ├── CodePipeline Service Role
  └── S3 Artifact Bucket (Encrypted with KMS Customer Managed Key)
                                  │
            ┌─────────────────────┴─────────────────────┐
            ▼                                           ▼
[ KMS Key Policy (Account A) ]               [ S3 Bucket Policy (Account A) ]
 Grants kms:Decrypt & GenerateDataKey          Grants s3:GetObject* & PutObject*
 to Account B Cross-Account Role               to Account B Cross-Account Role
            ▲                                           ▲
            └─────────────────────┬─────────────────────┘
                                  │ (STS AssumeRole)
                    [ Production Account B (Deploy Target) ]
                      └── CodeDeploy / CloudFormation Deployer Role

Triad of Cross-Account Requirements

1. KMS Key Policy (in Account A): Must explicitly grant kms:Decrypt, kms:DescribeKey, and kms:GenerateDataKey* to the deployment IAM role in Account B.
2. S3 Bucket Policy (in Account A): Must grant s3:GetObject*, s3:PutObject*, and s3:ListBucket to the deployment IAM role in Account B.
3. Deployment IAM Role (in Account B): Must possess IAM permissions allowing kms:Decrypt against the Account A KMS Key ARN, and must have a trust policy allowing CodePipeline in Account A (codepipeline.amazonaws.com) or the Account A Pipeline Role to assume it.

AWS CodeBuild Failure Modes & Diagnostics

AWS CodeBuild provisions isolated Docker containers to execute build specifications (buildspec.yml). When builds fail, DevOps engineers must systematically evaluate container networking, caching, credentials, and build phases.

1. VPC Subnet Routing & Missing Outbound Connectivity

A classic scenario tested on the DOP-C02 exam involves running CodeBuild inside a custom VPC (e.g., to access internal Amazon RDS databases, private code repositories, or internal microservices):

• The Failure: CodeBuild fails during the pre_build phase with timeout errors when pulling public Docker images from Docker Hub, downloading npm/pip packages, or connecting to AWS public endpoints (S3, ECR, Secrets Manager).
• The Cause: The CodeBuild project was configured to attach to public subnets with an Internet Gateway (IGW). CodeBuild provisions Elastic Network Interfaces (ENIs) inside the chosen subnets. However, CodeBuild ENIs do not receive public IPv4 addresses. Even if the subnet's route table points 0.0.0.0/0 to an Internet Gateway, traffic cannot egress because the ENIs have only private IPs and no public NAT mapping.
• The Resolution: CodeBuild ENIs MUST be placed in private subnets whose route tables route 0.0.0.0/0 through a NAT Gateway, or VPC Endpoints must be deployed for all required AWS services (ECR, S3, CloudWatch Logs, Secrets Manager).

2. Docker Daemon Initialization & Privileged Mode

When building Docker images inside CodeBuild, the buildspec.yml typically executes docker build or docker-compose. If the build fails with:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

• Root Cause: The CodeBuild project environment lacks root/daemon capabilities. Docker-in-Docker operations require enabling the privileged: true flag in the CodeBuild project settings (Environment.PrivilegedMode in CloudFormation).

3. ECR Authentication & Layer Caching

• ECR Login Failures: Running aws ecr get-login-password fails with AccessDeniedException if the CodeBuild service role lacks ecr:GetAuthorizationToken or repository permissions (ecr:BatchCheckLayerAvailability, ecr:BatchGetImage, ecr:PutImage).
• Missing Docker Layer Caching: Long build durations indicate Docker layer caching is inactive. To resolve, configure CodeBuild Local Docker Layer Cache (stores layers directly on the build host for repeated builds on same compute) or S3 Cache (persists layers across builds into an S3 bucket).

4. Buildspec Phase Exit Codes & Environment Variable Decryption

CodeBuild executes commands across sequential phases: install, pre_build, build, and post_build. By default, if any command in a phase returns a non-zero exit code, CodeBuild immediately terminates the build.

• Parameter Store / Secrets Manager Decryption Failures: If buildspec.yml references secrets in env.parameter-store or env.secrets-manager, CodeBuild attempts to resolve them before executing phases. If the CodeBuild service role lacks ssm:GetParameters, secretsmanager:GetSecretValue, or kms:Decrypt permissions, the build fails immediately during container initialization before any phase commands run.

---

AWS CodeDeploy Failure Modes & Resolution Strategies

AWS CodeDeploy automates application deployments across Amazon EC2, AWS Lambda, and Amazon ECS. Each deployment targets lifecycle event hooks defined in the appspec.yml.

Diagnostic Matrix for CodeDeploy Failures

---

AWS CloudFormation Rollback Triggers & Debugging

When a CloudFormation stack update fails, CloudFormation automatically initiates a rollback to the last known stable state. While this protects production from broken states, it destroys the failed resources and logs, complicating root-cause analysis.

Rollback Triggers & Monitoring Time

Rollback Triggers allow CloudFormation to monitor Amazon CloudWatch Alarms during stack creation or updates. If any configured alarm triggers during the deployment or during the specified monitoring time (e.g., MonitoringTimeInMinutes: 30 after resource creation completes), CloudFormation rolls back the entire deployment.

Preserving Failed Stacks for Root-Cause Analysis

To prevent CloudFormation from rolling back failed resources during troubleshooting:

• CLI / Pipeline Setting: Use --disable-rollback (or in CloudFormation console, set Stack failure options to Preserve successfully provisioned resources / OnFailure: DO_NOTHING).
• Forensic Capability: The failed resource remains in CREATE_FAILED status. Engineers can connect to the underlying EC2 instance via AWS Systems Manager Session Manager, inspect application logs in /var/log/, review cfn-init.log and cfn-wire.log, and diagnose configuration issues before recreating the stack.

Recovering from UPDATE_ROLLBACK_FAILED

If a resource cannot be rolled back (e.g., an S3 bucket cannot be deleted during rollback because objects were added during deployment, or an elastic IP was manually released), the stack enters UPDATE_ROLLBACK_FAILED.

• Resolution: Use the ContinueUpdateRollback API and specify the non-rollbackable resource in ResourcesToSkip. CloudFormation skips the specified resource and restores the stack to an UPDATE_ROLLBACK_COMPLETE operational state.