10.3 End-to-End MLOps CI/CD & Automated Retraining
Key Takeaways
- Amazon EventBridge acts as the real-time event bus connecting MLOps components, triggering SageMaker Pipelines on S3 data drops, Model Registry status changes, or Model Monitor drift alarms.
- SageMaker Pipelines is purpose-built for ML DAG orchestration with native Model Registry and Lineage tracking, whereas AWS Step Functions is designed for broad enterprise orchestration across disparate AWS services (Glue, Athena, EMR, Lambda).
- AWS CodePipeline and AWS CodeBuild provide automated CI/CD pipelines that execute unit tests, data quality validations, pipeline DAG compilation, and infrastructure-as-code (CDK/CloudFormation) blue/green endpoint deployments.
- Closed-loop automated retraining architectures continuously capture production inferences, detect drift via SageMaker Model Monitor, trigger CloudWatch alarms, and initiate automated retraining runs via EventBridge.
- Model promotion follows a decoupled pattern: engineers or evaluators approve a model package in the Model Registry (`Approved`), which emits an EventBridge event that triggers the staging/production deployment pipeline.
End-to-End MLOps CI/CD & Automated Retraining
Moving machine learning from research prototypes to enterprise production requires robust Continuous Integration, Continuous Delivery, and Continuous Training (CI/CD/CT) pipelines. An automated MLOps architecture must react to real-world events—such as new training data arriving in Amazon S3, human approval in the Model Registry, or statistical data drift detected by SageMaker Model Monitor—and orchestrate multi-step deployment workflows safely.
For the AWS Certified Machine Learning Engineer — Associate (MLA-C01) exam, you must master the integration between Amazon EventBridge, SageMaker Pipelines, AWS Step Functions, and AWS CodePipeline / AWS CodeBuild, and understand how to architect closed-loop automated retraining systems.
1. Event-Driven MLOps Architectures with Amazon EventBridge
Amazon EventBridge serves as the central nervous system of AWS MLOps architectures. By monitoring state changes emitted by AWS services, EventBridge rules can trigger automated actions without requiring polling loops or manual intervention.
+---------------------------------------------------------------------------------------------------+
| EVENT-DRIVEN MLOPS EVENTBRIDGE ARCHITECTURE |
| |
| EVENT PRODUCERS: |
| +--------------------------+ +---------------------------+ +------------------------------+ |
| | Amazon S3 (EventBridge) | | SageMaker Model Registry | | SageMaker Model Monitor | |
| | - New training data file | | - Package state changed | | - CloudWatch Drift Alarm | |
| | s3:ObjectCreated | | to "Approved" | | (Data / Concept Drift) | |
| +--------------------------+ +---------------------------+ +------------------------------+ |
| | | | |
| +-----------------------------+-------------------------------+ |
| | |
| v |
| [Amazon EventBridge Default Event Bus] |
| | |
| +-----------------------------+-------------------------------+ |
| | Event Pattern Filter: | Event Pattern Filter: | |
| | State == Approved | Source == aws.s3 | |
| v v v |
| TARGETS: +-------------------------+ +-------------------------+ +-------------------+ |
| | AWS CodePipeline | | SageMaker Pipeline | | Amazon SNS Alert | |
| | (Deploy Endpoint) | | (Trigger Retraining) | | (Notify On-Call) | |
| +-------------------------+ +-------------------------+ +-------------------+ |
+---------------------------------------------------------------------------------------------------+
Core EventBridge MLOps Event Patterns
Pattern 1: Triggering Automated Retraining on New S3 Data Arrival
When new batch data lands in an Amazon S3 bucket, S3 emits an Object Created event to EventBridge (requires S3 Event Notifications with EventBridge enabled):
{
"source": ["aws.s3"],
"detail-type": ["Object Created"],
"detail": {
"bucket": {
"name": ["mlops-training-data-production"]
},
"object": {
"key": [{"prefix": "incoming/daily-batch/"}]
}
}
}
- Target: Invokes
sagemaker:StartPipelineExecutionto launch the automated training pipeline DAG with the new S3 data URI passed as a parameter override.
Pattern 2: Triggering Deployment Pipeline on Model Approval
When an ML engineer or lead signs off on a model package in SageMaker Model Registry, changing its status from PendingManualApproval to Approved:
{
"source": ["aws.sagemaker"],
"detail-type": ["SageMaker Model Package State Change"],
"detail": {
"ModelPackageGroupName": ["CustomerChurnModels"],
"ModelApprovalStatus": ["Approved"]
}
}
- Target: Triggers an AWS CodePipeline deployment stage that compiles AWS CloudFormation or AWS CDK templates to update the production real-time endpoint using Blue/Green canary guardrails.
Pattern 3: Triggering Retraining on Model Monitor Drift Alarms
When SageMaker Model Monitor detects statistical feature drift (e.g., Kolmogorov-Smirnov test breach) or concept drift (accuracy drop), it sends metrics to Amazon CloudWatch. A CloudWatch Alarm enters the ALARM state, emitting an EventBridge event that kicks off automated model retraining.
2. Architectural Comparison: SageMaker Pipelines vs. AWS Step Functions
A critical decision on the MLA-C01 exam is selecting between SageMaker Pipelines and AWS Step Functions for workflow orchestration.
+---------------------------------------------------------------------------------------------------+
| SAGEMAKER PIPELINES VS. AWS STEP FUNCTIONS |
| |
| +---------------------------------------+ +-----------------------------------------------+ |
| | SAGEMAKER PIPELINES | | AWS STEP FUNCTIONS | |
| +---------------------------------------+ +-----------------------------------------------+ |
| | - Purpose-built for ML workflows | | - General-purpose enterprise orchestrator | |
| | - Native Model Registry integration | | - Connects 200+ AWS services (Glue, Athena) | |
| | - Native Lineage & Experiment tracking| | - Complex business processes & IT automation | |
| | - Python SDK-native (Data Science UX) | | - JSON/YAML Amazon States Language (ASL) | |
| | - Serverless control plane ($0) | | - Pay per state transition | |
| | - ML-specific step types & caching | | - Long-running workflows (up to 1 year) | |
| +---------------------------------------+ +-----------------------------------------------+ |
+---------------------------------------------------------------------------------------------------+
Detailed Architectural Decision Matrix
| Criteria | SageMaker Pipelines | AWS Step Functions |
|---|---|---|
| Primary Purpose | End-to-end machine learning lifecycle (Data prep, train, tune, eval, register). | Broad enterprise application workflows spanning multi-service architectures. |
| Developer Persona | Data Scientists and ML Engineers using the SageMaker Python SDK. | DevOps, Cloud Architects, and Backend Engineers using CDK, Terraform, or ASL. |
| ML Governance | Native: Automatic Lineage Tracking, Model Registry, Experiments, and Drift baselines. | Manual: Requires explicit API calls to SageMaker APIs in state task definitions. |
| Service Breadth | SageMaker-centric (Processing, Training, Tuning, Transform, Model Registry, Lambda). | 200+ AWS Services (AWS Glue, Amazon EMR, Amazon Athena, DynamoDB, Lambda, ECS). |
| Step Caching | Native: Built-in CacheConfig skipping redundant compute based on input ETags. | None: Must be custom-built using DynamoDB state lookups. |
| Cost Model | Free orchestration control plane; pay only for underlying SageMaker compute instances. | Pay-per-state transition (Standard Workflows) or duration (Express Workflows). |
Where do Apache Airflow and AWS CodeDeploy fit? Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is the managed option when an organization has standardized on open-source Apache Airflow DAGs to orchestrate data and ML jobs across AWS and third-party systems; choose MWAA when existing Airflow DAG investments must be preserved, and choose SageMaker Pipelines for ML-native governance (Lineage, Experiments, Model Registry). Within the developer-tools family, the blueprint also expects familiarity with AWS CodeDeploy for compute-layer deployment mechanics (EC2/ECS/Lambda blue-green and canary shifts); for SageMaker endpoints specifically, native Deployment Guardrails (Chapter 9) provide the equivalent blue/green, canary, and linear traffic shifting without CodeDeploy.
[!TIP] Exam Selection Rule:
- If the scenario involves ML-native training, model tuning, evaluation, Model Registry, and data science Python SDK authoring, choose SageMaker Pipelines.
- If the scenario involves enterprise-wide cross-service data pipelines spanning AWS Glue ETL, Amazon EMR clusters, Amazon Athena queries, and third-party SaaS before invoking ML, choose AWS Step Functions (or have Step Functions trigger a SageMaker Pipeline).
3. CI/CD Automation with AWS CodePipeline & AWS CodeBuild
Production ML systems require rigorous CI/CD to prevent buggy model inference code or malformed preprocessing scripts from breaking production endpoints.
+---------------------------------------------------------------------------------------------------+
| END-TO-END MLOPS CI/CD PIPELINE ARCHITECTURE |
| |
| [1. SOURCE STAGE] (AWS CodeCommit / GitHub) |
| - Triggers on git push to 'main' branch |
| - Contains: preprocessing.py, train.py, pipeline_dag.py, cdk_infra/ |
| | |
| v |
| [2. BUILD & TEST STAGE] (AWS CodeBuild) |
| - Runs unit tests (pytest) & linter (flake8) |
| - Validates data schemas & compiles pipeline DAG definition |
| - Calls `pipeline.upsert()` to update DAG in SageMaker Control Plane |
| | |
| v |
| [3. TRAINING & REGISTRATION STAGE] (SageMaker Pipelines) |
| - Executes automated training DAG |
| - Preprocesses data -> Trains model -> Evaluates AUC |
| - Registers new model package in SageMaker Model Registry (PendingManualApproval) |
| | |
| v |
| [4. STAGING DEPLOYMENT & INTEGRATION TEST] (AWS CodePipeline / CDK) |
| - Deploys model to staging endpoint |
| - Executes synthetic load tests & inference regression checks |
| | |
| v |
| [5. MANUAL / AUTOMATED APPROVAL GATE] (Model Registry State Change -> Approved) |
| | |
| v |
| [6. PRODUCTION DEPLOYMENT WITH GUARDRAILS] (AWS CloudFormation / CDK) |
| - Executes Blue/Green Canary update (10% traffic for 15-min baking period) |
| - Monitors CloudWatch Rollback Alarms (ModelLatency, 5XXErrors) |
| - Full promotion to 100% on success; Auto-rollback on alarm |
+---------------------------------------------------------------------------------------------------+
Key Components in the CI/CD Pipeline:
- AWS CodeBuild
buildspec.yml: Automates testing and DAG compilation:
version: 0.2
phases:
install:
runtime-versions:
python: 3.10
commands:
- pip install -r requirements.txt
pre_build:
commands:
- pytest tests/unit_tests/
- flake8 src/
build:
commands:
- python pipelines/customer_churn/pipeline.py --upsert
- python pipelines/customer_churn/pipeline.py --start
- SageMaker Model Registry Approval as the Decoupling Boundary: The CI pipeline updates the code and trains the model. The CD pipeline deploys the model. The two are cleanly decoupled by the Model Package Approval Status in the Model Registry.
4. Closed-Loop Continuous Retraining Architecture
In real-world applications, models suffer from concept drift (statistical relationships between features and targets change) and data drift (input feature distributions shift over time). A mature MLOps system implements a fully automated closed-loop continuous retraining architecture.
+---------------------------------------------------------------------------------------------------+
| CLOSED-LOOP AUTOMATED RETRAINING ARCHITECTURE |
| |
| +-----------------------------------+ |
| | Client Inference Requests | |
| +-----------------------------------+ |
| | |
| v |
| +-----------------------------------+ |
| | SageMaker Real-Time Endpoint | |
| +-----------------------------------+ |
| | |
| (SageMaker Data Capture: Payload Logging) |
| v |
| +-----------------------------------+ |
| | Amazon S3 Capture Bucket | |
| +-----------------------------------+ |
| | |
| v |
| +-----------------------------------+ |
| | SageMaker Model Monitor Job | |
| | (Hourly Baseline Comparison) | |
| +-----------------------------------+ |
| | |
| (Drift Threshold Breached) |
| v |
| +-----------------------------------+ |
| | CloudWatch Metric & Alarm | |
| +-----------------------------------+ |
| | |
| v |
| +-----------------------------------+ |
| | Amazon EventBridge Rule | |
| +-----------------------------------+ |
| | |
| (StartPipelineExecution API) |
| v |
| +-----------------------------------+ |
| | SageMaker Training Pipeline | |
| +-----------------------------------+ |
| | |
| (Model Evaluated & Registered) |
| v |
| +-----------------------------------+ |
| | SageMaker Model Registry | |
| +-----------------------------------+ |
| | |
| (Approved -> Blue/Green Canary) |
| v |
| +-----------------------------------+ |
| | Updated Real-Time Endpoint Fleet | |
| +-----------------------------------+ |
+---------------------------------------------------------------------------------------------------+
Step-by-Step Retraining Loop Breakdown:
- Data Capture: The SageMaker Endpoint is configured with
DataCaptureConfigto asynchronously record a percentage (e.g., 100%) of live request and response payloads to Amazon S3. - Drift Monitoring: A scheduled SageMaker Model Monitor job executes periodically (e.g., hourly or daily), comparing captured inference data against the baseline constraints generated during training.
- Alarm Generation: If the drift metric (e.g., data quality drift or prediction drift) violates the baseline threshold, Model Monitor publishes custom metrics to Amazon CloudWatch, placing an alarm in the
ALARMstate. - Event Routing: Amazon EventBridge catches the CloudWatch alarm state transition and triggers the target SageMaker Pipeline via the
StartPipelineExecutionAPI. - Retraining & Validation: The pipeline ingests the newly labeled dataset, retrains the model, evaluates performance metrics on held-out test data, and registers the new candidate model in the Model Registry.
- Safe Automated Promotion: Upon approval, a second EventBridge rule triggers a Blue/Green canary deployment with automated CloudWatch rollback alarms, safely updating the production endpoint with zero customer downtime.
A fraud detection machine learning model deployed to a real-time SageMaker endpoint has started experiencing performance degradation due to evolving fraud patterns (concept drift). The ML team wants to build an automated, event-driven retraining architecture that continuously checks production inferences for drift against the training baseline, automatically launches a SageMaker Pipeline retraining job when drift is detected, and notifies the team. Which combination of AWS services implements this closed-loop architecture with the least operational overhead?
An enterprise organization is designing an automated MLOps workflow. The workflow must first orchestrate complex data extraction across Amazon Athena, execute multi-table data transformations using AWS Glue ETL, initiate distributed feature extraction on an Amazon EMR Spark cluster, and finally trigger a SageMaker Pipeline for model training and registration. Which service should the team choose as the top-level workflow orchestrator?
An ML team uses AWS CodePipeline for MLOps CI/CD. The pipeline includes source control in GitHub, automated unit testing and pipeline registration via AWS CodeBuild, and automated model training via SageMaker Pipelines that registers candidates into the SageMaker Model Registry. The team requires that models are only deployed to the production SageMaker endpoint after a lead ML engineer inspects the model evaluation metrics and approves the model in the registry. Which event-driven pattern accomplishes this?
An ML engineer is writing an Amazon EventBridge rule pattern to automatically trigger an AWS CodePipeline deployment whenever a new model version in the CreditRiskModels Model Package Group is approved by a lead data scientist. Which EventBridge event pattern matches this specific condition?