12.2 Model Documentation & Auditing with Model Cards & Dashboard
Key Takeaways
SageMaker Model Cards provide standardized, centralized, immutable documentation of model lineage, intended business uses, risk ratings, training hyperparameters, evaluation metrics, and ethical considerations.
Model Cards automatically ingest metadata from SageMaker Training Jobs, Model Registry Model Packages, and Clarify Processing Jobs, and can be enriched manually via Studio or programmatically via the Python SDK and boto3.
Model Cards support cryptographic PDF exports stored in Amazon S3 for regulatory compliance audits across frameworks such as EU AI Act, NIST AI RMF, SR 11-7, and FDA requirements.
SageMaker Model Dashboard serves as a centralized single-pane-of-glass governance portal tracking all deployed models, Model Registry packages, associated Model Cards, active Model Monitor schedules, and CloudWatch alarms across AWS accounts.
Model Dashboard automatically detects governance gaps, highlighting unmonitored endpoints lacking Data Quality, Model Quality, Bias, or Explainability monitors.
Model Documentation & Auditing with Model Cards & Dashboard
As enterprise machine learning systems scale into production across multiple business units, organizations face significant operational and regulatory risks. Models deployed without centralized documentation, clear lineage tracking, defined operating boundaries, and active production monitoring create severe compliance liabilities. Global regulatory frameworks—such as the EU AI Act, NIST Artificial Intelligence Risk Management Framework (NIST AI RMF), Federal Reserve SR 11-7 Supervisory Guidance on Model Risk Management, and HIPAA/FDA clinical guidelines—mandate comprehensive, auditable documentation across the entire model lifecycle.
For the AWS Certified Machine Learning Engineer — Associate (MLA-C01) exam, you must master two key governance services in Amazon SageMaker: SageMaker Model Cards (standardized, auditable model documentation) and SageMaker Model Dashboard (centralized observability and governance monitoring across all account models).
1. Amazon SageMaker Model Cards Architecture & Schema
Amazon SageMaker Model Cards provide a standardized, immutable, and factory-integrated mechanism to record and share critical information about machine learning models throughout their development and production lifecycles.
+--------------------------------------------------------------------------------------------------+
| SAGEMAKER MODEL CARD FACTORY ARCHITECTURE |
| |
| [Automatic Metadata Ingestion] [Manual & Programmatic Enrichment] |
| - SageMaker Training Job Metadata - Intended Uses & Business Problem |
| (Algorithm, Instances, Hyperparameters) - Risk Rating (Low, Medium, High) |
| - Model Registry Package Details - Target User Demographics & Domain Constraints |
| (Model ARN, Framework Version, Approvals) - Ethical Considerations & Caveats |
| - SageMaker Clarify Processing Jobs - Human-in-the-Loop Oversight Guidelines |
| (Pre/Post-Training Bias, SHAP Metrics) |
| | | |
| v v |
| +------------------------------------------------------------------------------------------+ |
| | SAGEMAKER MODEL CARD | |
| | Status: Draft | PendingReview | Approved | Archived | |
| | Version: 1.0 (Immutable versioned audit trail in AWS Control Plane) | |
| +------------------------------------------------------------------------------------------+ |
| | | |
| v v |
| [SageMaker Model Dashboard] [Signed PDF Export in S3] |
| (Single-pane-of-glass governance) (Tamper-evident audit document) |
+--------------------------------------------------------------------------------------------------+
Model Card Information Architecture
A complete SageMaker Model Card is structured into six core standardized sections:
+--------------------------------------------------------------------------------------------------+
| SAGEMAKER MODEL CARD CORE SECTIONS |
| |
| 1. MODEL OVERVIEW & INTENDED USES |
| - Model Name, Version, Description, Owner, Creation Date |
| - Intended Business Use Case & Domain Scope |
| - Out-of-Scope Applications & Prohibited Use Cases |
| |
| 2. BUSINESS DETAILS & RISK RATING |
| - Business Problem Formulation & Financial/Operational Value |
| - Risk Rating: Low | Medium | High (Dictates approval gates & auditing frequency) |
| |
| 3. TRAINING DETAILS & LINEAGE |
| - Ingestion Sources: S3 Dataset URIs, Schemas, Row Counts |
| - Training Configuration: Algorithm (XGBoost/PyTorch), Instance Types, Hyperparameters |
| - Training Metrics: Objective loss curves, Training accuracy, Epoch runtimes |
| |
| 4. EVALUATION & BENCHMARK RESULTS |
| - Test Dataset Metadata & Validation Slicing |
| - Performance Metrics: ROC-AUC, F1 Score, Precision, Recall, RMSE |
| - Automated Clarify Ingestion: Bias Metrics (CI, DPL, DI) & Global SHAP Importance |
| |
| 5. ETHICAL CONSIDERATIONS & CAVEATS |
| - Demographic Subgroup Performance Gaps & Mitigations |
| - Known Edge-Case Failures & Performance Limitations |
| |
| 6. OPERATIONAL & HUMAN OVERSIGHT RECOMMENDATIONS |
| - Human-in-the-Loop (HITL) Thresholds for Manual Review |
| - Maintenance, Retraining Cadence, and Decommissioning Criteria |
+--------------------------------------------------------------------------------------------------+
Creation Workflows & Automation
- Automated Population from SageMaker Resources:
- When linked to an existing SageMaker Training Job, the model card automatically populates the algorithm container image, input dataset S3 locations, training instance types, hyperparameters, and environment variables.
- When linked to a SageMaker Model Registry Package, the model card automatically imports artifact locations, approvals, and framework metadata.
- When linked to a SageMaker Clarify Processing Job, the model card automatically imports pre-training bias metrics, post-training bias metrics, and SHAP feature attribution charts without manual data entry.
- Manual Enrichment via SageMaker Studio: Data scientists and compliance officers can collaborate interactively in SageMaker Studio UI to author business descriptions, risk assessments, and intended use boundaries.
- Programmatic Management via SDK / Boto3: Engineers can automate model card creation and updates within CI/CD pipelines using
sagemaker.model_cardor boto3create_model_card. - Auditor PDF Export: The
create_model_card_export_jobAPI generates a cryptographically signed, professionally formatted PDF report stored in Amazon S3, enabling immediate compliance handoffs for internal risk committees or external regulatory inspectors.
2. Amazon SageMaker Model Dashboard
Amazon SageMaker Model Dashboard is a centralized governance portal embedded directly within SageMaker Studio that provides a single pane of glass to track, monitor, and govern all machine learning models across an AWS account and region.
+--------------------------------------------------------------------------------------------------+
| SAGEMAKER MODEL DASHBOARD INTERFACE |
| |
| Total Models: 42 | Endpoints Active: 18 | Governance Violations: 4 | Active Alerts: 2 |
| +------------------------------------------------------------------------------------------+ |
| | Model Name | Model Card | Registry | Data Quality | Model Quality | Bias | CloudWatch| |
| | --------------- | ---------- | -------- | ------------ | ------------- | ---- | ----------| |
| | fraud-xgb-prod | Approved | Approved | Healthy | Healthy | Healthy| 0 Alarms |
| | churn-rf-v2 | Approved | Approved | Drift Alert! | Healthy | N/A | 1 ALARM |
| | credit-risk-v1 | Missing! | Approved | Unmonitored! | Unmonitored! | N/A | 0 Alarms |
| | recommender-v3 | Draft | Pending | Healthy | Drift Alert! | N/A | 1 ALARM |
| +------------------------------------------------------------------------------------------+ |
+--------------------------------------------------------------------------------------------------+
Core Governance Capabilities of Model Dashboard:
- Comprehensive Inventory Tracking:
- Automatically catalogs all SageMaker models, whether hosted on real-time endpoints, serverless endpoints, asynchronous endpoints, or registered in the Model Registry.
- Maps deployed endpoints back to their underlying model packages, container images, and S3 artifact locations.
- Governance Completeness Auditing:
- Instantly highlights governance gaps, such as models operating in production without an associated Model Card or with a Model Card left in
Draftstate. - Identifies models with missing risk ratings or unapproved registry statuses.
- Instantly highlights governance gaps, such as models operating in production without an associated Model Card or with a Model Card left in
- Production Model Monitor Health Tracking:
- Aggregates the scheduled execution status across all four SageMaker Model Monitor types:
- Data Quality: Detects feature distribution drift against baseline constraints.
- Model Quality: Detects accuracy, precision, and recall degradation against ground truth.
- Bias Drift: Detects changes in demographic fairness metrics over live inference traffic.
- Feature Attribution Drift: Detects shifts in SHAP feature importance relative to baseline.
- Surfaces unmonitored models that are receiving live production traffic without active monitoring schedules.
- Aggregates the scheduled execution status across all four SageMaker Model Monitor types:
- CloudWatch Alarm Aggregation:
- Directly consolidates CloudWatch alarms linked to endpoints (e.g., elevated
ModelLatency,Invocation5XXErrors, or custom drift alarms), allowing ML engineers to triage operational incidents without navigating across multiple AWS consoles.
- Directly consolidates CloudWatch alarms linked to endpoints (e.g., elevated
3. Python SDK Implementation: Programmatic Model Card Creation & Export
import boto3
import sagemaker
from sagemaker.model_card import (
ModelCard,
ModelOverview,
IntendedUses,
BusinessDetails,
TrainingDetails,
EvaluationDetail,
ModelCardStatusEnum,
RiskRatingEnum,
ObjectiveFunction
)
session = sagemaker.Session()
role = sagemaker.get_execution_role()
s3_bucket = session.default_bucket()
# 1. Define Model Overview and Intended Use Boundaries
overview = ModelOverview(
model_name="credit-risk-assessment-v2",
model_description="XGBoost binary classifier predicting loan default probability.",
model_version=2,
model_owner="mlops-governance-team@example.com",
model_status=ModelCardStatusEnum.PENDING_REVIEW
)
intended_uses = IntendedUses(
purpose_of_model="Automate credit risk scoring for retail personal loan applications under $50,000.",
intended_uses="Underwrite primary consumer lending applications submitted through online web portal.",
factors_affecting_model_efficiency="Economic recessions, rapid interest rate changes.",
risk_rating=RiskRatingEnum.HIGH, # High risk due to financial/fair lending impact
explanations_for_risk_rating="Model directly influences consumer credit approval decisions subject to ECOA."
)
# 2. Define Business Context
business_details = BusinessDetails(
business_problem="Reduce loan default rate by 15% while maintaining regulatory demographic parity.",
business_stakeholders="Chief Risk Officer, Retail Lending VP, Fair Lending Compliance Officer"
)
# 3. Populate Training and Evaluation Details (Automatically links to SageMaker Training Job)
training_details = TrainingDetails(
objective_function=ObjectiveFunction(description="Minimize binary classification log-loss: binary:logistic"),
training_job_details=None # Passing TrainingJobName auto-populates hyperparameters & S3 datasets
)
# 4. Instantiate and Populate the Model Card
model_card = ModelCard(
name="credit-risk-assessment-v2-card",
sagemaker_session=session,
model_overview=overview,
intended_uses=intended_uses,
business_details=business_details,
training_details=training_details
)
# Automatically ingest metadata from completed Training and Clarify jobs
model_card.load_training_job_details(training_job_name="credit-risk-xgb-training-2026-08")
# 5. Persist the Model Card to SageMaker
model_card.create()
# 6. Generate an Audit-Ready PDF Export in S3 for Regulators
export_job = model_card.export(
s3_output_path=f"s3://{s3_bucket}/governance-audits/model-cards/",
export_job_name="credit-risk-v2-compliance-export-2026"
)
print(f"Model card PDF export initiated: {export_job}")
4. Governance Comparison Table
| Feature Dimension | SageMaker Model Registry | SageMaker Model Cards | SageMaker Model Dashboard |
|---|---|---|---|
| Primary Purpose | Versioning, artifact packaging, approval state transitions, and CI/CD deployment automation. | Comprehensive, immutable, human/auditor-readable documentation of model lineage, risk, bias, and intended use. | Centralized single-pane-of-glass operational and governance monitoring portal across all account models. |
| Primary Audience | MLOps Engineers, DevOps, Automated Deployment Pipelines. | Compliance Officers, Risk Committees, External Regulators, Auditors. | ML Platform Engineers, Operations Teams, Governance Leads. |
| Key Artifacts | ModelPackageGroup, ModelPackageArn, approval status (Approved/Rejected). | ModelCard JSON object, cryptographically signed PDF reports in S3. | Studio Governance Console, aggregated drift alarms, compliance gap views. |
| Automation Source | CI/CD build scripts, SageMaker Pipelines RegisterModel step. | Auto-populated from Training Jobs, Registry, and Clarify Processing Jobs. | Continuous automated polling of account endpoints, monitors, and CloudWatch. |
A multinational financial services corporation is preparing for an annual regulatory audit under the Federal Reserve SR 11-7 model risk management guidelines. The compliance team requires a cryptographically verifiable, standardized document detailing the model architecture, training data provenance, hyperparameter configurations, pre-training and post-training Clarify bias metrics, and approved intended use boundaries for an automated credit scoring system. What is the most efficient and AWS-recommended method to generate this documentation?
Write a custom Python script that queries CloudWatch log streams and formats training logs into a CSV spreadsheet.
Instruct the MLOps engineering team to manually author a Confluence wiki page copying metrics from SageMaker Experiments.
Create an Amazon SageMaker Model Card linking the Training Job and Clarify Processing Job, populate the intended uses and risk rating, and execute a Model Card export job to output a signed PDF document to Amazon S3.
Export the AWS CloudTrail event history for all SageMaker API calls into an Amazon QuickSight dashboard.
A chief AI ethics officer discovers that several machine learning models deployed across production real-time endpoints in the enterprise AWS account lack data quality drift monitors, have not undergone bias evaluations, and operate without documented risk ratings or intended use boundaries. Which SageMaker feature provides the governance team with a unified, centralized dashboard to immediately identify unmonitored models, missing model cards, and active CloudWatch alarms across all deployed endpoints?
Amazon SageMaker Model Dashboard
Amazon SageMaker Model Registry
Amazon SageMaker Feature Store
AWS Lake Formation Console
An MLOps engineer is designing an automated CI/CD pipeline using SageMaker Pipelines to train and register computer vision defect detection models. The company's compliance policy dictates that every newly trained model must have its training hyperparameters, container image URI, S3 input datasets, and validation accuracy metrics recorded into a governance record before model promotion. Which programmatic approach satisfies this policy with the least operational overhead?
Store all hyperparameters in an Amazon DynamoDB table and use AWS Lambda to generate HTML documentation.
Create a SageMaker Model Card using the SageMaker Python SDK, call load_training_job_details() passing the training job name to automatically ingest training metadata, and persist the card using model_card.create().
Write custom Python parsing logic to extract hyperparameters from the model.tar.gz artifact in S3 and upload raw JSON to an S3 audit bucket.
Configure an Amazon EventBridge rule that captures EC2 instance launch events and writes instance types to AWS Secrets Manager.
In the SageMaker Model Card schema, an ML engineering team must assign a risk rating to a newly registered machine learning model. How does Amazon SageMaker categorize risk ratings within Model Cards, and what is the primary governance implication of this classification?
Risk ratings are boolean values (True or False) indicating whether the model container has unpatched Linux kernel vulnerabilities.
Risk ratings range from 1 to 10 based on the calculated Disparate Impact metric from SageMaker Clarify.
Risk ratings are dynamically computed based on the endpoint's monthly AWS infrastructure billing cost.
Risk ratings are categorical levels (Low, Medium, High) that reflect business impact and fair lending/safety risk, guiding review prioritization and auditing frequency.
Sections you finish are checked off in the contents.