13.1 IAM Roles and Least Privilege for ML
Key Takeaways
- Amazon SageMaker uses a service execution role model where the SageMaker service principal (sagemaker.amazonaws.com) assumes a customer-managed IAM role to access S3 buckets, ECR repositories, CloudWatch Logs, and KMS keys on behalf of the user.
- The iam:PassRole permission allows developers and automated CI/CD pipelines to pass an IAM execution role to SageMaker APIs (CreateTrainingJob, CreateProcessingJob, CreateModel), and must be strictly scoped to specific role ARNs and the PassedToService: sagemaker.amazonaws.com condition.
- Least-privilege SageMaker execution roles must restrict Amazon S3 permissions (s3:GetObject, s3:PutObject) to designated project prefixes and ECR permissions (ecr:GetDownloadUrlForLayer, ecr:BatchGetImage) to specific algorithm repositories.
- Customer-Managed KMS Keys (CMKs) require explicit key policies granting kms:Decrypt, kms:GenerateDataKey, and kms:CreateGrant to the SageMaker execution role for both S3 dataset decryption and EBS volume encryption.
- IAM condition keys such as sagemaker:InstanceTypes, sagemaker:VolumeKmsKey, and sagemaker:VpcSubnets enforce organizational guardrails preventing costly instance launches, unencrypted storage, or jobs running outside private VPCs.
IAM Roles and Least Privilege for ML
Security is a shared responsibility across all layers of the machine learning lifecycle on AWS. In enterprise machine learning environments, data scientists, ML engineers, automated CI/CD pipelines, and managed compute clusters interact with highly sensitive training datasets, proprietary model weights, container registries, and specialized hardware accelerators. Implementing least-privilege access control and robust identity governance is essential to protect intellectual property, satisfy regulatory compliance frameworks (such as GDPR, HIPAA, and PCI-DSS), and prevent costly unauthorized resource usage.
On the AWS Certified Machine Learning Engineer — Associate (MLA-C01) exam, you must master the architecture of AWS Identity and Access Management (IAM) as it applies to Amazon SageMaker and related ML services. You will be tested on the relationship between user identities and SageMaker execution roles, the mechanics and security constraints of iam:PassRole, granular S3 and ECR resource policies, customer-managed KMS key delegation, and IAM condition keys that enforce security guardrails across training and inference workloads.
1. SageMaker IAM Architecture & Identity Delegation
Amazon SageMaker operates on an assumed execution role model. Unlike standard EC2 instances where applications run under an attached instance profile, SageMaker compute clusters (for training, processing, batch transform, and real-time inference) are managed by the AWS SageMaker service control plane. To access your AWS resources (such as downloading training datasets from S3 or pulling container images from ECR), SageMaker must assume an IAM role that you specify.
+--------------------------------------------------------------------------------------------------+
| SAGEMAKER IAM DELEGATION & PASSROLE FLOW |
| |
| +---------------------------------------+ |
| | Caller Identity (User / CI/CD) | |
| | - Role / User: DataScientistAdmin | |
| | - Calls: sagemaker:CreateTrainingJob |
| | - Requires: iam:PassRole permission| |
| +---------------------------------------+ |
| | |
| | 1. API Call with ExecutionRoleArn |
| v |
| +------------------------------------------------------------------------------------------+ |
| | SAGEMAKER SERVICE CONTROL PLANE (AWS Managed) | |
| | Service Principal: sagemaker.amazonaws.com | |
| +------------------------------------------------------------------------------------------+ |
| | |
| | 2. sts:AssumeRole (Validates Trust Policy) |
| v |
| +------------------------------------------------------------------------------------------+ |
| | SAGEMAKER EXECUTION ROLE (Customer IAM) | |
| | Trust Policy: Allow sagemaker.amazonaws.com to AssumeRole | |
| | Permissions Policy: | |
| | - Amazon S3: Read s3://ml-bucket/train/*, Write s3://ml-bucket/models/* | |
| | - Amazon ECR: Pull custom container image | |
| | - CloudWatch Logs: Stream stdout/stderr container logs | |
| | - AWS KMS: Decrypt training data & encrypt output model artifacts | |
| +------------------------------------------------------------------------------------------+ |
| | |
| | 3. Assumes Role & Obtains Ephemeral STS Tokens |
| v |
| +------------------------------------------------------------------------------------------+ |
| | EPHEMERAL TRAINING / PROCESSING / INFERENCE COMPUTE INSTANCE | |
| | Mounted with Temporary Credentials (/opt/ml environment) | |
| +------------------------------------------------------------------------------------------+ |
+--------------------------------------------------------------------------------------------------+
The Trust Policy (AssumeRole)
Every SageMaker execution role must include a Trust Policy (resource-based policy on the IAM role itself) that grants the SageMaker service principal permission to assume the role via AWS Security Token Service (sts:AssumeRole). Without this trust relationship, any SageMaker API call referencing the role will fail immediately:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "SageMakerAssumeRoleTrustPolicy",
"Effect": "Allow",
"Principal": {
"Service": "sagemaker.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
The iam:PassRole Mechanism & Security Scoping
When a developer, data scientist, or CI/CD pipeline role initiates an asynchronous SageMaker workload (such as calling CreateTrainingJob, CreateProcessingJob, CreateTransformJob, or CreateModel), they pass the ARN of the execution role to the SageMaker API. SageMaker then assumes that role to perform the work.
To prevent unauthorized privilege escalation, AWS requires the calling user to have the iam:PassRole permission.
[!WARNING] Privilege Escalation Risk: If an IAM user has
iam:PassRolegranted withResource: "*", that user can pass a highly privileged administrator role to a SageMaker training job or notebook, write a custom Python script inside the training job to execute arbitrary administrative actions (such as creating new admin users or exporting sensitive databases), and completely bypass their own permission boundaries.
To enforce least privilege, the calling identity's iam:PassRole permission must be strictly scoped down using two mandatory controls:
- Explicit Resource ARNs: Restrict the allowed roles to specific, pre-approved SageMaker execution role ARNs.
iam:PassedToServiceCondition: Enforce that the role can only be passed tosagemaker.amazonaws.com, preventing the user from passing the role to other services (such as EC2 or Lambda).
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ScopedPassRoleForSageMakerJobs",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::123456789012:role/SageMaker-ProjectX-ExecutionRole",
"Condition": {
"StringEquals": {
"iam:PassedToService": "sagemaker.amazonaws.com"
}
}
}
]
}
[!TIP] SageMaker Role Manager: Instead of hand-authoring execution-role JSON from scratch, SageMaker Role Manager (in the SageMaker console) generates least-privilege IAM roles from persona-based templates (such as data scientist or MLOps engineer) with pre-scoped permissions for common SageMaker activities — the fastest way to bootstrap compliant execution roles.
2. Least-Privilege Execution Role Policies
While the AWS-managed policy AmazonSageMakerFullAccess is useful for sandbox prototyping, it grants broad read/write access to all S3 buckets, full ECR access, and extensive SageMaker API administrative rights. In production enterprise architectures, you must construct customer-managed least-privilege execution policies divided into discrete resource access domains.
+--------------------------------------------------------------------------------------------------+
| LEAST-PRIVILEGE PERMISSION MATRIX |
| |
| Service Domain Minimum Required Actions Scoping / Resource Target |
| --------------- --------------------------------------- ---------------------------------- |
| Amazon S3 s3:GetObject, s3:ListBucket arn:aws:s3:::corp-ml-data/train/* |
| s3:PutObject, s3:AbortMultipartUpload arn:aws:s3:::corp-ml-models/output/* |
| |
| Amazon ECR ecr:BatchCheckLayerAvailability, arn:aws:ecr:region:acc:repository/ |
| ecr:GetDownloadUrlForLayer, custom-xgboost-training |
| ecr:BatchGetImage |
| ecr:GetAuthorizationToken Resource: "*" (Token auth only) |
| |
| CloudWatch Logs logs:CreateLogStream, logs:PutLogEvents, arn:aws:logs:region:acc:log-group: |
| logs:DescribeLogStreams /aws/sagemaker/TrainingJobs:* |
| |
| AWS KMS kms:Decrypt, kms:DescribeKey, arn:aws:kms:region:acc:key/ |
| kms:GenerateDataKey, kms:CreateGrant cmk-key-id-for-ml |
+--------------------------------------------------------------------------------------------------+
2.1 Granular S3 Access Policy
SageMaker training jobs need read-only access to input datasets and write access to model artifact output locations. Granting broad bucket permissions allows accidental overwrites or data leakage:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ReadTrainingDataOnly",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::corp-ml-datasets",
"arn:aws:s3:::corp-ml-datasets/credit-risk/v1/*"
]
},
{
"Sid": "WriteModelArtifactsOnly",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:AbortMultipartUpload"
],
"Resource": [
"arn:aws:s3:::corp-ml-model-artifacts/credit-risk/v1/*"
]
}
]
}
2.2 Granular Amazon ECR Permissions
To pull Docker images for training and hosting, the execution role requires ECR permissions. Note that ecr:GetAuthorizationToken does not support resource-level scoping and must be granted on *, while layer download and batch image actions must be restricted to specific repository ARNs:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ECRAuthToken",
"Effect": "Allow",
"Action": "ecr:GetAuthorizationToken",
"Resource": "*"
},
{
"Sid": "ECRRepositoryPull",
"Effect": "Allow",
"Action": [
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage"
],
"Resource": "arn:aws:ecr:us-east-1:123456789012:repository/credit-risk-containers"
}
]
}
2.3 CloudWatch Logs Scoping
SageMaker containers stream stdout and stderr logs to Amazon CloudWatch Logs. To ensure operational visibility without granting account-wide logging permissions, scope the log groups to the standard SageMaker namespaces:
/aws/sagemaker/TrainingJobs/aws/sagemaker/ProcessingJobs/aws/sagemaker/Endpoints/aws/sagemaker/TransformJobs
3. Customer-Managed KMS Key (CMK) Policies & Delegation
When training data in Amazon S3 or SageMaker EBS storage volumes are encrypted using AWS Key Management Service Customer Managed Keys (SSE-KMS), the SageMaker execution role requires explicit KMS permissions.
Access to KMS keys is governed by both the KMS Key Policy (resource-based policy on the key) and the IAM Execution Role Policy (identity-based policy). Both must permit the action.
+--------------------------------------------------------------------------------------------------+
| AWS KMS PERMISSIONS FOR SAGEMAKER |
| |
| Operation Required KMS API Actions |
| ------------------------------- ----------------------------------------------------------- |
| Read Encrypted S3 Training Data kms:Decrypt, kms:DescribeKey |
| Write Encrypted S3 Artifacts kms:GenerateDataKey, kms:Encrypt, kms:DescribeKey |
| Encrypt Attached EBS Volumes kms:CreateGrant, kms:GenerateDataKeyWithoutPlaintext, |
| kms:Decrypt, kms:DescribeKey |
+--------------------------------------------------------------------------------------------------+
Why kms:CreateGrant is Required for EBS Volume Encryption
When you specify a VolumeKmsKeyId for a SageMaker Training Job, Processing Job, or Endpoint, SageMaker must attach an encrypted Amazon EBS volume to the ephemeral EC2 instances. Because SageMaker provisions and manages these instances in the background, the SageMaker service requires permission to create an ephemeral cryptographic grant on the KMS key. The grant delegates key usage directly to the underlying AWS storage subsystem for the duration of the job.
Customer KMS Key Policy Statement for SageMaker Execution Role:
{
"Sid": "AllowSageMakerExecutionRoleKMSUsage",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:role/SageMaker-ProjectX-ExecutionRole"
},
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey",
"kms:DescribeKey",
"kms:CreateGrant"
],
"Resource": "*",
"Condition": {
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
}
Cross-Account KMS Access Pattern
In enterprise multi-account architectures, training datasets often reside in a centralized Data Lake Account (Account A), while SageMaker training jobs execute in a Data Science Project Account (Account B):
- Account A (Data Lake): The KMS Key Policy and S3 Bucket Policy in Account A must grant cross-account permissions (
kms:Decrypt,s3:GetObject) to the SageMaker Execution Role ARN in Account B. - Account B (ML Account): The IAM Execution Role in Account B must include an identity-based policy allowing
kms:Decryptagainst Account A's KMS Key ARN ands3:GetObjectagainst Account A's S3 bucket.
4. Enterprise IAM Guardrails & Condition Keys
Security administrators use IAM policy conditions as guardrails to prevent data scientists or automated pipelines from creating non-compliant or overly expensive resources. Amazon SageMaker provides service-specific condition keys that can be attached to Developer IAM roles or AWS Organizations Service Control Policies (SCPs).
+--------------------------------------------------------------------------------------------------+
| SAGEMAKER IAM CONDITION KEYS SUMMARY |
| |
| Condition Key Policy Enforcement Target |
| ------------------------------- ----------------------------------------------------------- |
| sagemaker:InstanceTypes Restricts allowed compute instance types for training, |
| processing, and endpoints (prevents costly p4de/p5 launches) |
| sagemaker:VolumeKmsKey Enforces that EBS storage volumes must be encrypted with a |
| specific customer-managed KMS key |
| sagemaker:VpcSubnets Mandates that training/endpoint instances must be deployed |
| inside specified private VPC subnets |
| sagemaker:VpcSecurityGroupIds Mandates attachment of designated VPC security groups |
| sagemaker:NetworkIsolation Enforces EnableNetworkIsolation=True on training/models |
| aws:RequestTag / aws:ResourceTag Enforces tag-based access control (e.g., CostCenter, Project) |
+--------------------------------------------------------------------------------------------------+
Example: Guardrail Policy Enforcing Instance Types, KMS, and VPC Subnets
The following IAM policy statement prevents developers from creating SageMaker training jobs unless they use approved instance families, attach an authorized KMS key for EBS encryption, and run inside designated private VPC subnets:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "EnforceTrainingGuardrails",
"Effect": "Deny",
"Action": "sagemaker:CreateTrainingJob",
"Resource": "arn:aws:sagemaker:*:123456789012:training-job/*",
"Condition": {
"ForAnyValue:StringNotLike": {
"sagemaker:InstanceTypes": [
"ml.m5.*",
"ml.c5.*",
"ml.g4dn.*"
]
}
}
},
{
"Sid": "EnforceVolumeEncryptionAndVpcAttachment",
"Effect": "Deny",
"Action": "sagemaker:CreateTrainingJob",
"Resource": "arn:aws:sagemaker:*:123456789012:training-job/*",
"Condition": {
"Null": {
"sagemaker:VolumeKmsKey": "true",
"sagemaker:VpcSubnets": "true"
}
}
}
]
}
[!TIP] Exam Rapid Decision Rules:
- If a developer receives
AccessDeniedExceptionwhen callingCreateTrainingJob$\rightarrow$ Check the developer's IAM role for missingiam:PassRolepermissions targeting the execution role ARN.- If a training job fails during EBS volume allocation $\rightarrow$ Check the KMS key policy for missing
kms:CreateGrantpermissions on the SageMaker execution role.- If the exam requires preventing unauthorized expensive instances (e.g.
ml.p4de.24xlarge) $\rightarrow$ Apply an IAM policy or SCP using thesagemaker:InstanceTypescondition key.- If cross-account data decryption fails $\rightarrow$ Both the resource policy (KMS Key Policy in Data Lake account) and the identity policy (IAM Role in ML account) must explicitly allow
kms:Decrypt.
An ML engineer is building an automated AWS CodePipeline CI/CD pipeline to deploy machine learning training jobs. When the pipeline executes the SageMaker CreateTrainingJob API call using an IAM service role, the pipeline fails with an AccessDeniedException indicating that the role is not authorized to perform actions on the specified SageMaker execution role. Which permission must be granted to the CI/CD pipeline IAM service role to resolve this error following the principle of least privilege?
A security architect wants to implement an organization-wide Service Control Policy (SCP) to prevent data scientists from creating SageMaker training jobs that use costly multi-GPU instance families (such as ml.p4de.24xlarge and ml.p5.48xlarge), while permitting cost-effective CPU and entry-level GPU instance types. Which IAM condition key should the architect use in the SCP statement?
A SageMaker training job executing in a private subnet fails immediately during startup with a KMS.AccessDeniedException while attempting to provision encrypted storage for the training job. The training job configuration specifies a Customer Managed Key (CMK) under the VolumeKmsKeyId parameter. What permission was most likely missing from the KMS Key Policy for the SageMaker execution role?
A data science team needs to configure an IAM execution role for a production SageMaker real-time endpoint that hosts an XGBoost model. The model reads real-time feature baseline files from S3 and streams operational metrics to CloudWatch Logs. According to AWS least-privilege security best practices, which set of permissions should be attached to this execution role?