11.1 Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions

Fine-Grained Access Control with AWS IAM Policies, Roles & Conditions

Security is a zero-compromise domain in data engineering. Building robust data pipelines on AWS requires strict enforcement of the Principle of Least Privilege. AWS Identity and Access Management (IAM) provides the foundation for authentication and authorization across all data processing, storage, and analytics services. For the AWS Certified Data Engineer - Associate (DEA-C01) exam, you must master IAM policy structure, permission evaluation logic, Attribute-Based Access Control (ABAC), and service-specific Fine-Grained Access Control (FGAC) techniques.

---

IAM Policy Architecture & Evaluation Logic

Every access request made to an AWS API is evaluated by the IAM policy engine. To grant or restrict access, IAM evaluates several policy types attached to or surrounding an IAM principal:

1. Identity-Based Policies: Attached directly to IAM Users, Groups, or Roles (Managed or Inline).
2. Resource-Based Policies: Attached directly to AWS resources (e.g., S3 Bucket Policies, KMS Key Policies, Lambda Function Policies, Glue Resource Policies).
3. Permissions Boundaries: Advanced features that set the maximum allowable permissions an identity-based policy can grant.
4. Service Control Policies (SCPs): Governance boundaries applied at the AWS Organizations OU or Account level.
5. Session Policies: Passed programmatically during temporary session creation (sts:AssumeRole).

The IAM Policy Evaluation Flow

Understanding policy evaluation order is a high-frequency exam topic:

• Default State: By default, all requests are implicitly denied (Implicit Deny).
• Explicit Deny Overrides Everything: If any policy applicable to the request contains an Explicit Deny, the final decision is immediately DENIED, regardless of how many Explicit Allow statements exist.
• Explicit Allow and Boundaries: An identity- or resource-based allow must apply, and guardrails such as permissions boundaries, session policies, SCPs, and resource control policies must also allow the request where their evaluation model requires an intersection. A resource-based policy can affect this evaluation differently for a user, role, or role session, so use the documented matrix for edge cases.

Request Received --> Explicit deny anywhere?
                       |-- YES --> [ DENIED ]
                       +-- NO  --> Applicable identity/resource allow?
                                      |-- NO --> [ DENIED (Implicit) ]
                                      +-- YES --> Required boundary, session, SCP/RCP intersections permit it?
                                                     |-- YES --> [ ALLOWED ]
                                                     +-- NO  --> [ DENIED ]

---

Fine-Grained Access Control (FGAC) with IAM Conditions

While basic IAM policies grant broad permissions on actions (e.g., s3:GetObject), data engineering workflows require fine-grained constraints based on runtime context. IAM Condition blocks restrict permissions using key-value comparisons.

Essential Global Condition Keys for Data Pipelines

Attribute-Based Access Control (ABAC)

Instead of creating separate IAM policies for every team or project, ABAC allows you to scale security dynamically using tags. For instance, an analytics team member tagged Department=Finance can access supported AWS resources whose authorization documentation exposes compatible resource-tag condition keys. For S3 general purpose buckets, first enable S3 ABAC in the account and Region before relying on bucket tags for authorization. The following policy demonstrates the pattern after enablement; do not assume every database object supports IAM resource tags:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ABACS3BucketRead",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::corporate-data-lake",
        "arn:aws:s3:::corporate-data-lake/*"
      ],
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/Department": "${aws:PrincipalTag/Department}"
        }
      }
    }
  ]
}

---

Service-Specific FGAC Implementation Patterns

1. Amazon S3 Virtual Folder Security (s3:prefix & s3:delimiter)

Data lakes partition data into S3 prefixes (e.g., s3://my-lake/raw/finance/ vs s3://my-lake/raw/hr/). To restrict a team's read access strictly to their folder, condition keys MUST regulate both s3:GetObject and s3:ListBucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowListBucketPrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-company-data-lake",
      "Condition": {
        "StringLike": {
          "s3:prefix": ["raw/finance/*", "raw/finance"]
        }
      }
    },
    {
      "Sid": "AllowGetObjectInsidePrefix",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-company-data-lake/raw/finance/*"
    }
  ]
}

2. Amazon DynamoDB Row-Level & Column-Level Security

For transactional or operational data stores in DynamoDB, fine-grained access control protects sensitive user data without duplicating tables:

• Row-Level Security (dynamodb:LeadingKeys): Restricts a user or application to querying/updating items where the Partition Key equals their specific identity (e.g., UserId).
• Column-Level Security (dynamodb:Attributes): Specifies exact attribute names that can be read or written, combined with dynamodb:Select to prevent retrieving forbidden fields.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DynamoDBRowAndColumnFGAC",
      "Effect": "Allow",
      "Action": ["dynamodb:GetItem", "dynamodb:Query"],
      "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/UserProfileStore",
      "Condition": {
        "ForAllValues:StringEquals": {
          "dynamodb:LeadingKeys": ["${aws:PrincipalTag/ApplicationId}"],
          "dynamodb:Attributes": ["UserId", "Email", "Preferences"]
        },
        "StringEqualsIfExists": {
          "dynamodb:Select": "SPECIFIC_ATTRIBUTES"
        }
      }
    }
  ]
}

---

IAM Roles for Data Pipelines & Cross-Account Access

Data processing services (AWS Glue, Amazon EMR, AWS Lambda, Amazon Redshift) do not use static IAM User credentials. Instead, they assume service Execution Roles via AWS Security Token Service (STS) to obtain temporary security credentials.

Cross-Account Access & The Confused Deputy Problem

When a third-party vendor or a separate production AWS account needs to access your data lake, cross-account IAM role assumption must be established using a Trust Policy (Resource Policy on the target Role).

To prevent the Confused Deputy Problem (where an attacker tricks a third-party service into accessing another customer's resources), require a third-party-generated, unique-per-customer sts:ExternalId in the trust policy. An external ID is an authorization discriminator, not a secret, so do not treat it as a password:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CrossAccountRoleTrustPolicy",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::987654321098:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "vendor-generated-customer-12345"
        }
      }
    }
  ]
}

Summary Checklist for DEA-C01 Exam

• Explicit Deny always wins regardless of where it is declared.
• ABAC uses tags (aws:PrincipalTag vs aws:ResourceTag) to scale permissions without policy sprawl.
• DynamoDB FGAC uses dynamodb:LeadingKeys (row) and dynamodb:Attributes (column).
• Cross-account access uses sts:AssumeRole + sts:ExternalId.

---

Amazon Redshift Database Users, Groups, and Roles

IAM and Redshift database authorization operate at different layers. IAM can authorize Redshift API calls, temporary database credentials, IAM Identity Center integration, or a service role that lets the warehouse call S3 for COPY and UNLOAD. After a session reaches a database, native Redshift privileges decide whether that identity can use a schema, read a table, execute a function, or create an object.

A database user is a login identity. A group is the older way to collect users. A database role is the preferred reusable collection of privileges and can be granted to users or other roles to form a hierarchy. Grant only the required schema and object privileges—for example, a reporting role commonly needs USAGE on a schema plus SELECT on specified tables. Avoid routine superuser grants, and configure default privileges deliberately if future objects should inherit access.

The layers are not interchangeable. Granting an IAM role permission to call Redshift APIs does not grant SELECT inside a database. Granting a database role SELECT does not let COPY read an S3 object unless the warehouse also assumes an IAM role with the required S3 and KMS permissions. Troubleshoot authorization by identifying which layer denied the request: identity federation or temporary credentials, database CONNECT and object grants, or the service role used for an external AWS call.