11.2 Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies

Data Encryption at Rest & in Transit with AWS KMS & S3 Bucket Policies

Data encryption is a fundamental requirement for regulatory compliance (HIPAA, GDPR, PCI-DSS, SOC 2) and corporate data governance. In AWS data engineering pipelines, data must be protected both at rest (stored on disk in S3, Redshift, EBS, DynamoDB, EMR) and in transit (moving over the network between services). AWS Key Management Service (KMS) serves as the central security service for cryptographic key management.

---

AWS KMS Key Architecture & Envelope Encryption

AWS KMS manages symmetric (AES-256-GCM) and asymmetric cryptographic keys. KMS keys never leave AWS KMS unencrypted security modules (HSMs).

KMS Key Types

1. AWS Owned Keys: Created and managed by AWS services for internal service use. Free of charge, but cannot be viewed, audited, or managed directly by customers.
2. AWS Managed Keys: Created automatically by AWS on your behalf (e.g., aws/s3, aws/redshift, aws/glue). Free key storage, but API calls incur standard KMS charges. Automatically rotated every year; customers cannot change that schedule.
3. Customer managed KMS keys: Created, owned, and managed by you. Support custom key policies, explicit cross-account sharing, on-demand rotation, configurable automatic rotation periods for eligible symmetric keys, and deletion scheduling. Pricing varies by Region and key type.
4. Custom and external key stores: AWS CloudHSM custom key stores use customer-controlled HSMs in AWS, while external key stores connect KMS to supported key material outside AWS.

The Envelope Encryption Mechanics

Encrypting multi-gigabyte or terabyte dataset files directly via the KMS API (kms:Encrypt) is impossible because KMS APIs have a strict payload limit of 4 KB. Instead, AWS uses Envelope Encryption:

[ KMS key ] 
       |
       v  (kms:GenerateDataKey)
[ Plaintext Data Key (DEK) ]  <--->  Encrypts Large File Data locally
       |
       +--> Encrypted by KMS --> [ Encrypted DEK ] (Stored in file metadata)

The Encryption Process:

1. The application/service calls KMS API kms:GenerateDataKey supplying the KMS key ARN.
2. KMS returns two components: a Plaintext Data Encryption Key (DEK) and an Encrypted DEK (encrypted by the KMS key).
3. The application uses the Plaintext DEK in memory to encrypt the dataset locally (using AES-256).
4. The application erases the Plaintext DEK from memory.
5. The application saves the ciphertext file along with the Encrypted DEK stored in the file header or object metadata.

The Decryption Process:

1. The application reads the file and extracts the Encrypted DEK from metadata.
2. The application passes the Encrypted DEK to KMS via kms:Decrypt.
3. KMS decrypts the DEK using the KMS key and returns the Plaintext DEK to the application.
4. The application decrypts the dataset in memory and wipes the Plaintext DEK.

---

Amazon S3 Encryption Options & Bucket Keys

Amazon S3 supports multiple server-side and client-side encryption options:

Optimizing Costs & Performance: Amazon S3 Bucket Keys

When running massive EMR, Glue, or Athena queries that read/write millions of S3 objects encrypted with SSE-KMS, default SSE-KMS behavior executes a kms:GenerateDataKey or kms:Decrypt call for every single S3 object. This rapidly leads to:

• KMS request cost: per-request rates vary by Region and key type.
• KMS throttling: exceeding the applicable request quota can cause ThrottlingException; monitor Service Quotas and request metrics for the selected Region.

Solution: S3 Bucket Keys (s3:BucketKeyEnabled = true) When S3 Bucket Keys are enabled on a bucket, S3 requests a short-lived bucket-level Data Key from KMS. S3 uses this key to derive DEKs for new objects written to the bucket. This can reduce KMS API calls by up to 99%, cutting request costs and substantially reducing KMS throttling pressure. It does not remove every KMS call or guarantee that throttling is impossible, so retain retries and quota monitoring.

---

KMS Key Policies vs IAM Policies

Unlike most AWS services where an IAM policy is sufficient to grant access, KMS operates under a dual-authorization model:

CRITICAL EXAM PRINCIPLE: A customer managed KMS key cannot be used unless its KMS Key Policy explicitly allows the principal or explicitly delegates access control to the account's IAM policies!

To allow identity-based IAM policies to manage access to a KMS key, the Key Policy MUST contain the default root delegation statement:

{
  "Sid": "EnableIAMUserPermissions",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::123456789012:root"
  },
  "Action": "kms:*",
  "Resource": "*"
}

---

Enforcing Encryption via S3 Bucket Policies

To guarantee regulatory compliance, data engineers must enforce two common bucket-policy controls when policy requires them:

1. Enforce Encryption in Transit (HTTPS / TLS): Block unencrypted HTTP connections (aws:SecureTransport: false).
2. Enforce Specific KMS Encryption Key: Reject uploads (s3:PutObject) that do not use SSE-KMS with an approved CMK.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EnforceTLSInTransit",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::finance-data-lake-prod",
        "arn:aws:s3:::finance-data-lake-prod/*"
      ],
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      }
    },
    {
      "Sid": "EnforceApprovedKMSEncryption",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::finance-data-lake-prod/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption-aws-kms-key-id": "arn:aws:kms:us-east-1:123456789012:key/abc-123-def-456"
        }
      }
    }
  ]
}

Summary Checklist for DEA-C01 Exam

• KMS Payload Limit: 4 KB (requires Envelope Encryption for data files).
• S3 Bucket Keys: Reduces KMS API requests by ~99% for SSE-KMS data lakes.
• Enforce TLS: Set Effect: Deny when aws:SecureTransport is false.
• Cross-Account KMS: Must update BOTH the KMS Key Policy in the key account and the IAM Policy in the calling account.