12.3 Configuration Auditing & Compliance Enforcement with AWS Config

Configuration Auditing & Compliance Enforcement with AWS Config

In enterprise data engineering environments, data compliance requires continuous auditing of resource configurations. Static periodic audits or manual spot-checks fail to prevent security drift—such as an engineer temporarily exposing an Amazon S3 data lake bucket publicly or disabling encryption on an Amazon Redshift cluster.

AWS Config is a continuous configuration auditing service that records AWS resource state changes, evaluates configurations against compliance baselines, and executes automated remediations when non-compliant states are detected.

---

Core Architecture: Configuration Items & History

AWS Config operates by recording resource configuration state transitions as structured JSON documents called Configuration Items (CIs).

Anatomy of a Configuration Item (CI)

A CI is generated whenever a supported resource (e.g., S3 Bucket, IAM Role, KMS Key, Redshift Cluster, Glue Job) is created, modified, or deleted. The CI captures:

• Metadata: Resource ID, ARN, AWS region, account ID, resource creation timestamp.
• Attributes: Direct configuration parameters (e.g., S3 bucket encryption configuration, public access block settings, bucket policy).
• Relationships: Associated AWS resources (e.g., S3 bucket attached to a specific KMS Key ARN).

CIs are delivered continuously to an Amazon S3 bucket (Configuration History) and streamed to an Amazon SNS topic for real-time processing.

---

AWS Config Rules for Data Engineering & Storage

AWS Config evaluates resource configurations against Config Rules. Rules evaluate resources either upon configuration change or on a periodic schedule (e.g., every 24 hours).

Choose the trigger from the control's semantics. A configuration-change rule is appropriate when a recorded property changes, while a periodic rule can reassess conditions that are not tied to one resource update. A rule reports compliance; it does not block the original API call. Automatic remediation is a later, asynchronous action and can fail because of stale compliance results, missing permissions, concurrent changes, or a runbook error. Use preventive controls such as SCPs or resource policies when the requirement is to reject an action before it succeeds, and use Config to detect, inventory, and remediate drift. Each governed account and Region also needs the required Config recorder and delivery configuration for the resource types being evaluated.

Essential AWS Managed Rules for Data Platforms

• s3-bucket-server-side-encryption-enabled: Verifies that Amazon S3 buckets enforce default Server-Side Encryption (SSE-S3 or SSE-KMS).
• s3-bucket-ssl-requests-only: Ensures S3 bucket policies deny HTTP requests and enforce TLS (aws:SecureTransport).
• s3-bucket-public-read-prohibited: Evaluates whether S3 buckets permit public read access.
• redshift-cluster-configuration-check: Checks whether Amazon Redshift clusters enforce encryption at rest, automated snapshots, and non-public accessibility.
• kms-key-rotation-enabled: Confirms that automatic rotation is enabled for eligible AWS KMS customer managed keys. The rotation period is configurable, so verify the configured interval against policy instead of assuming one year.

Custom Config Rules

When compliance requirements exceed pre-built managed rules, organizations deploy Custom Config Rules using AWS Lambda or CloudFormation Guard DSL rules. Custom rules analyze incoming CIs and return a compliance state of COMPLIANT, NON_COMPLIANT, or NOT_APPLICABLE back to AWS Config.

---

Declarative Compliance via Conformance Packs

A Conformance Pack is a collection of AWS Config rules and associated remediation actions packaged into a single declarative YAML template. Conformance packs enable security teams to deploy standardized compliance frameworks across an organization.

AWSTemplateFormatVersion: '2010-09-09'
Description: 'Data Lake Storage Security Conformance Pack'
Resources:
  S3BucketEncryptionRule:
    Type: 'AWS::Config::ConfigRule'
    Properties:
      ConfigRuleName: 's3-bucket-server-side-encryption-enabled'
      Source:
        Owner: 'AWS'
        SourceIdentifier: 'S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED'

  S3SSLRequestsOnlyRule:
    Type: 'AWS::Config::ConfigRule'
    Properties:
      ConfigRuleName: 's3-bucket-ssl-requests-only'
      Source:
        Owner: 'AWS'
        SourceIdentifier: 'S3_BUCKET_SSL_REQUESTS_ONLY'

  S3EncryptionRemediation:
    Type: 'AWS::Config::RemediationConfiguration'
    Properties:
      ConfigRuleName: !Ref S3BucketEncryptionRule
      TargetId: 'AWS-EnableS3BucketEncryption'
      TargetType: 'SSM_DOCUMENT'
      Automatic: true
      MaximumAutomaticAttempts: 3
      RetryAttemptSeconds: 60
      Parameters:
        BucketName:
          ResourceValue:
            Value: 'RESOURCE_ID'
        SSEAlgorithm:
          StaticValue:
            Values:
              - 'aws:kms'

Organization Conformance Packs

Deploying Conformance Packs at the AWS Organizations management account level ensures that compliance baselines are automatically enforced across all current and future member accounts. Member account administrators cannot modify or delete Organization Conformance Packs, securing the governance boundary.

---

Automated Remediation with SSM Automation

Identifying non-compliant resources is insufficient without automated resolution. AWS Config integrates directly with AWS Systems Manager (SSM) Automation runbooks to fix compliance drift automatically.

Remediation Execution Flow

1. Evaluation: AWS Config rule (e.g., s3-bucket-server-side-encryption-enabled) evaluates a newly created S3 bucket and flags it as NON_COMPLIANT.
2. Target Invocation: The remediation configuration triggers the target SSM Automation runbook (AWS-EnableS3BucketEncryption).
3. Execution: Systems Manager executes the runbook using an IAM remediation role, applying SSE-KMS encryption to the target S3 bucket.
4. Re-Evaluation: AWS Config re-evaluates the bucket configuration and updates the resource status to COMPLIANT.

---

Centralized Multi-Account Governance & Advanced Queries

In multi-account AWS Organizations, compliance tracking requires aggregating configuration states across regions and accounts using an AWS Config Aggregator.

Advanced SQL Querying

AWS Config includes an Advanced Query engine that allows data architects to execute SQL queries across multi-account aggregators to inventory non-compliant resources instantaneously.

SELECT
  accountId,
  awsRegion,
  resourceId,
  configuration.targetResourceType,
  configuration.targetResourceId,
  configuration.complianceType
WHERE
  resourceType = 'AWS::Config::ResourceCompliance'
  AND configuration.targetResourceType = 'AWS::S3::Bucket'
  AND configuration.complianceType = 'NON_COMPLIANT'

---

AWS Governance & Security Triad Comparison