12.2 Automated Sensitive Data Discovery & PII Protection with Amazon Macie

Automated Sensitive Data Discovery & PII Protection with Amazon Macie

Data lakes accumulated over years of operation frequently suffer from "dark data"—unstructured or semi-structured datasets stored in Amazon S3 that contain uncataloged sensitive data, such as Personally Identifiable Information (PII), Protected Health Information (PHI), financial records, or secret access credentials. Compliance frameworks such as GDPR, CCPA, HIPAA, and PCI-DSS require organizations to inventory, monitor, and restrict access to sensitive data assets.

Amazon Macie is a fully managed data security and privacy service that uses machine learning and pattern matching to automatically discover, classify, and protect sensitive data in Amazon S3.

---

Core Data Discovery Mechanisms

Amazon Macie evaluates S3 object contents using two classification rule mechanisms: Managed Data Identifiers and Custom Data Identifiers.

Managed Data Identifiers

Managed Data Identifiers are built-in, AWS-maintained detection definitions designed to identify common sensitive data types across global jurisdictions:

• Personal Credentials: AWS secret access keys, private keys, API tokens, passwords.
• National Identification: U.S. Social Security Numbers (SSN), UK National Insurance Numbers, Canadian Social Insurance Numbers.
• Financial Data: Credit card numbers (Visa, Mastercard, AMEX), International Bank Account Numbers (IBAN), financial account credentials.
• Health Data: Medical record numbers (MRN), health insurance identifiers.

Custom Data Identifiers (CDIs)

When standard managed identifiers do not cover proprietary business formats (such as internal employee badges, custom customer IDs, or proprietary account numbers), data engineers configure Custom Data Identifiers (CDIs).

A CDI consists of:

1. Regular Expression (Regex): Defines the character sequence pattern (e.g., \bEMP-[0-9]{6}-[A-Z]{2}\b).
2. Keywords & Proximity: Defines exact words or phrases that must appear within a specified distance (1 to 300 characters) of the regex match to confirm intent and prevent false positives.
3. Ignore/Exclusion Patterns: Defines regex or exact text strings to exclude from matches (e.g., test sample IDs).

{
  "name": "Employee_ID_Custom_Identifier",
  "description": "Detects internal employee badge numbers in S3 data lake files",
  "regex": "\\bEMP-[0-9]{6}-[A-Z]{2}\\b",
  "keywords": ["employee", "staff_id", "badge_num", "payroll"],
  "maximumMatchDistance": 50,
  "ignoreWords": ["EMP-000000-TEST"]
}

---

Discovery Modes: Automated Discovery vs. Targeted Jobs

Amazon Macie provides two complementary operational modes for identifying sensitive data in Amazon S3.

Automated Sensitive Data Discovery

Automated sensitive data discovery continuously evaluates an organization's S3 data estate. Macie automatically analyzes bucket security posture, identifies public or unencrypted buckets, and samples objects across all S3 buckets to calculate a Sensitivity Score (High, Medium, Low, Not Sensitive) for every bucket.

• Cost Efficiency: Instead of scanning terabytes of unchanged log data continuously, automated discovery applies intelligent sampling algorithms to assess bucket sensitivity dynamically within a predictable budget.
• Heatmap Visibility: Provides an enterprise-wide inventory view of sensitive data distribution.

Targeted Sensitive Data Discovery Jobs

Targeted discovery jobs perform deep, comprehensive inspection across specified S3 buckets or object prefixes.

• Scope Definition: Configured using bucket criteria (tags, encryption type), object properties (file extension: .parquet, .csv, .json, .orc), file size, and last modified timestamp filters.
• Sampling Control: Jobs can be configured to scan 100% of matching objects for complete compliance audits or a specified percentage sample for periodic validation.
• Scheduling: Can run as one-time executions or on a recurring daily, weekly, or monthly schedule.

---

Finding Types, Sensitivity Scoring & Automated Remediation

When Macie detects policy risks or sensitive data, it generates Findings categorized into two types:

1. Policy Findings: Highlight security posture anomalies, such as an S3 bucket becoming publicly accessible, missing server-side encryption, or sharing access with an external AWS account.
2. Sensitive Data Findings: Detail specific sensitive data instances discovered inside S3 objects, including the bucket name, object key, location offset, and matched identifier types.

Event-Driven Automated Remediation Architecture

Macie findings are automatically published to Amazon EventBridge within minutes of generation. Data engineers configure EventBridge rules to trigger automated remediation workflows using AWS Lambda or AWS Systems Manager (SSM) Automation.

{
  "source": ["aws.macie"],
  "detail-type": ["Macie Finding"],
  "detail": {
    "type": ["SensitiveData:S3Object/Personal"],
    "severity": { "description": ["High"] }
  }
}

When a High-severity sensitive data finding is published, EventBridge triggers a Lambda remediation function:

import boto3
import json

def lambda_handler(event, context):
    finding = event['detail']
    bucket_name = finding['resourcesAffected']['s3Bucket']['name']
    object_key = finding['resourcesAffected']['s3Object']['key']
    
    s3 = boto3.client('s3')
    
    # Merge a classification tag; a tag restricts access only when a policy uses it.
    current_tags = s3.get_object_tagging(Bucket=bucket_name, Key=object_key)['TagSet']
    tags = {tag['Key']: tag['Value'] for tag in current_tags}
    tags['SensitivityClassification'] = 'RESTRICTED_PII'
    s3.put_object_tagging(
        Bucket=bucket_name,
        Key=object_key,
        Tagging={'TagSet': [{'Key': key, 'Value': value} for key, value in tags.items()]}
    )
    
    # Block public access configuration on the bucket if public
    s3.put_public_access_block(
        Bucket=bucket_name,
        PublicAccessBlockConfiguration={
            'BlockPublicAcls': True,
            'IgnorePublicAcls': True,
            'BlockPublicPolicy': True,
            'RestrictPublicBuckets': True
        }
    )
    print(f'Applied access controls and classification tag to {object_key} in {bucket_name}')

---

Multi-Account Governance with AWS Organizations

In enterprise environments managed by AWS Organizations, Amazon Macie operates in a delegated multi-account topology:

• Delegated Administrator Account: A central security or data governance account is designated as the Macie Delegated Administrator. This account manages Macie configuration, creates discovery jobs, defines Custom Data Identifiers, and inspects findings across all member accounts.
• Configured Auto-Enablement: Macie is Regional. A delegated administrator can enable the organization setting that automatically adds and enables new organization accounts in that Region; it is not an unconditional global behavior.
• Centralized Finding Export: Raw findings from all member accounts are aggregated and exported to a centralized, KMS-encrypted S3 bucket in the security account for long-term audit archiving.

---

Macie Operational Modes Comparison