1.4 Amazon Macie & Automated Data Classification

1.4 Amazon Macie & Automated Data Classification

Cloud data storage scalability makes Amazon S3 the primary repository for enterprise data lakes, application backups, and business analytics. However, decentralized storage frequently leads to shadow data, unencrypted personal information, exposed credentials, and accidental public disclosure. Amazon Macie is a specialized data security and privacy service that applies machine learning, pattern matching, and rule-based detectors to discover, classify, and protect sensitive data stored in Amazon S3.

---

Macie Architecture & Operational Objectives

Macie operates across two distinct operational domains within Amazon S3:

1. Bucket-Level Posture Evaluation: Continuously assesses security controls across all S3 buckets in an account or AWS Organization, evaluating public read/write access, default encryption settings, and cross-account or external replication permissions.
2. Object-Level Sensitive Data Classification: Inspects individual object contents to identify personally identifiable information (PII), payment card data, national identification numbers, healthcare records, and secret credentials.

Loading diagram...

Multi-Account Administration

In enterprise environments, Macie integrates with AWS Organizations. An AWS Organizations management account designates a dedicated security account as the Macie Delegated Administrator. The delegated administrator can:

• Enable and configure Macie across all existing and newly created member accounts.
• View aggregated bucket security posture and sensitive data heatmaps across the entire multi-account inventory.
• Create and execute targeted sensitive data discovery jobs across member accounts.
• Centrally deploy Custom Data Identifiers (CDIs) and suppression rules.

---

Continuous Automated Discovery vs Targeted Classification Jobs

Macie provides two operational models for content inspection, each tailored to distinct operational and budgetary requirements.

Automated Sensitive Data Discovery

When automated sensitive data discovery is activated, Macie continuously analyzes your S3 bucket inventory. It uses intelligent sampling algorithms to select small batches of objects across all buckets, inspecting them for sensitive data without scanning petabytes of redundant data. Macie calculates a sensitivity score for each bucket (1–100, with higher values meaning more sensitive data, or −1 when Macie can't analyze the objects) and assigns a matching sensitivity label. This highlights unknown shadow buckets containing unencrypted sensitive records.

Targeted Sensitive Data Classification Jobs

When legal or regulatory compliance mandates exhaustive proof that an S3 bucket is free of sensitive data (e.g., preparing a dataset for third-party machine learning training), security teams execute Targeted Classification Jobs:

• Execution Frequency: Can be configured as a One-Time Job or a Scheduled Recurring Job (daily, weekly, or monthly) that evaluates only newly added or modified objects.
• Scope Filtering: Allows granular inclusion/exclusion criteria based on S3 bucket name, object key prefix (folder path), object tag, file extension (e.g., .csv, .parquet, .json, .pdf), file size (e.g., maximum 50 MB), and last modified timestamp.
• Storage Class Support: Scans objects in S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, S3 One Zone-IA, and S3 Glacier Instant Retrieval. (Deep archive storage classes like Glacier Flexible and Deep Archive must be restored before Macie can inspect them).

---

Managed Data Identifiers vs Custom Data Identifiers

Macie inspects files using two classes of pattern recognition algorithms:

Managed Data Identifiers (MDIs)

Managed Data Identifiers are built-in detectors created and maintained by AWS. They incorporate regular expressions, checksum validation algorithms, and natural language context rules to identify standard sensitive data types:

• Credentials: AWS secret access keys, private encryption keys (RSA/DSA/EC), GitHub personal access tokens, OpenSSH private keys, HTTP basic auth headers.
• Financial Identifiers: Credit card numbers (validated against the Luhn checksum algorithm), international bank account numbers (IBAN), American Bankers Association (ABA) routing transit numbers.
• Personal Identifiers (PII): US Social Security Numbers (SSN), US driver's license numbers, passport numbers (US, UK, Canada, Germany, etc.), national identity numbers.
• Healthcare Data: Health Insurance Claim Numbers (HICN), National Provider Identifiers (NPI).

Custom Data Identifiers (CDIs)

Many enterprises handle proprietary data that standard detectors cannot recognize, such as employee serial numbers, customer account formats, internal project codenames, or medical record numbers. Security engineers create Custom Data Identifiers (CDIs) to define proprietary detection rules.

A Custom Data Identifier combines one required pattern with optional refinements:

1. Regular Expression (Regex): Defines the primary pattern match (up to 512 characters) (e.g., \b[A-Z]{3}-\d{6}-[A-Z]\b).
2. Keywords (Proximity Matching): Up to 50 keywords (3–90 characters each) that must precede the matched text within a maximum match distance of 1–300 characters (default 50). This dramatically reduces false positives. For example, if searching for an 8-digit customer ID, requiring proximity keywords like account, customer, cid, or client ensures random 8-digit transaction numbers are ignored.
3. Ignore Words: Up to 10 ignore words (4–90 characters each, case sensitive) that exclude a match (e.g., excluding test strings like EXAMPLE-ACCOUNT-00000000).
4. Occurrence-Based Severity: Optional thresholds that assign Low, Medium, or High severity according to how many matches an object contains. Without them, findings from a CDI are Medium.

---

Macie Finding Taxonomy: Policy vs Sensitive Data Findings

Macie generates two distinct categories of findings, both formatted in ASFF and published to EventBridge and Security Hub:

1. Policy Findings (Policy:IAMUser/...)

Policy findings alert on S3 bucket-level permission and configuration vulnerabilities that expose data to risk:

• Policy:IAMUser/S3BucketPublic: An S3 bucket permissions policy or access control list (ACL) allows public read or write access.
• Policy:IAMUser/S3BucketSharedExternally: An S3 bucket is shared with an external AWS account outside your AWS Organization.
• Policy:IAMUser/S3BucketReplicatedExternally: Bucket replication rules replicate data to a bucket in an external, unverified AWS account.
• Policy:IAMUser/S3BucketEncryptionDisabled: An S3 bucket does not enforce default server-side encryption (SSE).

2. Sensitive Data Findings (SensitiveData:S3Object/...)

Sensitive data findings indicate that a specific S3 object contains sensitive information:

• SensitiveData:S3Object/Personal: Detected PII (names, SSNs, passports).
• SensitiveData:S3Object/Financial: Detected credit cards, bank accounts, or financial transactions.
• SensitiveData:S3Object/Credentials: Detected private keys, API tokens, or passwords.
• SensitiveData:S3Object/CustomIdentifier: Detected matches against an active Custom Data Identifier.

Sensitive Data Sample Retrieval

By default, to maintain strict data privacy, Macie finding payloads include metadata (object ARN, bucket name, count of matches, identifier type) but do not display the raw sensitive data strings (e.g., the actual credit card digits). To allow security analysts to inspect the exact text that triggered the finding during investigation, an administrator can configure Sensitive Data Sample Retrieval:

• Requires a configured repository (an S3 bucket) for sensitive data discovery results, because Macie uses those results to locate each occurrence.
• Macie extracts the first 1–10 occurrences reported by the finding, up to 128 characters each, and encrypts them with an AWS KMS key that you specify.
• Only principals allowed to call the reveal operation and use that key can see the samples, so restrict both with IAM policies and the key policy.

---

Specialty Exam Pitfalls & Data Protection Scenarios

1. KMS Key Policies and Macie Permissions: If an S3 bucket contains objects encrypted with AWS Key Management Service (SSE-KMS) using a customer-managed key (CMK), Macie cannot inspect those objects by default. The classification job will fail with an access denied error. To resolve this, you must update the KMS key policy to grant kms:Decrypt permissions to the Macie service-linked role (arn:aws:iam::<account>:role/aws-service-role/macie.amazonaws.com/AWSServiceRoleForAmazonMacie).
2. Unsupported Object Types: Macie cannot inspect encrypted files that have been client-side encrypted prior to upload if Macie does not possess the decryption key. Furthermore, Macie inspects common file formats (plain text, CSV, JSON, XML, Parquet, Avro, PDF, DOCX, XLSX, TAR, GZ, ZIP) but cannot execute binary database engines or inspect proprietary binary compiled files.
3. Policy Finding vs Sensitive Data Finding Remediation: Remediating a Policy finding (e.g., Policy:IAMUser/S3BucketPublic) requires modifying bucket-level access controls, such as enabling S3 Block Public Access via the PutPublicAccessBlock API. Conversely, remediating a Sensitive Data finding requires object-level lifecycle actions: moving the object to an isolated quarantine bucket, applying stricter object ACLs, re-encrypting with a restricted KMS key, or triggering object deletion.
4. Sampling Is Not Proof of Absence: Automated sensitive data discovery can use custom data identifiers and allow lists, so it can watch an organization's buckets for a proprietary employee ID format. It still samples objects, however. When an auditor needs proof that every object in scope was inspected, run a targeted classification job instead.