12.2 Sensitive Data Masking in Telemetry & Messaging

12.2 Sensitive Data Masking in Telemetry & Messaging

Software applications routinely generate massive volumes of diagnostic logs, operational traces, and asynchronous event messages. During application debugging, error handling, or high-throughput transaction processing, developers and automated systems frequently log raw request payloads, stack traces, and database queries. Without strict guardrails, these telemetry streams inadvertently capture sensitive Personally Identifiable Information (PII)—such as Social Security numbers (SSNs), credit card primary account numbers (PANs), passport numbers, driver's license details, and cryptographic API tokens.

Exposing sensitive data within logging repositories and messaging brokers introduces severe compliance violations under regulatory standards such as PCI DSS Requirement 3 (protecting cardholder data), HIPAA (safeguarding electronic protected health information), and GDPR/CCPA (protecting consumer privacy). To neutralize these threats without burdening application developers with complex sanitization logic, AWS provides native, inline data protection capabilities: Amazon CloudWatch Logs Data Protection Policies and Amazon SNS Message Data Protection.

---

CloudWatch Logs Data Protection Architecture

Amazon CloudWatch Logs data protection policies audit and mask sensitive data in log groups. Detection happens as each event is ingested; masking is applied at every egress point (console, GetLogEvents, FilterLogEvents, Logs Insights, metric filters, and subscription filters). CloudWatch Logs keeps the original event, which is why a principal with logs:Unmask can still view it. You can attach one policy to a log group or create an account-level policy (PutAccountPolicy) that covers every existing and future log group in the account; when both exist, identifiers from both apply.

Loading diagram...

Operational Mechanics: Auditing vs. Masking

A CloudWatch Logs data protection policy supports two distinct operational modes within its policy statements:

1. Audit (Audit):
   • Scans incoming log events and detects sensitive data patterns without modifying the underlying log text.
   • Generates structured audit metrics and emits comprehensive finding logs to designated destinations (such as a separate security log group, an Amazon S3 compliance bucket, or an Amazon Data Firehose delivery stream).
   • Allows security teams to quantify PII exposure and establish baseline metrics prior to enforcing active redaction.
2. De-identify / Mask (Deidentify):
   • Replaces each detected value with asterisks whenever the data leaves CloudWatch Logs.
   • Example: A log line containing User SSN: 123-45-6789 is shown to a user without logs:Unmask with the whole SSN replaced by asterisks; the digits are not partially revealed.
   • Logs Insights queries, metric filters, and subscription filters also receive masked values, so downstream consumers process sanitized data. Masking applies only to events ingested after the policy exists.

---

Managed Data Identifiers vs. Custom Data Identifiers

The data protection policy engine detects sensitive data using two mechanisms:

1. Managed Data Identifiers

AWS provides pre-configured Managed Data Identifiers, which combine pattern matching and machine learning (some also require nearby keywords), that detect standard sensitive data formats across multiple geographic jurisdictions and compliance frameworks:

• Financial Identifiers: Credit card numbers (validated using the Luhn checksum algorithm to eliminate false positives), American Express, Diners Club, Discover, JCB, MasterCard, Visa, US bank routing numbers, International Bank Account Numbers (IBAN).
• Personal Identifiers (PII): US Social Security numbers (SSNs, validating format, valid area numbers, and range exclusions), US individual taxpayer identification numbers (ITIN), passport numbers (US, UK, Germany, Canada), driver's license numbers, postal addresses, email addresses, phone numbers.
• National Identifiers: UK National Insurance Numbers (NINO), Canadian Social Insurance Numbers (SIN), French INSEE, German National Identity Cards.
• Security Credentials: AWS Access Key IDs (AKIA...), AWS Secret Access Keys, SSH private keys, PGP private keys, JSON Web Tokens (JWT).

2. Custom Data Identifiers

Organizations frequently handle proprietary or domain-specific sensitive data that standard managed identifiers cannot classify—such as internal employee badge IDs, patient medical record numbers (MRNs), or custom loyalty card numbers. CloudWatch Logs supports Custom Data Identifiers:

• Name and Regex: Each custom identifier is a name (up to 128 characters) and a regular expression (up to 200 characters) matching the exact format of the proprietary data, for example EMP-[0-9]{6}-[A-Z]{2}. Omit ^ and $ anchors, because the value usually sits in the middle of a log line.
• Limits: Up to 10 custom identifiers per policy, defined in the policy's Configuration block and referenced by name. Unlike Amazon Macie, CloudWatch Logs custom identifiers have no keyword or proximity settings, so write specific patterns to limit false positives.

Data Protection Policy JSON Structure

The following policy audits findings to an S3 security bucket and masks the same identifiers, including one custom identifier. The Audit and Deidentify statements must list exactly the same data identifiers:

{
  "Name": "payment-service-data-protection",
  "Description": "Audit and mask card numbers, SSNs, secret keys, and employee IDs",
  "Version": "2021-06-01",
  "Configuration": {
    "CustomDataIdentifier": [
      { "Name": "EmployeeId", "Regex": "EMP-[0-9]{6}-[A-Z]{2}" }
    ]
  },
  "Statement": [
    {
      "Sid": "AuditSensitiveTelemetry",
      "DataIdentifier": [
        "arn:aws:dataprotection::aws:data-identifier/CreditCardNumber",
        "arn:aws:dataprotection::aws:data-identifier/Ssn-US",
        "arn:aws:dataprotection::aws:data-identifier/AwsSecretKey",
        "EmployeeId"
      ],
      "Operation": {
        "Audit": {
          "FindingsDestination": {
            "S3": { "Bucket": "centralized-security-compliance-audit-logs" }
          }
        }
      }
    },
    {
      "Sid": "DeidentifySensitiveTelemetry",
      "DataIdentifier": [
        "arn:aws:dataprotection::aws:data-identifier/CreditCardNumber",
        "arn:aws:dataprotection::aws:data-identifier/Ssn-US",
        "arn:aws:dataprotection::aws:data-identifier/AwsSecretKey",
        "EmployeeId"
      ],
      "Operation": {
        "Deidentify": { "MaskConfig": {} }
      }
    }
  ]
}

---

De-anonymization & Unmask Permissions: logs:Unmask

A central design principle of CloudWatch Logs data protection is least-privilege visibility. When data protection is enabled on a log group:

• Standard users, developers, and security analysts possessing standard permissions (logs:FilterLogEvents, logs:GetLogEvents, logs:StartQuery) only see masked data.
• The CloudWatch Logs console displays a prominent banner indicating that data protection is active and that sensitive terms have been redacted.

The logs:Unmask IAM Permission

In authorized operational scenarios—such as a fraud investigation, security incident triage, or regulatory audit—a privileged engineer may require access to the raw, unmasked data. Viewing unmasked data requires the explicit IAM permission logs:Unmask.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowPrivilegedUnmaskWithMFA",
      "Effect": "Allow",
      "Action": "logs:Unmask",
      "Resource": "arn:aws:logs:us-east-1:123456789012:log-group:/aws/production/payment-service:*",
      "Condition": {
        "Bool": {
          "aws:MultiFactorAuthPresent": "true"
        }
      }
    }
  ]
}

Operational Unmasking Mechanics

• Console Experience: A user with logs:Unmask can click an "Unmask" toggle button within the CloudWatch Logs console. The console prompts the user to confirm temporary unmasking. Once confirmed, the raw data appears in the current session.
• API / CLI Experience: GetLogEvents, FilterLogEvents, and GetLogRecord accept an unmask parameter (--unmask in the CLI). The call succeeds with original values only if the caller has logs:Unmask; Logs Insights queries use the unmask(@message) function with the same permission.
• CloudTrail Auditability: Every invocation of logs:Unmask is explicitly logged in AWS CloudTrail. The CloudTrail record captures the exact IAM principal, timestamp, source IP address, target log group, and session context. This non-repudiation mechanism ensures that any attempt to view unmasked PII generates an immutable audit trail for security compliance.

---

Amazon SNS Message Data Protection

While CloudWatch Logs protects telemetry, Amazon Simple Notification Service (SNS) provides pub/sub messaging across distributed microservices, serverless architectures, and third-party webhook endpoints. When an upstream payment service publishes transaction notifications, message payloads may contain credit card numbers or customer addresses.

Amazon SNS Message Data Protection scans message payloads in real time as they are published to an SNS topic, evaluating the payload against a topic Data Protection Policy.

Loading diagram...

Enforcement Actions: Block vs. De-identify

An SNS data protection policy can execute two distinct operational actions upon detecting sensitive data:

1. Block (Deny):
   • Publisher Rejection: With DataDirection: Inbound, the publish request is rejected and the publisher receives an AuthorizationError. With DataDirection: Outbound, delivery to the matching subscriptions is blocked instead.
   • Zero Ingestion: The message is dropped completely and never delivered to any topic subscribers. This provides an ironclad boundary against sensitive data ingress.
2. De-identify / Redact (Deidentify):
   • Inline Payload Transformation: The message is accepted by the SNS topic, but the sensitive data strings are automatically masked or redacted before the message is fanned out to downstream subscribers.
   • Subscriber Protection: Downstream subscribers (such as Amazon SQS queues, AWS Lambda functions, Amazon Kinesis streams, HTTPS webhooks, or email recipients) receive only sanitized payloads.

SNS Data Protection Policy Example

The following policy configures an Amazon SNS topic to block any message containing Social Security numbers or AWS secret keys from being published, while auditing detections to Amazon CloudWatch Logs:

{
  "Name": "payment-topic-data-protection",
  "Description": "Block SSNs and AWS secret keys; audit card numbers",
  "Version": "2021-06-01",
  "Statement": [
    {
      "Sid": "BlockCredentialsAndSSN",
      "DataDirection": "Inbound",
      "Principal": ["*"],
      "DataIdentifier": [
        "arn:aws:dataprotection::aws:data-identifier/Ssn-US",
        "arn:aws:dataprotection::aws:data-identifier/AwsSecretKey"
      ],
      "Operation": { "Deny": {} }
    },
    {
      "Sid": "AuditFinancialData",
      "DataDirection": "Inbound",
      "Principal": ["*"],
      "DataIdentifier": [
        "arn:aws:dataprotection::aws:data-identifier/CreditCardNumber"
      ],
      "Operation": {
        "Audit": {
          "SampleRate": "99",
          "FindingsDestination": {
            "CloudWatchLogs": { "LogGroup": "/aws/vendedlogs/sns/payment-topic-audit" }
          }
        }
      }
    }
  ]
}

---

Comparison: CloudWatch Logs vs. Amazon SNS Data Protection

---

Specialty Exam Pitfalls & Architectural Traps

1. The Retrospective Masking Fallacy: Believing that enabling a CloudWatch Logs data protection policy will retroactively mask pre-existing log events in the log group. Sensitive data is detected as events are ingested, so events ingested before the policy was set are never masked. To reduce exposure in historical logs, restrict read access to the log group, shorten retention, or delete old log streams after exporting what compliance requires.
2. The AdministratorAccess Unmask Assumption: Assuming that an IAM principal with the AWS managed policy AdministratorAccess can view unmasked logs by default. While AdministratorAccess grants * on *, if an explicit Deny or Permissions Boundary restricts logs:Unmask, or if an analyst views the log group without initiating an unmask session, the data remains masked. Always verify whether an analyst possesses explicit, positive authorization for logs:Unmask.
3. SNS Block Breaking Critical Pipelines: Configuring an SNS message data protection policy with a Deny (Block) action on high-priority operational topics. If an application encounters an unexpected error and includes an API key or email address in a critical alert message, the SNS topic will reject the message entirely, dropping the alert and blinding operations teams to an active outage. Use Deidentify rather than Deny when message delivery must be maintained.
4. Overly Broad Custom Regex: Defining custom data identifier patterns that are too generic (CloudWatch Logs custom identifiers have no keyword or proximity option to narrow them). Unanchored regular expressions frequently trigger massive false positive rates across standard diagnostic telemetry, resulting in unintended masking of harmless hexadecimal identifiers, hash values, and system UUIDs.