12.1 AWS Secrets Manager & Systems Manager Parameter Store

12.1 AWS Secrets Manager & Systems Manager Parameter Store

Hardcoded credentials, static API tokens, and unencrypted configuration files embedded within source code or container images represent one of the most critical security vulnerabilities in modern cloud infrastructure. A compromised database password or third-party service credential can grant unauthorized actors immediate access to production data stores. To enforce least privilege, prevent credential leakage, and satisfy regulatory mandates like PCI DSS and SOC 2, organizations must decouple sensitive secrets and configuration variables from application code.

AWS delivers two core services for managing configuration parameters and confidential credentials: AWS Secrets Manager and AWS Systems Manager Parameter Store. While both services integrate natively with AWS Key Management Service (AWS KMS) for envelope encryption and AWS Identity and Access Management (IAM) for granular authorization, their operational architectures, rotation lifecycles, cross-account mechanics, and pricing models differ substantially.

---

AWS Secrets Manager Architecture & Secrets Lifecycle

AWS Secrets Manager is a purpose-built secrets management service designed to encrypt, store, retrieve, and automatically rotate credentials throughout their operational lifecycle. A secret in Secrets Manager can store arbitrary plaintext strings, structured JSON key-value pairs (commonly database host, port, username, and password), or binary data up to 64 KB in size.

Secret Versioning & Staging Labels

Secrets Manager uses an immutable versioning model. When a secret value is updated or rotated, Secrets Manager does not overwrite the existing value. Instead, it generates a new Version ID (a UUID) containing the updated secret string or binary payload. Secrets Manager organizes versions using Staging Labels (version stages):

• AWSCURRENT: The active, production-ready version of the secret that client applications consume by default when requesting a secret value without specifying a version ID.
• AWSPREVIOUS: The immediately preceding version of the secret. If an application encounters connection issues during a rotation event or an emergency rollback is required, the application can explicitly request AWSPREVIOUS.
• AWSPENDING: The staging label assigned to a newly generated secret version while the automated rotation workflow validates and updates the target resource.
• Custom Staging Labels: User-defined labels (e.g., STAGING, BLUE, GREEN) used in custom deployment pipelines or phased rollouts.

Important Versioning Rule: A single version can have multiple staging labels attached, but a specific staging label (such as AWSCURRENT) can point to only ONE version at any given time within a secret.

---

Automated Credential Rotation Architecture

One of Secrets Manager's most powerful enterprise security capabilities is automated, scheduled credential rotation without administrative intervention. Secrets Manager coordinates rotation by invoking an AWS Lambda rotation function using a deterministic 4-step rotation lifecycle.

Loading diagram...

The 4-Step Rotation Lifecycle

When Secrets Manager initiates rotation (either via a scheduled cron interval or on-demand via the RotateSecret API), it invokes the configured Lambda function four sequential times, passing the SecretId, ClientRequestToken (which becomes the new VersionId), and the operational Step:

1. createSecret:
   • The Lambda function checks whether a version with the ClientRequestToken already exists in Secrets Manager.
   • If not, the function generates a cryptographically secure random password and stores it in Secrets Manager by calling PutSecretValue with the staging label AWSPENDING. If createSecret fails and retries, idempotency ensures it does not overwrite an existing pending version.
2. setSecret:
   • The Lambda function connects to the target service or database and updates the user's password to match the AWSPENDING credential.
   • In single-user rotation, it connects using the current credentials and alters its own password. In multi-user rotation, it connects using an administrative superuser secret to alter the target application user's password.
3. testSecret:
   • The Lambda function attempts to establish a real connection and execute a test query (e.g., SELECT 1) against the target database using the AWSPENDING credentials.
   • This verification step confirms that the database accepted the password change and that network connectivity, authentication plugins, and user permissions remain intact.
4. finishSecret:
   • The Lambda function calls UpdateSecretVersionStage to move the AWSCURRENT staging label from the old version to the newly verified AWSPENDING version.
   • Secrets Manager automatically assigns AWSPREVIOUS to the older version and removes AWSPENDING. The rotation is complete.

Single-User vs. Multi-User (Alternating) Rotation Strategies

Choosing the proper rotation strategy is critical for preventing application downtime during password changes:

Native Database Rotation

Secrets Manager provides native, pre-configured Lambda rotation templates for Amazon RDS (MySQL, PostgreSQL, MariaDB, Oracle, SQL Server), Amazon Aurora, Amazon DocumentDB, and Amazon Redshift. When configured via the AWS Management Console or AWS CloudFormation, Secrets Manager automatically creates the Lambda function inside the target database's private VPC subnets and configures required security group rules.

Client-Side Caching SDKs

Calling GetSecretValue on every microservice request or web transaction introduces substantial API latency, exhausts Secrets Manager service quotas, and generates unnecessary API billing costs. AWS provides open-source Secrets Manager caching clients (for example Java, Python, .NET, Go, and Rust) and the Secrets Manager Agent, a local HTTP caching service for any language:

• Secrets are retrieved once and cached in application memory; the caching clients refresh each secret after a configurable interval (default 1 hour).
• The Secrets Manager JDBC driver wraps database connections: if authentication fails because the password was rotated, it refreshes the secret and retries the connection. Plain caching clients don't retry on their own, so applications should refresh the cached secret when a login fails.

---

Cross-Account Secret Access & KMS Cryptographic Requirements

In multi-account enterprise landing zones, centralized shared services or database accounts frequently host secrets that must be accessed by application workloads running in spoke member accounts. Establishing cross-account access requires precise coordination of IAM policies and KMS key policies.

Loading diagram...

The AWS-Managed KMS Key Limitation

Critical Exam Rule: Cross-account access to a secret encrypted with the default AWS-managed KMS key (aws/secretsmanager) is impossible. The key policy of an AWS-managed key is maintained strictly by AWS and cannot be edited to add external AWS account principals. To share a secret across accounts, you must encrypt the secret using a Customer Managed Key (CMK).

The Three Policy Checkpoints

For a workload in Account A (111122223333) to read a secret in Account B (444455556666), three synchronized policies must explicitly permit the transaction:

1. IAM Identity Policy (Account A): Attached to the calling IAM role, granting permissions to read the external secret and decrypt using the external KMS key:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "AllowReadExternalSecret",
         "Effect": "Allow",
         "Action": "secretsmanager:GetSecretValue",
         "Resource": "arn:aws:secretsmanager:us-east-1:444455556666:secret:ProdDBSecret-a1b2c3"
       },
       {
         "Sid": "AllowDecryptWithExternalKMS",
         "Effect": "Allow",
         "Action": "kms:Decrypt",
         "Resource": "arn:aws:kms:us-east-1:444455556666:key/12345678-1234-1234-1234-123456789012"
       }
     ]
   }
   

2. Secret Resource-Based Policy (Account B): Attached directly to the secret in Secrets Manager, granting access to the principal in Account A:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "AllowCrossAccountAccess",
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::111122223333:role/MicroserviceRole"
         },
         "Action": "secretsmanager:GetSecretValue",
         "Resource": "*"
       }
     ]
   }
   

3. KMS Key Policy (Account B): Attached to the Customer Managed Key encrypting the secret, permitting the external IAM role to decrypt:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "AllowExternalRoleToDecrypt",
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::111122223333:role/MicroserviceRole"
         },
         "Action": [
           "kms:Decrypt",
           "kms:DescribeKey"
         ],
         "Resource": "*"
       }
     ]
   }
   

---

Multi-Region Secret Replication

For globally distributed applications and multi-region disaster recovery (DR) architectures, Secrets Manager provides Multi-Region Secret Replication:

• Primary vs. Replica Secrets: A secret is designated as the primary secret in a source region (e.g., us-east-1). Administrators configure replication to one or more destination regions (e.g., us-west-2, eu-west-1).
• Regional KMS Encryption: Replica secrets are encrypted using a KMS key in the replica's destination region. This can be an AWS-managed key for Secrets Manager in that region or a destination-region Customer Managed Key. Secrets Manager also supports AWS KMS Multi-Region Keys.
• Automated Synchronization: Whenever the primary secret's value, staging labels, tags, or metadata are modified, Secrets Manager automatically replicates the changes to all replica regions asynchronously within seconds.
• Read-Only Replicas: Replicas are read-only. Applications in destination regions execute GetSecretValue against their local regional endpoint, achieving single-digit millisecond latency and avoiding cross-region network dependencies.
• Disaster Recovery Promotion: If the primary region suffers an outage, an administrator or automated failover script can invoke secretsmanager:PromoteSecret on a replica secret. This detaches the replica from the primary, converting it into an independent, fully writable primary secret capable of managing its own rotation and replicas.

---

Systems Manager Parameter Store Architecture

AWS Systems Manager Parameter Store provides centralized, secure, hierarchical storage for configuration parameters, license codes, environment flags, and secrets. Parameter Store supports three core data types:

1. String: Plaintext configuration data (e.g., AMI IDs, URLs, logging levels).
2. StringList: Comma-separated strings (e.g., subnet IDs subnet-1234,subnet-5678).
3. SecureString: Sensitive data encrypted at rest using AWS KMS envelope encryption.

Parameter Tiers: Standard vs. Advanced

Parameter Store offers two parameter tiers that govern capacity, features, and cost:

Advanced Parameter Lifecycle Policies

Advanced parameters allow administrators to attach automated lifecycle policies to enforce operational governance:

• Expiration Policy: Automatically deletes a parameter at a specified ISO-8601 timestamp. Ideal for temporary access tokens, ephemeral credentials, or trial license keys.
• ExpirationNotification Policy: Emits an Amazon EventBridge event a specified number of days or hours before the parameter's expiration timestamp, allowing security teams to initiate automated renewal workflows.
• NoChangeNotification Policy: Emits an EventBridge event if a parameter has not been updated within a specified duration (e.g., 90 days). This alerts administrators that a static credential or configuration has become stale and requires manual review.

Parameter Store Cross-Account Limitations

Parameter Store has narrower cross-account options than Secrets Manager:

• Standard parameters can't be shared. An application in Account A must assume an IAM role in Account B that has ssm:GetParameter (and kms:Decrypt for SecureString values).
• Advanced parameters can be shared with other accounts or an organization through AWS Resource Access Manager (RAM). Consumers get read-only access and must reference the parameter by its full ARN. A shared SecureString parameter must be encrypted with a customer managed KMS key whose key policy lets the consumer accounts decrypt.

---

Secrets Manager vs. Parameter Store: Decision Matrix

When designing cloud architectures, security engineers must evaluate trade-offs between Secrets Manager and Parameter Store across multiple operational criteria:

---

Specialty Exam Pitfalls & Architectural Traps

1. The AWS-Managed KMS Key Cross-Account Trap: Attempting to share a secret across AWS accounts while encrypting the secret with the default aws/secretsmanager key. The caller role will receive an AccessDeniedException on the KMS decrypt step because AWS-managed key policies cannot be modified to include cross-account principals. Always provision a Customer Managed Key (CMK) for cross-account secrets.
2. The Lambda Rotation VPC Isolation Trap: Deploying a Lambda rotation function inside a private VPC subnet to access an internal RDS instance, but failing to provide the Lambda function with access to Secrets Manager. Because Secrets Manager is a public AWS service, the Lambda function must either have outbound internet access via a NAT Gateway or, preferably, route through an AWS Secrets Manager VPC Interface Endpoint (com.amazonaws.<region>.secretsmanager) located within the VPC.
3. Out-of-Band Password Desynchronization: Manually changing a database master password in the Amazon RDS console or database CLI without updating Secrets Manager. When the automated rotation function subsequently executes setSecret, it attempts to authenticate using the old password stored in Secrets Manager, resulting in authentication failures and a broken rotation state.
4. Overlooking Parameter Store Throughput Limits: Relying on standard Parameter Store throughput (40 TPS) in high-volume containerized microservice architectures. When hundreds of container tasks boot concurrently and call ssm:GetParameter, Parameter Store throttles requests with ThrottlingException. For high-concurrency environments, you must either enable Higher Throughput (up to 10,000 TPS) or deploy client-side caching.