14.4 Automated Secrets Management & Secret Rotation with Secrets Manager

AWS Secrets Manager Architecture & Core Capabilities

Hardcoded database credentials, API tokens, and private encryption keys inside source code repositories or configuration files represent one of the most severe security vulnerabilities in modern DevOps pipelines. AWS Secrets Manager eliminates this risk by providing centralized, encrypted secret storage with native automated lifecycle rotation.

Secret Storage and Encryption Mechanics

• Payload Format: Secrets Manager stores secrets as JSON key-value pairs (ideal for database credentials: username, password, host, port, dbname) or arbitrary binary/plaintext strings.
• AWS KMS Envelope Encryption: All secrets are encrypted at rest using AWS Key Management Service (KMS). Organizations can utilize the default AWS-managed key (aws/secretsmanager) or a Customer Managed Key (CMK). A CMK is mandatory when secrets must be shared across AWS accounts or when strict regulatory compliance mandates customer-controlled key rotation and auditing.
• Resource-Based Policies: Secrets Manager secrets support resource-based policies. This allows a secret in a centralized Security or Shared Services account to be accessed directly by IAM execution roles in spoke application accounts without requiring cross-account role assumption.

---

Secret Versioning & Staging Labels

Secrets Manager versions are immutable. Each modification or rotation creates a new version containing a unique VersionId (UUID). Versions are tracked and manipulated using Staging Labels:

---

The 4-Step Rotation Protocol

When automated rotation is enabled (e.g., every 30 days) or manually triggered via the RotateSecret API, Secrets Manager initiates an execution workflow that invokes an AWS Lambda rotation function four consecutive times. Each invocation passes a specific Step parameter in the event payload:

Secrets Manager Rotation Engine
       │
       ├─ 1. Step: "createSecret" ──> Lambda creates AWSPENDING version with new random password
       │
       ├─ 2. Step: "setSecret" ────> Lambda connects to DB (as AWSCURRENT or Master) and updates DB password
       │
       ├─ 3. Step: "testSecret" ───> Lambda tests DB connection using AWSPENDING password
       │
       └─ 4. Step: "finishSecret" ─> Lambda moves AWSCURRENT label to AWSPENDING; marks old version AWSPREVIOUS

The Four Steps in Detail

1. createSecret: The Lambda function queries Secrets Manager to determine if a version with the AWSPENDING staging label already exists. If not, it generates a cryptographically secure random password (e.g., using secrets.token_urlsafe or GetRandomPassword) and writes the new credentials to Secrets Manager under a new VersionId labeled AWSPENDING. If AWSPENDING already exists, it leaves it intact to ensure idempotency across automated retries.
2. setSecret: The Lambda function retrieves the current credentials (from AWSCURRENT or master credentials) and establishes an administrative connection to the target database engine (e.g., Amazon RDS MySQL, PostgreSQL, or Aurora). It executes an ALTER USER or SET PASSWORD statement to update the database user's password in the database engine to match the password stored in the AWSPENDING secret version.
3. testSecret: The Lambda function attempts to establish a brand-new database connection using the AWSPENDING credentials. It executes a lightweight validation query (such as SELECT 1;). If authentication fails, the function raises an exception, the rotation process halts, and AWSCURRENT remains completely unchanged.
4. finishSecret: Once testSecret succeeds, the Lambda function calls the Secrets Manager UpdateSecretVersionStage API. It moves the AWSCURRENT label from the old version to the AWSPENDING version, automatically applies AWSPREVIOUS to the older version, and strips the AWSPENDING label. Applications querying GetSecretValue now receive the newly validated credentials.

[!IMPORTANT] Idempotency Requirement: Secrets Manager can retry any rotation step if a transient network failure occurs. The Lambda rotation function must be written to be strictly idempotent. For instance, in setSecret, if the database password was already updated in an earlier attempt that timed out during response delivery, the Lambda function must verify connectivity with AWSPENDING before failing.

---

Private VPC Networking for Secrets Manager Rotation

In enterprise production environments, databases (Amazon RDS, Amazon Aurora) are deployed inside isolated private VPC subnets without public internet access or public IP addresses. This creates a critical networking requirement for the rotation Lambda function.

┌────────────────────────────────────────────────────────────────────────┐
│ Amazon VPC (10.0.0.0/16)                                               │
│                                                                        │
│   ┌────────────────────────────────┐    ┌──────────────────────────┐   │
│   │ Private Application Subnet     │    │ Private Database Subnet  │   │
│   │                                │    │                          │   │
│   │  [ Lambda Rotation Function ]  │───>│ [ Amazon RDS Database ]  │   │
│   │     Attached to VPC Subnets    │TCP │   Port 3306 / 5432       │   │
│   │     Security Group: Lambda-SG  │    │   Security Group: DB-SG  │   │
│   └───────────────┬────────────────┘    └──────────────────────────┘   │
│                   │                                                    │
│                   │ HTTPS (Port 443) via AWS PrivateLink               │
│                   ▼                                                    │
│   ┌────────────────────────────────┐    ┌──────────────────────────┐   │
│   │ VPC Interface Endpoint:        │    │ VPC Interface Endpoint:  │   │
│   │ com.amazonaws.<reg>.secretsmanager │    │ com.amazonaws.<reg>.kms  │   │
│   └────────────────────────────────┘    └──────────────────────────┘   │
└────────────────────────────────────────────────────────────────────────┘

Mandatory Networking Components

1. VPC Subnet Attachment: The Lambda rotation function must be configured with VPC attachment, placing its Elastic Network Interfaces (ENIs) inside the private subnets where the database is reachable.
2. Security Group Rules:
   • Database Security Group: Must have an inbound rule allowing TCP traffic on the database port (e.g., 3306 for MySQL, 5432 for PostgreSQL) originating from the Lambda function's security group.
   • Lambda Security Group: Outbound rule allowing traffic to the database security group on the database port, and outbound rule allowing HTTPS (port 443) to the VPC Interface Endpoints.
3. AWS PrivateLink (VPC Interface Endpoints): Because the Lambda function is inside a private VPC without internet access (no NAT Gateway or Internet Gateway), it cannot reach the public Secrets Manager or AWS KMS API endpoints. You must provision VPC Interface Endpoints for:
   • com.amazonaws.<region>.secretsmanager
   • com.amazonaws.<region>.kms (if using customer-managed KMS keys)
   • Both endpoints must have Private DNS enabled and attach a security group allowing inbound HTTPS (port 443) from the Lambda rotation function.

---

Rotation Strategies: Single-User vs. Alternating Dual-User

[ Dual-User Zero-Downtime Rotation Cycle ]

Phase 1 (Normal Operations):
  - Database Users: User_A (Active), User_B (Standby)
  - Secrets Manager: AWSCURRENT points to User_A
  - Applications connect using User_A

Phase 2 (Rotation Triggered):
  - Master Secret used to connect to database
  - Lambda alters User_B's password in the database to AWSPENDING password
  - Lambda tests connection using User_B with AWSPENDING password
  - Applications continue processing transactions using User_A uninterrupted!

Phase 3 (Promotion):
  - finishSecret promotes User_B to AWSCURRENT; User_A becomes AWSPREVIOUS
  - Applications refresh credentials and transition seamlessly to User_B

---

Client-Side Caching & Scalability

In high-throughput microservices architectures (e.g., an Amazon EKS cluster processing 20,000 requests per second), applications must never call GetSecretValue on every incoming transaction. Secrets Manager API calls are rate-limited per region and incur costs ($0.05 per 10,000 API calls). Querying Secrets Manager per request triggers ThrottlingException: Rate exceeded and generates massive AWS bills.

AWS Secrets Manager Client-Side Caching Libraries

AWS provides native open-source caching libraries (for Java, Python, Go, Node.js, and .NET):

• In-Memory Caching: Credentials are cached in application process memory with a configurable Time-to-Live (TTL, e.g., default of 300 seconds / 5 minutes to 1 hour).
• Asynchronous Background Refresh: Caching libraries refresh secrets in the background before the TTL expires, ensuring zero added latency for user requests.
• Authentication Failure Hook: If the application experiences an authentication failure (AccessDenied or SQLException), the caching library automatically invalidates the cached secret immediately and performs a synchronous fresh GetSecretValue call to fetch the newly rotated AWSCURRENT credentials.