6.3 Patch Manager, Parameter Store vs Secrets Manager & AppConfig

SSM Patch Manager: Automated Fleet Patching & Compliance

Maintaining current security patches across heterogeneous operating system fleets is essential for regulatory compliance and vulnerability management. AWS Systems Manager Patch Manager automates the process of scanning, approving, and installing operating system and security patches across Amazon EC2 and hybrid instances at scale.

Patch Baselines: Rules, Approval Delays, and Overrides

A Patch Baseline defines which patches are approved for installation on your managed instances. Systems Manager provides predefined AWS baselines (e.g., AWS-AmazonLinux2DefaultPatchBaseline, AWS-UbuntuDefaultPatchBaseline, AWS-WindowsDefaultPatchBaseline), but enterprise governance typically requires Custom Patch Baselines.

A custom patch baseline consists of four core elements:

1. Approval Rules: Define criteria for automatic patch approval based on:
   • Product: Operating system release (e.g., Amazon Linux 2023, Ubuntu 22.04, Windows Server 2022).
   • Classification: Patch category (e.g., Security, Bugfix, Enhancement, Recommended).
   • Severity: Vendor severity rating (e.g., Critical, Important, Medium, Low).
2. Auto-Approval Delay: The number of days to wait after a vendor publishes a patch before it becomes automatically approved (e.g., 7 days). This buffer allows DevOps teams to test patches in development and staging environments before automated production rollout.
3. Explicit Approved Patches List: Patches explicitly approved regardless of approval rules. Used for emergency zero-day CVE remediation to bypass the 7-day delay.
4. Explicit Rejected Patches List: Patches explicitly prohibited from installation, even if they match approval rules. Essential for blocking known buggy packages, breaking kernel revisions (e.g., kernel-5.15*), or unvalidated dependencies.

[ Patch Release from Vendor ]
  │
  ├──> Is patch in Rejected List? ───────> YES ──> DO NOT INSTALL (Blocked)
  │      │ (NO)
  ├──> Is patch in Approved List? ───────> YES ──> APPROVE IMMEDIATELY (Zero-Day)
  │      │ (NO)
  └──> Matches Classification & Severity?
         │ (YES)
         └──> Has Auto-Approval Delay Elapsed? (e.g., 7 Days)
                ├── YES ──> APPROVED
                └── NO  ──> PENDING (Wait for delay)

Patch Groups: Associating Instances with Baselines

To apply different patch policies to different environments (e.g., approving patches with a 0-day delay in Development, but a 10-day delay in Production), use Patch Groups:

• Instances are assigned to a patch group by attaching an EC2 tag with the exact key Patch Group (case-sensitive!). The tag value represents the patch group name (e.g., Web-Production, Database-Staging).
• In Patch Manager, the patch baseline is registered to that specific patch group name.
• Cardinality Rule: An instance can belong to only one patch group. A patch baseline can be associated with multiple patch groups, but a patch group can only be registered to one patch baseline per operating system type.

Maintenance Windows: Duration vs. Cutoff Mechanics

Patch installation should not occur during peak traffic hours. Maintenance Windows define scheduled blocks of time during which disruptive tasks (such as package updates and reboots) can execute. A Maintenance Window consists of four structural parameters:

[!IMPORTANT] The Duration vs. Cutoff Rule: If a maintenance window has a Duration of 4 hours and a Cutoff of 1 hour, tasks can only initiate during the first 3 hours (02:00 to 05:00). At the 3-hour mark (05:00), the cutoff begins; Systems Manager will not dispatch any new tasks. Currently running tasks are permitted to complete during the final hour until the window closes at 06:00.

Scanning vs. Installing: Compliance Reporting

The AWS-RunPatchBaseline command document accepts an Operation parameter with two modes:

• Scan: Inspects the instance against the assigned baseline, generates a compliance inventory of installed, missing, and failed patches, and publishes the results to Systems Manager Compliance and AWS Config without modifying installed packages.
• Install: Resolves approved patches, downloads and installs them, and performs a reboot if RebootOption is set to RebootIfNeeded.

Systems Manager Parameter Store vs. AWS Secrets Manager

Managing application configuration and sensitive credentials requires choosing between two core AWS services: AWS Systems Manager Parameter Store and AWS Secrets Manager. On the AWS DevOps Professional exam, selecting the correct service depends on specific functional requirements, including automated rotation, cross-account sharing, value size limits, and cost profile.

Comprehensive Decision Matrix

Automated Rotation Lifecycle in Secrets Manager

The fundamental differentiator between Parameter Store and Secrets Manager is Secrets Manager's native automated rotation engine. When configured (e.g., every 30 days), Secrets Manager invokes a dedicated AWS Lambda rotation function that coordinates a 4-step rotation protocol:

1. createSecret: The Lambda function generates a new random password and stores it in Secrets Manager under the staging label AWSPENDING. The current production secret remains untouched under AWSCURRENT.
2. setSecret: The Lambda function connects to the target database (e.g., Amazon RDS Aurora PostgreSQL) using existing superuser credentials and alters the target user's password to match the AWSPENDING credential.
3. testSecret: The Lambda function opens a fresh connection to the database using the new AWSPENDING credentials to verify that authentication succeeds.
4. finishSecret: The Lambda function moves the AWSCURRENT version staging label to the newly validated secret version. The previous version receives the label AWSPREVIOUS. This supports rotation without planned downtime when the application refreshes credentials and handles the transition correctly.

Rotation Strategies: Secrets Manager supports Single-User rotation (updating the secret in place, with a momentary authentication blip) and Multi-User rotation (alternating between two user accounts, e.g., user_a and user_b, ensuring 100% uninterrupted zero-downtime database access).

Hierarchical Organization & Dynamic References

Parameter Store uses forward slashes to organize configuration into trees (e.g., /ecommerce/production/database/endpoint). Applications can retrieve entire configuration trees in a single API call using GetParametersByPath with --recursive.

CloudFormation templates can dynamically resolve parameters and secrets without hardcoding sensitive strings or passing parameters as plaintext:

Resources:
  AppDatabase:
    Type: AWS::RDS::DBInstance
    Properties:
      MasterUsername: '{{resolve:ssm:/app/prod/db/user}}'
      MasterUserPassword: '{{resolve:secretsmanager:app/prod/db/creds:SecretString:password}}'

---

AWS AppConfig: Runtime Configuration & Automated Rollbacks

Traditional application configuration changes (such as toggling feature flags, changing rate limits, or updating routing endpoints) frequently require re-deploying code artifacts or restarting container tasks. AWS AppConfig, a capability of AWS Systems Manager, enables runtime configuration deployment with zero application downtime, automated validation, and safety guardrails.

AppConfig Structural Hierarchy

1. Application: The logical boundary representing your application (e.g., PaymentService).
2. Environment: A deployment environment (e.g., Development, Staging, Production).
3. Configuration Profile: Defines the source and schema of the configuration data. Sources include AppConfig hosted store, SSM Parameter Store, Systems Manager Documents, Amazon S3, or AWS CodeCommit.

Dual-Layer Validation Gates

Before AppConfig deploys a new configuration version, it passes the data through mandatory validation gates to prevent syntax errors or invalid business states from entering production:

• Syntactic Validators (JSON Schema / YAML Schema): Validates structural formatting, required keys, integer ranges, and regex patterns (e.g., ensuring max_connections is an integer between 1 and 500).
• Semantic Validators (AWS Lambda Function): Executes programmatic business logic against the new configuration. For example, a Lambda validator can verify that an updated external payment gateway URL responds to health checks before approving the deployment.

Deployment Strategies & Bake Time

An AppConfig Deployment Strategy controls how rapidly the configuration is distributed across the target infrastructure:

• Growth Type: Linear (distributes fixed percentage increments over time) or Exponential (canary ramp-up).
• Step Percentage: The percentage of targets that receive the configuration at each interval (e.g., 20%).
• Deployment Duration: The total time across which the step percentages are applied (e.g., 20 minutes).
• Bake Time (Soak Time): A mandatory observation period after 100% of the configuration has been deployed (e.g., 15 minutes). AppConfig monitors attached Amazon CloudWatch alarms throughout the entire deployment duration and the bake time.

Automated Rollback via CloudWatch Alarms

You can associate one or more Amazon CloudWatch Alarms (such as HTTP 5xx error rate, synthetic canary failure, or latency spikes) with an AppConfig Environment. If any associated alarm transitions into the ALARM state at any point during deployment or bake time, AppConfig immediately initiates an automated rollback, reverting all clients to the previous known good configuration version.

Client Caching & AWS AppConfig Agent

Applications retrieve configuration via the AWS AppConfig Agent (running as a local sidecar container in Amazon ECS/EKS or local daemon on EC2) or SDK extensions. The agent polls AppConfig periodically, caches configuration locally in memory, and serves data to the application over localhost HTTP with sub-millisecond latency. If the configuration has not changed, AppConfig returns an HTTP 304 Not Modified status, eliminating unnecessary data transfer costs and API throttling.