Secrets, Least Privilege Audits, and Traceability

Secrets and least privilege on a stack that already serves traffic

Quick Answer: On HarborPay's already-deployed AWS Organizations landing zone, move rotating credentials into AWS Secrets Manager, keep short configuration in AWS Systems Manager Parameter Store, audit over-broad IAM with an unused access analyzer (not another external access analyzer), and use AWS CloudTrail to reconstruct who changed what.

HarborPay is a PCI-scoped payments processor with member accounts, a checkout API behind an Application Load Balancer (ALB), and Amazon Elastic Compute Cloud (Amazon EC2) in Auto Scaling groups. AWS IAM Identity Center federation was designed when the landing zone was born. That Domain 1 identity work is not the job here. Domain 3 asks you to improve the running org: hardcoded secrets in AMIs, IAM roles that collected AdministratorAccess, and missing who-did-what evidence. Independent SAP-C02 study material by OpenExamPrep treats this as operations on an existing estate, not a greenfield IAM redesign.

The AWS Well-Architected Framework Security pillar describes similar loops (detect, restrict, trace). Use that vocabulary as AWS names it; do not treat this chapter as a substitute for Identity Center setup taught elsewhere.

Secrets Manager versus Parameter Store

AWS Secrets Manager manages credentials, API keys, OAuth tokens, and other secrets through their lifecycle. Applications replace hardcoded strings with a runtime GetSecretValue call (or a cache such as the AWS Parameters and Secrets Lambda Extension). AWS publishes a maximum secret value size of 65,536 bytes, a default quota of 500,000 secrets per Region, 100 versions per secret, and 10,000 GetSecretValue requests per second. AWS also warns you not to call PutSecretValue or UpdateSecret more than about once every 10 minutes on a sustained basis, because unlabeled versions less than 24 hours old are not pruned and you can hit the version quota.

Parameter Store is a hierarchical key-value store for configuration. Parameter types are:

• String — plaintext such as AMI IDs and endpoints. Never store passwords here.
• StringList — comma-separated plaintext. If a value itself contains a comma, you must use String instead.
• SecureString — the value is encrypted with AWS Key Management Service (AWS KMS). Names, descriptions, and other metadata are not encrypted.

AWS's Parameter Store documentation recommends Secrets Manager for database credentials, API keys, and tokens because Parameter Store has no credential rotation. You can still retrieve a Secrets Manager secret through Parameter Store using the reserved path /aws/reference/secretsmanager/<secret-name> so older Systems Manager-aware tools keep a single reference style.

You may mix tiers: AWS documents up to 10,000 standard plus 100,000 advanced parameters in the same account and Region. You can promote Standard to Advanced; you cannot demote Advanced to Standard, because the value would be truncated from 8 KB to 4 KB and policies would drop. Throughput is independent of tier. Default retrieval (GetParameter, GetParameters, and GetParametersByPath combined) is 40 transactions per second (TPS). Higher throughput raises GetParameter to 10,000 TPS, GetParameters to 1,000 TPS, and GetParametersByPath to 100 TPS, and AWS bills those API interactions. A burst of Auto Scaling launches that all call GetParameter at boot is a classic throttling trap; cache locally or enable higher throughput before a sale weekend.

Rotation and when a customer managed key matters

Rotation updates both the secret and the database or partner system. Secrets Manager supports:

1. Managed rotation — for many secrets created through the owning service, such as Amazon Relational Database Service (Amazon RDS) master-user secrets with ManageMasterUserPassword. No Lambda function.
2. Rotation by Lambda function — steps create_secret, set_secret, test_secret, and finish_secret, with staging labels such as AWSCURRENT, AWSPENDING, and AWSPREVIOUS. Use single-user or alternating-users strategies for databases.
3. Managed external secrets — partner-held secrets without a Lambda function.

HarborPay's golden AMI that still bakes an 18 KB JSON blob of RDS users, partner API keys, and a TLS client certificate belongs in Secrets Manager, not in three Standard parameters and not in Lambda environment variables.

AWS KMS encrypts Secrets Manager secrets and Parameter Store SecureString values. An AWS managed key (aws/secretsmanager or aws/ssm) encrypts at rest. Use a customer managed key when the key policy must allow kms:Decrypt only for the checkout task role and a break-glass role, when cardholder data must use a different key than marketing config, or when you need CloudTrail events tied to a key HarborPay owns. A customer managed key does not replace IAM on the secret; still deny secretsmanager:GetSecretValue to humans who should use Identity Center for console access.

Unused access versus external access

IAM Access Analyzer is the audit toolset for a running organization. You create separate analyzers:

• External access — resource-based policies that grant access outside the zone of trust (the account or organization you select). Sharing an Amazon S3 bucket from one member account to another member is not a finding if the organization is the zone of trust; sharing to a vendor account is. External access is Regional; enable it in every Region where you have supported resources. AWS states that external access findings are offered at no additional IAM Access Analyzer charge.
• Unused access — unused roles, unused IAM user passwords and access keys, and unused service- or action-level permissions, using last-accessed information over the number of days you specify when you create the analyzer. Unused-access findings are not Region-specific; you do not create one per Region. AWS charges unused-access analysis per IAM user and role per analyzer per month. Service-linked roles are not analyzed.
• Internal access — possible access paths inside the zone of trust. AWS charges internal access by resources monitored. Add it when the scenario needs intra-org path analysis, not as a substitute for unused permissions.

A common trap is to keep adding external-access analyzers and wonder why AdministratorAccess on an internal role never appears. Over-broad identity policies are an unused-access and last-accessed problem. Do not rebuild Identity Center from scratch to fix IAM users that accumulated over three years. Review those users: unused access keys, unused console passwords, and roles that have not been assumed inside the tracking window. Attach a permissions boundary, replace inline * policies with job-sized policies, then re-scan.

CloudTrail for who-did-what on existing stacks

AWS CloudTrail records management events from the console, CLI, and APIs. Event history gives 90 days of management events per Region with no CloudTrail charge for viewing. A trail delivers events to Amazon S3 (and optionally Amazon CloudWatch Logs) for longer retention; AWS does not charge CloudTrail for one copy of ongoing management events to S3, but S3 storage is billed. Organization-trail topology, data events, and CloudTrail Lake retention (AWS publishes up to 3,653 days or 2,557 days depending on the Lake pricing option) belong to the data-protection chapter. Here you use the trail HarborPay already has.

Query who called iam:PutRolePolicy, iam:AttachRolePolicy, secretsmanager:GetSecretValue, or kms:Decrypt against the checkout role. Combine that with Access Analyzer findings so you can show an assessor both that a role can do too much and which principal attached the policy.

Run this improvement sequence on the live org:

1. Inventory secrets in AMIs, user data, Lambda environment variables, and Parameter Store String parameters; move credentials to Secrets Manager and non-secret config to Parameter Store.
2. Encrypt PCI secrets with a customer managed key and deny decrypt except to the application role.
3. Create an unused-access analyzer at organization scope; keep external-access analyzers in every Region that has supported resources.
4. Tighten IAM users and roles that grew; convert unused IAM users to Identity Center users only when that is the actual finding, not as a reflex rebuild.
5. Prove the change with CloudTrail lookups on the existing trail.

Traps: storing secrets in String parameters because Standard storage has no additional charge; expecting Parameter Store to rotate RDS passwords; treating unused access as a free Regional replica of external access; and using a Domain 1 federation redesign as a substitute for least privilege on roles that already exist.

Official resources

• What is AWS Secrets Manager?
• AWS Secrets Manager quotas
• AWS Systems Manager Parameter Store
• Managing Parameter Store throughput
• IAM Access Analyzer findings
• What is AWS CloudTrail?