8.3 Secret Delivery, Rotation, and IAP

Secret Manager: Architecture, Access & Delivery Patterns

Google Cloud Secret Manager provides centralized, audited, and encrypted storage for API keys, database credentials, TLS certificates, and sensitive configuration strings.

Key Concepts

• Secrets and Versions: A Secret is a global resource that holds metadata and rotation policies. The actual sensitive payload resides inside immutable Secret Versions (e.g., projects/my-project/secrets/db-pass/versions/1).
• Access Control: Enforce least privilege using granular IAM roles:
  • roles/secretmanager.secretAccessor: Permits fetching the secret payload (secretmanager.versions.access). This is the only role application service accounts should receive.
  • roles/secretmanager.viewer: Permits reading secret metadata only (names, versions, labels); cannot view secret payloads.
  • roles/secretmanager.admin: Grants full administrative control.

Secret Delivery Patterns: Dynamic SDK vs. Env Vars vs. Volume Mounts

Exam Trap: When secrets are injected into Cloud Run as environment variables using :latest, Cloud Run evaluates the secret version only when the container instance starts. If a developer adds a new secret version in Secret Manager, running container instances continue using the old secret value until they are restarted or redeployed. To achieve automated rotation without restarts, mount the secret as a volume or retrieve it dynamically using the Client Library.

---

Automated Secret Rotation Patterns

Secret Manager supports automated rotation schedules configured via Cloud Pub/Sub and Cloud Functions (or Cloud Run jobs):

+--------------------------------------------------------------------------+
| Automated Secret Rotation Architecture                                   |
+--------------------------------------------------------------------------+
|                                                                          |
| [ Secret Manager ]                                                       |
|        |                                                                 |
|        | 1. Rotation Schedule Timer Fires (e.g., every 30 days)          |
|        v                                                                 |
| [ Cloud Pub/Sub Topic ] ('projects/my-project/topics/secret-rotate')     |
|        |                                                                 |
|        | 2. Delivers Rotation Notification Event                         |
|        v                                                                 |
| [ Cloud Function: Rotator ]                                              |
|        |                                                                 |
|        +---> 3. Generates New Secure Password / API Key                  |
|        +---> 4. Updates Downstream Service (e.g., ALTER USER in Cloud SQL|
|        +---> 5. Adds New Secret Version in Secret Manager                |
|        +---> 6. Verifies Adoption; Retires Old Version after Rollback Window               |
+--------------------------------------------------------------------------+

Zero-Downtime Dual-Version Strategy

When rotating database passwords or API keys, downstream systems must support a dual-credential grace period:

1. The rotation Cloud Function creates a new database user or secondary API key.
2. The function adds the new credential as a new version in Secret Manager.
3. Microservices pick up the new secret version.
4. Once all clients are verified to have transitioned, including slow workers and connection pools (a fixed one-hour delay is not proof), the old credential version is disabled and destroyed.

---

Identity-Aware Proxy (IAP) & Zero-Trust Validation

Identity-Aware Proxy (IAP) establishes a zero-trust perimeter around web applications deployed on App Engine, Compute Engine, GKE, or Cloud Run. It verifies user identity and context (device health, geographic location, IP address) at the Google Cloud Load Balancing edge before permitting traffic to reach backend services.

+--------------------------------------------------------------------------+
| Identity-Aware Proxy (IAP) Zero-Trust Ingress Flow                       |
+--------------------------------------------------------------------------+
| [ Corporate User ]                                                       |
|        |                                                                 |
|        | 1. HTTPS GET /dashboard                                         |
|        v                                                                 |
| [ Cloud Load Balancer + Identity-Aware Proxy (IAP) ]                     |
|        |                                                                 |
|        | 2. Evaluates Google Identity & BeyondCorp Context Policies      |
|        |    (Checks roles/iap.httpsResourceAccessor)                     |
|        | 3. Injects Cryptographically Signed Assertion Header:           |
|        |    'X-Goog-IAP-JWT-Assertion'                                   |
|        v                                                                 |
| [ Backend Application (GKE / Cloud Run) ]                                |
|        |                                                                 |
|        | 4. Backend Cryptographically Validates JWT Signature via JWKS   |
|        | 5. Verifies 'aud' claim: /projects/NUM/global/backendServices/ID|
|        | 6. Extracts verified user identity: claims['email']             |
+--------------------------------------------------------------------------+

The Developer Trap: Plain Header Spoofing

When IAP successfully authenticates a user, it injects plain HTTP headers into the proxied request:

• X-Goog-Authenticated-User-Email
• X-Goog-Authenticated-User-Id

Exam Trap: Never rely solely on plain HTTP headers like X-Goog-Authenticated-User-Email for authorization. If an attacker gains access to the internal VPC or compromises another service on the same network, they can forge plain HTTP headers and bypass IAP authentication. To guarantee authenticity, the backend application must cryptographically validate the signed JWT assertion header.

Validating X-Goog-IAP-JWT-Assertion in Code

IAP injects a signed JSON Web Token (JWT) in the X-Goog-IAP-JWT-Assertion header. Secure backend applications must perform four validation steps:

1. Fetch Public Keys: Retrieve Google's public JSON Web Key Set (JWKS) from https://www.gstatic.com/iap/verify/public_key-jwk.
2. Verify Cryptographic Signature: Verify the JWT's ECDSA signature using the public keys.
3. Validate Issuer (iss): Confirm that the issuer claim equals https://cloud.google.com/iap.
4. Validate Audience (aud): Confirm that the audience claim matches your specific backend service string:

   /projects/PROJECT_NUMBER/global/backendServices/SERVICE_ID
   

## Python Flask IAP JWT Cryptographic Verification
from google.auth.transport import requests
from google.oauth2 import id_token

def validate_iap_jwt(iap_jwt, expected_audience):
    try:
        # Decodes and verifies signature against Google's public keys
        decoded_jwt = id_token.verify_token(
            iap_jwt,
            requests.Request(),
            audience=expected_audience,
            certs_url='https://www.gstatic.com/iap/verify/public_key-jwk'
        )
        if decoded_jwt.get('iss') != 'https://cloud.google.com/iap':
            raise PermissionError('Unexpected IAP issuer')
        return decoded_jwt['email'], decoded_jwt['sub']
    except Exception as e:
        raise PermissionError(f"IAP JWT verification failed: {e}")

---

Application Security & Identity Architecture Matrix