9.1 Infrastructure as Code (IaC) with Terraform on Google Cloud

Infrastructure as Code (IaC) with Terraform on Google Cloud

Architectural Objective: Infrastructure as Code (IaC) is the cornerstone of repeatable, auditable, and resilient enterprise cloud operations. A Google Professional Cloud Architect must design robust Terraform architectures using Google Cloud Storage remote backends, structure enterprise-grade modular blueprints, isolate multi-environment state boundaries, automate CI/CD pipeline execution with least-privilege service account impersonation, manage infrastructure drift, and evaluate Kubernetes-native declarative tools like Google Cloud Config Connector.

---

Declarative Infrastructure Principles & The Google Cloud Providers

In modern cloud engineering, infrastructure automation follows two paradigms:

+---------------------------------------------------------------------------------------------------+
|                         IMPERATIVE VS. DECLARATIVE INFRASTRUCTURE                                 |
+---------------------------------------------------------------------------------------------------+
| PARADIGM    | MECHANISM                   | EXECUTION CHARACTERISTICS     | FAILURE PROFILE       |
+-------------+-----------------------------+-------------------------------+-----------------------+
| Imperative  | Bash scripts, gcloud CLI,   | Defines step-by-step procedural| Partial failures leave|
|             | Client SDKs, REST APIs      | instructions on *how* to build| orphaned resources and|
|             |                             | resources sequentially.       | unrecoverable state.  |
+-------------+-----------------------------+-------------------------------+-----------------------+
| Declarative | HashiCorp Terraform (HCL),  | Defines the *desired end state*| Engine calculates dependency|
|             | Config Connector (CRDs),    | of infrastructure; engine     | graph and applies only|
|             | KRM (Kubernetes Resource)   | reconciles current vs target. | necessary deltas.     |
+---------------------------------------------------------------------------------------------------+

The google and google-beta Providers

HashiCorp Terraform interacts with Google Cloud APIs through two distinct upstream providers:

1. google Provider: Contains generally available (GA) resources and features supported by Google Cloud SLAs. Used for production-critical infrastructure components.
2. google-beta Provider: Exposes preview, alpha, and beta capabilities alongside GA resources. In enterprise architectures, google-beta is referenced when bleeding-edge features (such as newer GKE Autopilot settings or private service connect attributes) are required before general availability.

# Provider configuration pattern for enterprise blueprints
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.30.0"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
      version = "~> 5.30.0"
    }
  }
}

---

Terraform State Architecture: Google Cloud Storage Remote Backend

Terraform records metadata and mappings between declared HCL configurations and real-world GCP infrastructure in a terraform.tfstate file. Storing state locally on developer laptops introduces severe concurrency risks, lost state, and security leaks (as state files contain cleartext metadata and potential secrets).

+-----------------------------------------------------------------------------------+
|                     GCS REMOTE STATE BACKEND ARCHITECTURE                         |
+-----------------------------------------------------------------------------------+
| [ Developer / CI/CD ] ──> terraform apply ──> [ GCS Backend: gs://tf-state-prod ]  |
|                                                      │                            |
|                                    ┌─────────────────┴─────────────────┐          |
|                                    │  1. Object Generation Locking     │          |
|                                    │  2. Object Versioning (Backups)   │          |
|                                    │  3. Uniform Bucket-Level Access   │          |
|                                    │  4. CMEK Encryption (Cloud KMS)   │          |
|                                    └───────────────────────────────────┘          |
+-----------------------------------------------------------------------------------+

Essential Architectural Guardrails for GCS State Backends

1. Native Object Generation State Locking: Unlike AWS (which requires a separate DynamoDB table for distributed state locks), Google Cloud Storage natively supports object generation preconditions (x-goog-if-generation-match). When Terraform initiates an execution, it creates a default.tflock object in the GCS bucket. If another process attempts a concurrent apply, GCS rejects the write with a 412 Precondition Failed error, guaranteeing lock safety without additional database infrastructure.
2. Object Versioning Enabled: State corruption can occur during unexpected network interruptions or faulty manual state manipulations. Enabling Object Versioning on the GCS state bucket ensures every state mutation preserves prior historical versions, allowing immediate restoration of previous state snapshots.
3. Uniform Bucket-Level Access & IAM Isolation: All access to the state bucket must be governed strictly through IAM roles (roles/storage.objectAdmin or roles/storage.objectViewer) assigned to specific automation service accounts. ACLs must be disabled by enforcing Uniform Bucket-Level Access.
4. Customer-Managed Encryption Keys (CMEK): Because Terraform state files may contain sensitive connection strings, database passwords, or private IP schemas, encrypting the GCS bucket with a Cloud KMS key ensures strict organizational cryptographic control and auditability.

# GCS Backend Configuration
terraform {
  backend "gcs" {
    bucket                      = "cft-tfstate-prod-uscentral1-8934"
    prefix                      = "infrastructure/vpc-networking"
    impersonate_service_account = "sa-tf-deployer@cft-seed-prod.iam.gserviceaccount.com"
  }
}

---

Modular Architecture & The Cloud Foundation Toolkit (CFT)

Enterprise Terraform code must be modular, reusable, and composable to prevent massive, unmaintainable monolithic state files ("blast radius concentration").

+-----------------------------------------------------------------------------------+
|                    ENTERPRISE TERRAFORM MODULE TAXONOMY                           |
+-----------------------------------------------------------------------------------+
| 1. FOUNDATIONAL MODULES | Highly opinionated, reusable building blocks (VPC,      |
|    (Child Modules)      | GKE, Cloud SQL, IAM Bindings) curated by Platform SRE.  |
+-------------------------+---------------------------------------------------------+
| 2. ENTERPRISE BLUEPRINTS| Composed stacks assembling multiple child modules into  |
|    (Landing Zones)      | full organizational environments (Core Shared VPC + IAM)|
+-------------------------+---------------------------------------------------------+
| 3. ROOT MODULES         | The executable entry points defining environment-       |
|    (Environment Layer)  | specific variables (dev/main.tf, prod/main.tf).         |
+-----------------------------------------------------------------------------------+

Google Cloud Foundation Toolkit (CFT)

The Cloud Foundation Toolkit (CFT) provides open-source, production-ready, CIS-benchmark-compliant Terraform modules developed and maintained by Google Cloud engineers. Using CFT modules dramatically reduces boilerplate and guarantees compliance with Google Cloud Well-Architected Framework best practices.

Module Design Principles

• Single Responsibility: A child module should manage a single logical domain (e.g., terraform-google-network for subnets and Cloud NAT, or terraform-google-kubernetes-engine for GKE clusters).
• Explicit Inputs and Outputs: Expose strictly typed variable inputs with sensible defaults and output resource IDs, self-links, and service account emails to enable composition.
• Avoid Hardcoded Secrets: Pass sensitive values as variables sourced from Secret Manager or Cloud KMS rather than hardcoding credentials inside modules.

---

Environment Isolation: Workspaces vs. Multi-Project Directory Structures

A critical decision on the Cloud Architect exam involves how to isolate environments (Development, Staging, Production).

TERRAFORM WORKSPACES (ANTI-PATTERN FOR PROD)   DIRECTORY-SEPARATED HIERARCHY (BEST PRACTICE)
┌────────────────────────────────────────┐     ┌────────────────────────────────────────────────────────┐
│ Root Module                            │     │ terraform-live/                                        │
│  ├── main.tf                           │     │  ├── environments/                                     │
│  └── (Workspaces: dev, stage, prod)    │     │  │   ├── dev/                                          │
│  [!] Shares single backend bucket      │     │  │   │   ├── main.tf (Backend: dev-state-bucket)       │
│  [!] Shares single IAM Service Account │     │  │   │   └── terraform.tfvars                          │
│  [!] High Blast Radius: apply in prod  │     │  │   ├── staging/                                      │
│      can corrupt dev state             │     │  │   └── prod/                                         │
└────────────────────────────────────────┘     │          ├── main.tf (Backend: prod-state-bucket)      │
                                               │          └── (Isolated Service Account & IAM)          │
                                               │  [OK] Absolute Blast Radius & Security Isolation       │
                                               └────────────────────────────────────────────────────────┘

Architectural Comparison: Workspaces vs. Directory-Based Isolation

---

Automated CI/CD Execution via Cloud Build & Service Account Impersonation

Executing Terraform from developer workstations creates compliance risks and security vulnerabilities. Enterprise deployments must utilize automated pipelines.

+-----------------------------------------------------------------------------------+
|                 SECURE TERRAFORM CI/CD EXECUTION FLOW                             |
+-----------------------------------------------------------------------------------+
| 1. Developer pushes Git PR ──> Cloud Build Trigger activates                      |
| 2. Cloud Build Runner (Short-Lived Worker)                                         |
|    └── Impersonates: sa-terraform-runner@project.iam.gserviceaccount.com          |
| 3. Execution Phase:                                                               |
|    ├── terraform init & terraform validate                                        |
|    ├── Policy-as-Code Check: terraform-validator / Conftest / Google PolicyEngine  |
|    └── terraform plan -out=tfplan.binary                                          |
| 4. PR Approved & Merged to main ──> Manual Approval Gate ──> terraform apply      |
+-----------------------------------------------------------------------------------+

Service Account Impersonation vs. Static Keys

• Static Service Account Keys (.json keys): Present severe security risks due to accidental Git commits, credential theft, and lack of automatic expiration. Google Cloud strongly discourages static JSON keys.
• Service Account Impersonation: The Cloud Build service account is granted the roles/iam.serviceAccountTokenCreator role on the target Terraform Deployment Service Account. During pipeline execution, Cloud Build requests a short-lived OAuth 2.0 access token (valid for 1 hour) to execute infrastructure provisioning operations, eliminating stored credentials entirely.

Policy-as-Code: Terraform Validator & Policy Intelligence

Before applying infrastructure changes, pipelines enforce organizational compliance guardrails using Terraform Validator (or Google Cloud Policy Controller / OPA Conftest):

• Converts the binary tfplan into Google Cloud Resource Manager JSON.
• Evaluates proposed resources against Organization Policy Constraints (e.g., forbidding public IP addresses on Cloud SQL or enforcing CMEK encryption on GCS buckets).
• Fails the build before any API call is made if a policy violation is detected.

---

Managing Drift, State Import & Resource Lifecycle

Infrastructure Drift Detection & Remediation

• Drift Origin: Infrastructure drift occurs when resources managed by Terraform are modified out-of-band via the Google Cloud Console, gcloud CLI, or automated cloud agents.
• Detection: Running terraform plan -refresh-only queries Google Cloud APIs, updates the local state representation without proposing resource modifications, and highlights differences between code, state, and actual cloud assets.
• Remediation: Running terraform apply overwrites the out-of-band changes, forcing the real-world GCP infrastructure back into compliance with the declared HCL code.

Importing Existing Resources (terraform import & import {} blocks)

When adopting Terraform for pre-existing cloud environments, resources must be imported into the state file to prevent duplicate creation errors:

# Modern declarative import block (Terraform 1.5+)
import {
  to = google_compute_network.custom_vpc
  id = "projects/prod-networking-host/global/networks/vpc-prod-core"
}

Essential Resource Lifecycle Meta-Arguments

---

Kubernetes-Native Declarative IaC: Google Cloud Config Connector (KCC)

For organizations operating heavily on Google Kubernetes Engine (GKE), Google Cloud Config Connector (KCC) offers a Kubernetes-native alternative to Terraform.

+-----------------------------------------------------------------------------------+
|                   CONFIG CONNECTOR CONTINUOUS RECONCILIATION                      |
+-----------------------------------------------------------------------------------+
| Developer writes K8s YAML ──> kubectl apply ──> GKE Control Plane                 |
|                                                      │                            |
|                                          [ Config Connector Operator ]            |
|                                                      │ (Workload Identity)        |
|                                                      v                            |
|                                            Google Cloud REST APIs                 |
|                                        (Creates Cloud SQL / PubSub)               |
|                                                      │                            |
|                                          [ Continuous Reconciliation Loop ]       |
|                                          (Automatically heals out-of-band drift)  |
+-----------------------------------------------------------------------------------+

Terraform vs. Config Connector Architectural Decision Matrix

---

Concrete Architectural Scenario: Enterprise Multi-Tier Landing Zone

Scenario Profile

• Organization: Global financial services enterprise migrating retail banking microservices to Google Cloud.
• Compliance & Availability SLA: Zero state loss, 100% auditability of infrastructure changes, no static credentials, strict prevention of accidental production database destruction.

Solution Architecture Blueprint

1. State Storage: Separate GCS buckets provisioned per environment (tf-state-dev, tf-state-stage, tf-state-prod) with Object Versioning, CMEK encryption via Cloud KMS, and Uniform Bucket-Level Access. State locking is handled natively by GCS object generation preconditions.
2. Directory Structure: Directory-separated environment layout using Cloud Foundation Toolkit (CFT) modules for VPC, GKE, and IAM. Workspaces are strictly prohibited for production environments.
3. CI/CD Execution: Cloud Build pipeline triggered on GitHub pull requests. The build runner uses Service Account Impersonation (roles/iam.serviceAccountTokenCreator) to generate short-lived tokens. The pipeline executes terraform-validator against organization policies before generating a deterministic binary execution plan.
4. Database Protection: Production Cloud Spanner and Cloud SQL instances configured with lifecycle { prevent_destroy = true } to eliminate risk of human error in CI/CD automation.

---

[!IMPORTANT] Exam Watch: For the PCA exam, remember that Google Cloud Storage remote backends for Terraform provide automatic native state locking without requiring an external database. If an exam question asks how to organize multi-environment Terraform infrastructure securely with minimal blast radius and independent access controls, choose directory-separated project structures with distinct backend buckets and service accounts, never Terraform workspaces.