1.1 Cloud IAM, Service Accounts, and Resource Hierarchy in GCP Data Systems

1.1 Cloud IAM, Service Accounts, and Resource Hierarchy in GCP Data Systems

Exam Focus: The Google Cloud Professional Data Engineer exam rigorously tests your ability to enforce the Principle of Least Privilege, architect robust multi-tenant access models across decoupled compute and storage tiers, eliminate static credentials using Workload Identity Federation, and safeguard sensitive analytical repositories against data exfiltration using VPC Service Controls.

In modern enterprise data platforms, security cannot be treated as an external boundary or an afterthought. Analytical systems aggregate high-value, sensitive data assets from disparate operational databases, third-party streams, and external partners. Configuring appropriate security controls requires a deep understanding of Google Cloud's resource hierarchy, Identity and Access Management (IAM) evaluation logic, service account topologies, and exfiltration prevention boundaries.

---

1. GCP Resource Hierarchy and Additive Policy Inheritance

Google Cloud organizes all cloud resources into a hierarchical structure consisting of four levels: Organization, Folders, Projects, and Resources (such as BigQuery datasets, Cloud Storage buckets, Cloud Spanner instances, and Pub/Sub topics).

Organization (example.com)
  └── Data Platform Folder
        ├── Analytics Core Project
        │     ├── BigQuery Dataset (Marketing_Analytics)
        │     └── Cloud Storage Bucket (gs://marketing-raw-lake)
        └── Ingestion Pipeline Project
              └── Pub/Sub Topic (clickstream-events)

Additive Policy Union

The fundamental rule governing IAM policy evaluation across the hierarchy is that policies are strictly additive. When a client requests access to a specific resource, Google Cloud IAM evaluates the union of all allow policies attached to that resource and all of its ancestor nodes.

$\text{Effective Permissions} = \text{Resource Policy} \cup \text{Project Policy} \cup \text{Folder Policy} \cup \text{Organization Policy}$

If a user or service account is granted roles/storage.objectAdmin at the Folder level, that identity possesses object administrator privileges over every bucket within every project under that folder. That permission cannot be removed or overridden at a project or bucket level using standard IAM allow policies. Attempting to revoke a role on a child resource when it was granted at a parent node has no effect on the effective access.

Exam Trap: You cannot use an IAM allow policy on a BigQuery dataset to "deny" or "restrict" access granted at the parent project level. To restrict permissions at child levels, permissions must be granted at the lowest possible tier of the hierarchy (e.g., at the dataset level rather than the project level). Where explicit organizational denial is necessary, Google Cloud IAM Deny Policies must be configured at the Organization or Folder level, which take precedence over all allow policies.

Control Plane vs. Data Plane Permissions

A critical distinction tested on the exam is the separation between Control Plane (Resource Manager) and Data Plane permissions:

• Control Plane: Operations that manage metadata, project configurations, and resource lifecycles (e.g., resourcemanager.projects.get, bigquery.datasets.create, storage.buckets.update).
• Data Plane: Operations that inspect, read, append, or mutate the actual data stored inside resources (e.g., bigquery.tables.getData, storage.objects.get, pubsub.messages.pull).

Enterprise architectures strictly isolate these planes. For instance, data analysts frequently require data plane read access across datasets without being granted project-level administrative or modification permissions.

---

2. IAM Member Types and Group-Based Access Control

Google Cloud IAM binds roles to specific member types (identities):

1. Google Account: Represents a single human user identified by an email address (e.g., analyst@company.com). Direct assignment to individual users should be avoided in production.
2. Google Group: A collection of Google Accounts and service accounts managed under a single email address (e.g., data-engineers@company.com). Best Practice: Always grant IAM roles to Google Groups rather than individuals. When engineers join, transition, or leave teams, group membership updates in the central identity provider (IdP) immediately adjust permissions without touching cloud IAM policies.
3. Service Account: A non-human identity used by applications, automated pipelines, and compute resources to authenticate and call Google Cloud APIs.
4. Google Workspace / Cloud Identity Domain: All accounts created within an organization's domain.
5. Special Identifiers: allUsers (anyone on the public internet) and allAuthenticatedUsers (any account authenticated with Google). In analytical systems, organizational policy constraints (constraints/iam.allowedPolicyMemberDomains) should block allUsers and allAuthenticatedUsers to prevent accidental public data exposure.

---

3. Role Taxonomy: Primitive, Predefined, and Custom Roles

Cloud IAM roles represent collections of granular permissions. They are categorized into three distinct classes:

Primitive Roles (Legacy)

• Roles: roles/viewer, roles/editor, roles/owner.
• Characteristics: Broad, coarse-grained roles that predate fine-grained IAM. For example, roles/editor grants read, write, and modify privileges across almost every service in the project, including deploying compute engines, altering network routes, and mutating storage.
• Production Rule: Never use primitive roles in production data pipelines or for human operators. They violate the principle of least privilege and dramatically expand the blast radius of any compromised credential.

Predefined Roles (Recommended Standard)

• Characteristics: Fine-grained, service-specific roles engineered and maintained by Google Cloud. When Google adds new APIs or permissions to a service, predefined roles are automatically updated.
• BigQuery Compute vs. Storage Decoupling: In BigQuery, query compute is completely decoupled from underlying table storage. This decoupling is reflected directly in predefined IAM roles:
  • roles/bigquery.jobUser: Grants permission to run query jobs, list jobs, and consume query slots within the compute project (bigquery.jobs.create). It does not grant permission to read any table data.
  • roles/bigquery.dataViewer: Grants read-only data plane access to view dataset metadata, tables, and views (bigquery.tables.getData). It does not grant permission to run queries or consume slots.
  • roles/bigquery.dataEditor: Grants permission to create, update, and delete tables and data within a dataset.
  • roles/bigquery.admin: Full administrative control over all BigQuery resources in the project.

Architecture Pattern: Cross-Project BigQuery Access

In enterprise environments, organizations frequently separate compute billing from centralized analytical datasets:

• Storage Project (data-lake-prod): Central dataset repository. The analyst group is granted roles/bigquery.dataViewer on the specific dataset.
• Compute Project (bi-queries-prod): Project hosting user queries and slot allocation. The analyst group is granted roles/bigquery.jobUser on this project.
• Result: Analysts run queries billed to bi-queries-prod while reading read-only assets from data-lake-prod, ensuring zero accidental data mutation and precise cost allocation.

-- Query executed in project 'bi-queries-prod', referencing storage in 'data-lake-prod'
SELECT customer_id, SUM(order_total) AS total_spend
FROM `data-lake-prod.finance_analytics.orders`
WHERE order_date >= '2026-01-01'
GROUP BY customer_id;

Custom Roles

• When to Use: When no predefined role matches the exact security requirement, or when a predefined role bundles unwanted permissions (e.g., you want to allow an automated script to read BigQuery schema definitions without viewing actual table row data).
• Limitations: Custom roles cannot be created at the folder level if managed via the console in older topologies, must be maintained manually when Google updates APIs, and cannot contain certain unsupported permissions.

---

4. Service Accounts Architecture: User-Managed, Default, and Google-Managed

Service accounts are identified by an email address and authenticate server-to-server interactions without human intervention. Understanding the three distinct types is vital for exam success:

1. User-Managed Service Accounts

• Naming Format: [sa-name]@[project-id].iam.gserviceaccount.com
• Characteristics: Explicitly created, configured, and managed by data platform engineers. They represent dedicated identities for applications (e.g., dataflow-etl-runner@my-data-proj.iam.gserviceaccount.com).
• Best Practice: Create a dedicated user-managed service account for each independent workload (e.g., one for ingestion, one for transformation, one for dashboard synchronization), granting each account only the exact predefined roles required for its specific operational boundary.

2. Default Service Accounts

• Compute Engine Default Service Account: [project-number]-compute@developer.gserviceaccount.com
• App Engine / Cloud Functions Default Service Account: [project-id]@appspot.gserviceaccount.com
• Security Hole: When the Compute Engine API is activated, Google Cloud historically created the Compute Engine default service account and automatically bound it to the roles/editor primitive role. If a Dataflow worker or Dataproc VM runs under this default service account, any code executed on that node has near-total administrative control over the project.
• Remediation: Enforce the Organization Policy Constraint constraints/iam.automaticIamGrantsForDefaultServiceAccounts. In all pipeline configurations (Dataflow, Dataproc, Cloud Run, Cloud Composer), explicitly override the execution identity by specifying a custom user-managed service account.

3. Google-Managed Service Accounts (Service Agents)

• Naming Format: service-[project-number]@[service-name].iam.gserviceaccount.com (e.g., service-1234567890@gcp-sa-bigquery.iam.gserviceaccount.com or bq-[project-number]@bigquery-encryption.iam.gserviceaccount.com).
• Characteristics: Created and operated internally by Google Cloud services to execute asynchronous platform tasks on behalf of the customer (e.g., BigQuery reading data from Cloud Storage, Pub/Sub pushing messages to a webhook, or Cloud Storage communicating with Cloud KMS for customer-managed encryption).
• Usage: You do not manage credentials for service agents; however, you must frequently grant specific IAM roles to these service agents on external resources (such as granting roles/cloudkms.cryptoKeyEncrypterDecrypter to the Cloud Storage service agent).

---

5. Eliminating Static Credentials: Workload Identity Federation & Impersonation

Historically, authenticating workloads running outside of Google Cloud (such as an Apache Spark cluster in AWS EMR, an application in Microsoft Azure, or an on-premises Jenkins worker) required creating a service account, downloading a static JSON private key file, and embedding it into the external host.

The Operational Vulnerabilities of Service Account JSON Keys

• Key Sprawl and Exfiltration: Static JSON keys do not expire automatically. They are frequently leaked via code repositories, compromised build logs, or unencrypted local developer storage.
• Rotation Overhead: Organizations struggle to manually rotate cryptographic key pairs across hundreds of distributed external clients every 90 days without causing unplanned pipeline outages.
• Zero Context Awareness: Anyone possessing the private key file can authenticate from any IP address globally without multi-factor verification.

The Modern Standard: Workload Identity Federation

Workload Identity Federation allows external workloads running in AWS, Microsoft Azure, or any environment supporting OpenID Connect (OIDC) or SAML 2.0 (such as GitHub Actions, GitLab, or on-premises Kubernetes) to authenticate directly to Google Cloud without service account keys.

External Workload (AWS / GitHub Actions / Azure)
       │  1. Authenticates to external IdP & receives external token (e.g., AWS STS / OIDC JWT)
       ▼
Google Cloud Security Token Service (STS)
       │  2. Validates external token signature & claims against Workload Identity Pool
       │  3. Exchanges external token for short-lived, federated GCP STS token
       ▼
Cloud IAM Credentials API
       │  4. Exchanges federated STS token to impersonate a User-Managed Service Account
       ▼
Short-Lived GCP Access Token (OAuth 2.0, max 1 hour duration)
       │  5. Calls BigQuery, GCS, or Pub/Sub APIs with least-privilege permissions
       ▼
Google Cloud Data Resources

1. Workload Identity Pool: A managed container inside Google Cloud that organizes and manages external identity providers.
2. Workload Identity Provider: Defines the relationship with the external identity provider (e.g., GitHub Actions OIDC endpoint or AWS account ID).
3. Attribute Mapping & Conditions: Maps claims from the external token (e.g., repository, sub, account_id) to Google Cloud identity attributes, and applies security conditions (e.g., only allow requests originating from attribute.repository == 'enterprise/financial-etl').
4. Service Account Impersonation: The federated identity is granted roles/iam.workloadIdentityUser on the target Google Cloud service account, allowing it to impersonate the service account and obtain an ephemeral OAuth 2.0 access token.

Short-Lived Credentials via Service Account Impersonation

For human operators and internal automation scripts, Google Cloud provides direct service account impersonation using the IAM Credentials API (iamcredentials.googleapis.com).

Instead of downloading a key, an engineer or CI runner with the roles/iam.serviceAccountTokenCreator role on a target service account requests a short-lived access token:

# Generate an ephemeral OAuth 2.0 access token valid for 1 hour
gcloud auth print-access-token \
    --impersonate-service-account=dataflow-deployer@my-data-proj.iam.gserviceaccount.com

• The caller's human identity is recorded in Cloud Audit Logs alongside the impersonation event.
• The generated OAuth token has an automatic expiration (typically 60 minutes), completely eliminating dormant credential risk.

---

6. Attribute-Based Access Control via IAM Conditions

Standard IAM bindings grant unconditional access. IAM Conditions allow data architects to enforce Attribute-Based Access Control (ABAC) using Common Expression Language (CEL). Access is granted only when specific conditional expressions evaluate to true.

Key Use Cases in Data Systems

1. Temporary or Emergency Access Windows: Granting an on-call data engineer debugging access to production BigQuery tables that automatically expires at a fixed timestamp.
2. Resource Name and Path Filtering: Restricting Cloud Storage write operations so a pipeline service account can only upload objects to a specific partition prefix.
3. Tag-Based Governance: Granting data analysts access to BigQuery tables only if the resource has been tagged with Environment = Analytics-Public via Resource Manager tags.

// Example: Grant access only during business hours and expiring on October 1st, 2026
request.time < timestamp('2026-10-01T00:00:00Z') &&
request.time.getHours('America/New_York') >= 9 &&
request.time.getHours('America/New_York') <= 17

// Example: Restrict Cloud Storage object access to a specific bucket prefix
resource.type == 'storage.googleapis.com/Object' &&
resource.name.startsWith('projects/_/buckets/customer-lake/audit-logs/')

---

7. Data Exfiltration Defense: VPC Service Controls (VPC-SC)

A common security misunderstanding is assuming that strong IAM policies prevent data theft. Consider this realistic threat scenario:

• A data engineer has valid credentials and legitimate IAM permissions to read data from gs://internal-confidential-lake in project enterprise-prod.
• The engineer logs in, executes an authorized script, but directs the output to gs://attacker-controlled-bucket residing in a personal project outside the company.
• Because IAM checks permissions on each bucket independently, and the engineer possesses write access to their personal bucket, standard IAM evaluates both operations as valid. The data is exfiltrated.

[ Traditional IAM Boundary ]
Authorized User ──(IAM Allow)──> Reads Enterprise Data Lake
Authorized User ──(IAM Allow)──> Writes to Personal Bucket (DATA EXFILTRATED!)

[ VPC Service Controls Perimeter ]
Inside Perimeter: Enterprise Project (BigQuery, GCS, Dataproc)
                      │
                      ├──> Requests to external GCS buckets BLOCKED at API network boundary
                      └──> Ingress/Egress tightly governed by cryptographic Access Policies

How VPC Service Controls Works

VPC Service Controls creates a logical Service Perimeter around Google-managed services (BigQuery, Cloud Storage, Cloud Spanner, Vertex AI, Dataproc).

1. Network-Level API Enforcement: When a service is protected inside a perimeter, its public API endpoints can only be accessed from authorized VPC networks and authorized IP ranges defined by Access Context Manager access levels.
2. Exfiltration Prevention: Even if an authenticated user inside the perimeter possesses valid IAM write permissions to an external Cloud Storage bucket or BigQuery dataset, the VPC-SC perimeter blocks the request at the Google API edge if the destination resource resides outside the perimeter.
3. Ingress and Egress Rules: Secure cross-perimeter data sharing is achieved by defining explicit, cryptographically verifiable ingress and egress rules that specify the allowed source identity, originating project, destination service, and target resource.

---

8. Realistic Exam Scenarios & Architecture Pitfalls