4.1 Data Pipeline Planning: Sources, Sinks, and Network Architecture (VPC, PSC, Interconnect)

4.1 Data Pipeline Planning: Sources, Sinks, and Network Architecture (VPC, PSC, Interconnect)

Exam Focus: The Google Cloud Professional Data Engineer exam rigorously tests your ability to design secure, high-throughput network topologies for data ingestion. You must master the architectural differences between Shared VPC and VPC Peering, configure Private Google Access (PGA) and DNS steering for VPC Service Controls, implement Private Service Connect (PSC) to overcome RFC 1918 overlapping subnets, evaluate hybrid ingestion bandwidth across Dedicated/Partner Cloud Interconnect and Cloud HA VPN, and properly size worker egress and firewall rules for distributed compute engines like Cloud Dataflow and Cloud Dataproc.

Data ingestion is the critical circulatory system of any modern analytics platform. Before a single transformation runs in Apache Beam or a single query executes in BigQuery, data must traverse network perimeters separating operational databases, SaaS applications, multi-cloud object stores, and edge IoT devices from Google Cloud storage repositories. A poorly designed network topology introduces latency bottlenecks, escalates cross-region egress expenses, creates compliance vulnerabilities, and causes silent pipeline failures. Designing robust data ingestion pipelines demands rigorous planning across data sources, destination sinks, and underlying Google Cloud networking primitives.

---

1. Heterogeneous Data Sources and Analytical Sinks

Enterprise data architectures must ingest data from diverse origins, each imposing distinct protocol, security, and latency requirements:

+-----------------------------------------------------------------------------------------+
|                              INGESTION SOURCE TAXONOMY                                  |
+-----------------------------+-----------------------------+-----------------------------+
| On-Premises & Colocation    | SaaS & Webhooks             | Multi-Cloud & Edge IoT      |
| - Oracle, SQL Server, DB2   | - Salesforce, Workday, SAP  | - AWS S3, Azure ADLS Gen2   |
| - Mainframe flat files      | - REST APIs, Webhooks       | - Telemetry, MQTT sensors   |
| - Latency-sensitive OLTP    | - Rate-limited, pagination  | - Terabyte-scale egress     |
+-----------------------------+-----------------------------+-----------------------------+
                                             │
                                             ▼
+-----------------------------------------------------------------------------------------+
|                           ENTERPRISE NETWORK TRANSIT FABRIC                             |
|  Shared VPC  │  Private Google Access  │  Private Service Connect  │  Cloud Interconnect  |
+-----------------------------------------------------------------------------------------+
                                             │
                                             ▼
+-----------------------------------------------------------------------------------------+
|                              TARGET ANALYTICAL SINKS                                    |
|  Cloud Storage (Raw Lake)   │  BigQuery (Warehouse)  │  Cloud Bigtable (Time-Series)     |
|  Cloud Spanner (Global OLTP)│  Memorystore (Cache)   │  Vertex AI Feature Store (ML)    |
+-----------------------------------------------------------------------------------------+

Ingestion Source Categories

1. On-Premises Relational Databases & Mainframes: Enterprise transaction backbones (Oracle Exadata, Microsoft SQL Server, IBM DB2) running in corporate datacenters. Challenges include strict firewall perimeters, non-routable private networks, change log retention constraints, and operational read lock contention.
2. SaaS Platforms and Cloud Business Applications: Modern enterprise applications (Salesforce, ServiceNow, Marketo, Workday, Stripe) emitting data via REST APIs, GraphQL endpoints, and real-time event webhooks. Ingestion requires secure webhook ingress endpoints, authentication token refresh lifecycles, and resilient rate-limit backoff logic.
3. Multi-Cloud Repositories: High-volume object stores in competing clouds (Amazon S3, Azure Data Lake Storage Gen2) hosting curated data assets. Ingestion patterns utilize Google Cloud Storage Transfer Service, Workload Identity Federation, and dedicated cross-cloud interconnects to bypass public transit.
4. Edge Telemetry, Connected Vehicles, and IoT: Fleets of distributed physical hardware (industrial SCADA sensors, connected vehicles, smart meters) generating millions of lightweight events per second over MQTT, HTTP/2, and WebSockets funneling into Google Cloud Pub/Sub.

Analytical Sinks

• Cloud Storage (Raw & Curated Lakehouse): The cost-effective landing zone for immutable raw files (Avro, Parquet, JSON, CSV). Supports multi-tiered lifecycle management (Standard -> Nearline -> Coldline -> Archive).
• Google BigQuery: Serverless, highly scalable enterprise data warehouse. Ingestion targets include native managed tables (via batch load jobs or Storage Write API) and external BigLake tables.
• Cloud Bigtable: Ultra-low-latency NoSQL wide-column database engineered for massive ingestion throughput (>100,000 QPS) and sub-10ms point reads. Ideal for real-time fraud scoring, time-series telemetry, and IoT stream ingestion.
• Cloud Spanner: Horizontally scalable, globally distributed relational database guaranteeing external consistency (ACID) via TrueTime. Target for consolidated multi-region operational ledgers.

---

2. Google Cloud VPC Network Topology: Shared VPC vs. VPC Peering

Google Cloud Virtual Private Cloud (VPC) networks are global software-defined networks spanning all Google Cloud regions worldwide. Subnets are regional resources carved out of RFC 1918 private address spaces (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16).

[!IMPORTANT] The Default VPC automatically provisions an auto-mode network with default subnets in every Google Cloud region, pre-populating permissive firewall rules (such as default-allow-internal and default-allow-ssh from 0.0.0.0/0). Production data engineering environments must strictly avoid Default VPCs. Always build Custom Mode VPCs with explicit CIDR allocations, deterministic subnet boundaries, and zero-trust firewall configurations.

When organizing enterprise data pipelines across multiple Google Cloud projects, data architects must choose between Shared VPC and VPC Network Peering:

+-----------------------------------------------------------------------------------------+
|                               SHARED VPC ARCHITECTURE                                   |
+-----------------------------------------------------------------------------------------+
|  HOST PROJECT (Managed by Central Network & Security Operations)                       |
|   - Central Shared VPC Network ('vpc-enterprise-prod')                                  |
|   - Subnet 1: 'dataflow-workers-uscentral1' (10.10.10.0/24)                             |
|   - Subnet 2: 'dataproc-clusters-uscentral1' (10.10.20.0/24)                            |
|   - Cloud Interconnect Attachments, Cloud Routers, Cloud NAT Gateways                   |
|   - Centralized Network Security & Firewall Rules                                      |
+-----------------------------------------------------------------------------------------+
          │                                                 │
          │ IAM: roles/compute.networkUser                  │ IAM: roles/compute.networkUser
          ▼                                                 ▼
+-----------------------------------+     +-----------------------------------------------+
| SERVICE PROJECT A                 |     | SERVICE PROJECT B                             |
| ('pipeline-ingest-prod')          |     | ('analytics-warehouse-prod')                  |
|  - Cloud Dataflow Workers         |     |  - Cloud Dataproc Spark Clusters              |
|  - Cloud Composer DAGs            |     |  - BigQuery Datasets & BigLake Tables         |
|  * Workers attach vNICs directly  |     |  * Workers attach vNICs directly              |
|    to Host Project Subnet 1       |     |    to Host Project Subnet 2                   |
+-----------------------------------+     +-----------------------------------------------+

Shared VPC (The Enterprise Standard)

Shared VPC enables an enterprise to connect resources from multiple projects to a common Virtual Private Cloud network, allowing them to communicate with each other securely and efficiently using internal IP addresses.

• Host Project: An administratively isolated project containing one or more Shared VPC networks, subnets, route tables, Cloud NAT gateways, Interconnect attachments, and firewall rules. Managed strictly by the central network engineering team.
• Service Projects: Workload projects attached to the host project by a Shared VPC Administrator. Data engineers launch compute resources (Dataflow worker pools, Dataproc clusters, Cloud Composer GKE nodes) inside service projects, but their virtual network interfaces (vNICs) are instantiated directly inside the host project's subnets.
• IAM Delegation and Least Privilege: Network administrators grant the roles/compute.networkUser role to the service project's service accounts (e.g., service-[SERVICE_PROJ_NUM]@dataflow-service-producer-prod.iam.gserviceaccount.com and the custom Dataflow worker service account) on specific host subnets. Data engineers can consume network capacity without possessing permissions to modify firewall rules, altering routing tables, or peering new networks.

VPC Network Peering

VPC Network Peering connects two independent VPC networks so that resources in each network can communicate with each other via internal IP addresses without public internet transit or intermediate NAT gateways.

• Decentralized and Autonomous: Each project team maintains full administrative control over their respective VPC, routing tables, and firewall configurations.
• Non-Transitive Routing: VPC Peering is strictly non-transitive. If VPC A is peered with VPC B, and VPC B is peered with VPC C, VPC A cannot communicate with VPC C through VPC B. To enable communication, an explicit peering between A and C must be established.
• Prohibition of Overlapping CIDRs: VPC Peering cannot be established if any subnet CIDR ranges overlap between the two VPC networks.

---

3. Private Google Access (PGA) and DNS Steering

In hardened enterprise environments, compute instances operating within data pipelines (Dataflow workers, Dataproc nodes, Compute Engine ingestion proxies) are explicitly configured without external public IP addresses (--no-use-public-ips). This eliminates public internet attack vectors. However, distributed pipelines must constantly communicate with Google APIs—reading data from Cloud Storage (storage.googleapis.com), loading tables into BigQuery (bigquery.googleapis.com), and pulling streaming messages from Pub/Sub (pubsub.googleapis.com).

How Private Google Access Works

Private Google Access (PGA) enables virtual machine instances and pipeline workers that possess only internal RFC 1918 private IP addresses to communicate privately with Google APIs and services.

• Subnet Configuration: PGA is toggled at the individual subnet level:

  gcloud compute networks subnets update dataflow-worker-subnet \
      --region=us-central1 \
      --enable-private-ip-google-access
  

• Internal Routing: When a worker VM initiates an HTTPS request to storage.googleapis.com, the packet is routed through the default internet gateway route (0.0.0.0/0). However, Google's software-defined network (Andromeda) intercepts packets destined for Google public VIPs and keeps the traffic entirely within Google's private network fabric, never routing packets across the public internet.

Private Google Access Domains: Default vs. Private vs. Restricted

Depending on compliance and security requirements, organizations route Google API traffic through different Virtual IP (VIP) ranges:

+-----------------------------------------------------------------------------------------+
|                        RESTRICTED GOOGLE APIS DNS STEERING                              |
+-----------------------------------------------------------------------------------------+
| Dataflow Worker VM (Private IP: 10.10.10.15, No Public IP)                              |
|  │                                                                                      |
|  │ 1. DNS Query: Resolve 'bigquery.googleapis.com'                                      |
|  ▼                                                                                      |
| Cloud DNS Private Response Policy / Managed Zone ('googleapis.com')                     |
|  │                                                                                      |
|  │ 2. CNAME: *.googleapis.com -> restricted.googleapis.com                             |
|  │ 3. A Records: restricted.googleapis.com -> 199.36.153.8, .9, .10, .11               |
|  ▼                                                                                      |
| Resolves to 199.36.153.8                                                                |
|  │                                                                                      |
|  │ 4. HTTPS API Request routed to 199.36.153.8                                          |
|  ▼                                                                                      |
| Custom Route: 199.36.153.8/30 -> Next Hop: Default Internet Gateway                    |
|  │                                                                                      |
|  ▼                                                                                      |
| Google Andromeda Fabric (VPC Service Controls Perimeter Verified)                      |
|  └── Successfully accesses BigQuery Dataset inside Enterprise Perimeter                 |
+-----------------------------------------------------------------------------------------+

Enforcing Restricted VIPs for Data Exfiltration Prevention

When enforcing VPC Service Controls (VPC-SC) perimeters around BigQuery and Cloud Storage, routing requests to default public IP addresses is blocked. To enforce the restricted VIP:

1. Cloud DNS Configuration: Create a private Cloud DNS zone for googleapis.com with a CNAME record mapping *.googleapis.com to restricted.googleapis.com., and create four A records mapping restricted.googleapis.com. to 199.36.153.8, 199.36.153.9, 199.36.153.10, and 199.36.153.11.
2. Custom Routing: In the VPC network, configure a custom route for destination 199.36.153.8/30 with the next hop set to the default-internet-gateway.
3. Result: All API requests originating from private VMs are steered deterministically to the restricted VIP, ensuring strict VPC Service Controls perimeter compliance.

---

4. Private Service Connect (PSC)

While VPC Peering and Private Google Access address many connectivity requirements, they present notable operational limitations: VPC Peering cannot handle overlapping RFC 1918 subnets, introduces routing quota ceilings, and exposes entire network topologies. Private Service Connect (PSC) is Google Cloud's modern, software-defined private connectivity framework.

How Private Service Connect Operates

PSC allows consumers to connect privately to services in another VPC (whether operated by Google, a third-party SaaS vendor like Snowflake or MongoDB Atlas, or an internal enterprise producer) using private endpoints (forwarding rules) deployed directly within the consumer's own subnet.

+-----------------------------------------------------------------------------------------+
|                        PRIVATE SERVICE CONNECT (PSC) ARCHITECTURE                       |
+-----------------------------------------------------------------------------------------+
| CONSUMER VPC ('vpc-analytics-prod')          | PRODUCER VPC ('vpc-saas-partner')        |
|                                              |                                          |
| Subnet: 'analytics-subnet' (10.100.0.0/24)   | Subnet: 'producer-subnet' (10.100.0.0/24)|
|                                              |  [Identical RFC 1918 CIDR Overlap!]      |
|  [ Dataflow Worker VM ]                      |                                          |
|         │                                    |                                          |
|         ▼ (Internal Request)                 |                                          |
|  [ PSC Forwarding Rule / Endpoint ]          |                                          |
|    Internal IP: 10.100.0.50                  |                                          |
|         │                                    |                                          |
|         └────────────── Google Transit Fabric ──────────────────┐                       |
|                         (Software-Defined Geneve Tunnel)        ▼                       |
|                                              | [ Service Attachment ]                   |
|                                              |   NAT Subnet: 172.16.0.0/24 (SNAT)       |
|                                              |         │                                |
|                                              |         ▼                                |
|                                              | [ Internal Load Balancer (ILB) ]         |
|                                              |         │                                |
|                                              |         ▼                                |
|                                              | [ Producer Database / Analytics Engine ] |
+-----------------------------------------------------------------------------------------+

Key Architectural Superpowers of PSC for Data Pipelines

1. Zero CIDR Conflict (Handles Overlapping Subnets): The consumer VPC and the producer VPC can both use the identical RFC 1918 CIDR block (e.g., 10.100.0.0/24). On the producer side, the Service Attachment uses a dedicated NAT Subnet to perform Source Network Address Translation (SNAT), completely decoupling the consumer's IP space from the producer's IP space.
2. Strictly Unidirectional Access: Traffic can only flow from the consumer endpoint to the producer's published service attachment. The producer has zero visibility into, and cannot initiate connections back into, the consumer VPC.
3. Granular Service-Level Segmentation: Instead of peering two entire networks and managing complex firewall rules, PSC exposes only a single, specific internal load-balanced service.
4. PSC for Google APIs: Data engineers can deploy a PSC endpoint with a dedicated internal IP address (e.g., 10.10.10.50) to access all Google APIs. By mapping *.p.googleapis.com to this local IP, all pipeline API traffic travels through a local consumer endpoint without needing default internet gateway routes.

---

5. Hybrid and Cross-Cloud Connectivity: Cloud Interconnect vs. Cloud HA VPN

Ingesting petabytes of historical data or streaming continuous transactional deltas from on-premises corporate datacenters into Google Cloud requires reliable, high-bandwidth hybrid connectivity. The primary choices are Cloud Interconnect and Cloud HA VPN:

+-----------------------------------------------------------------------------------------+
|                    HYBRID CONNECTIVITY: INTERCONNECT VS. HA VPN                         |
+-----------------------------------------------------------------------------------------+
| DEDICATED INTERCONNECT (Physical Cross-Connect)                                         |
| On-Premises Router ──(10G / 100G Direct Fiber)──> Google Edge Colocation ──> Google VPC |
| * Features: Dedicated physical circuits, lowest deterministic latency, 99.99% SLA       |
+-----------------------------------------------------------------------------------------+
| PARTNER INTERCONNECT (Service Provider Transit)                                         |
| On-Premises Router ──(VLAN Attachment)──> Equinix / Megaport ──> Google VPC             |
| * Features: Sub-10G to 50G, for facilities without physical Google Colocation presence  |
+-----------------------------------------------------------------------------------------+
| CLOUD HA VPN (IPsec over Public Internet)                                               |
| On-Premises VPN ──(IPsec Tunnels via BGP, max 3 Gbps/tunnel)──> Cloud HA VPN Gateway    |
| * Features: Fast setup, encrypted over public internet, throughput capped              |
+-----------------------------------------------------------------------------------------+

Dedicated Interconnect

• Physical Direct Link: Enterprise provisions physical fiber connections (10 Gbps or 100 Gbps circuits) directly into a Google Cloud Colocation Facility (Point of Presence / PoP).
• Highest Throughput and Deterministic Latency: Bypasses the public internet entirely. Ideal for continuous replication of high-volume OLTP databases and high-frequency analytical feeds.
• Availability Topologies:
  • 99.9% Availability (Non-Critical): Requires two single connections to two distinct edge routers in a single colocation facility in a single metropolitan area.
  • 99.99% Availability (Production Mission-Critical Standard): Requires at least four connections across two independent colocation facilities (two distinct metros) connected to two distinct Cloud Routers in paired Google Cloud regions.

Partner Interconnect

• Provider-Mediated Transit: Connects on-premises datacenters to Google Cloud through a supported third-party telecommunications partner (e.g., Equinix Fabric, Megaport, AT&T NetBond).
• Bandwidth Granularity: Capacities range from 50 Mbps up to 50 Gbps per VLAN attachment.
• Selection Criteria: The definitive choice when the enterprise data center is not physically co-located in a Google colocation facility or when bandwidth requirements do not justify a dedicated 10 Gbps/100 Gbps physical circuit.

Cloud HA VPN

• Encrypted Internet Transit: Establishes IPsec VPN tunnels between an on-premises VPN gateway and a Google Cloud HA VPN gateway over public internet infrastructure.
• High Availability Topology: Cloud HA VPN provides an SLA of 99.99% availability by requiring two active interfaces (Interface 0 and Interface 1) configured with dynamic routing via Border Gateway Protocol (BGP).
• Bandwidth Limits: Each HA VPN tunnel is hard-capped at 3 Gbps (approximately 250,000 packets per second). Achieving higher throughput requires deploying multiple parallel gateways and ECMP (Equal-Cost Multi-Path) routing, but aggregate throughput rarely scales efficiently beyond 10–15 Gbps.

Bandwidth Ingestion Sizing Equation

When planning network infrastructure for hybrid data ingestion, calculate required bandwidth based on data volume, ingestion window, and transport overhead:

$\text{Required Bandwidth (Gbps)} = \frac{\text{Data Volume (Bits)} \times (1 + \text{Overhead})}{\text{Ingestion Window (Seconds)}}$

$\text{Example: } 50\text{ TB daily migration in a 6-hour overnight window with 20\% protocol overhead:}$

$\text{Volume} = 50 \times 10^{12} \times 8\text{ bits} = 400 \times 10^{12}\text{ bits} = 400,000\text{ Gb}$

$\text{Window} = 6\text{ hours} \times 3,600\text{ seconds} = 21,600\text{ seconds}$

$\text{Required Bandwidth} = \frac{400,000}{21,600} \times 1.20 \approx 22.2\text{ Gbps}$

Architectural Conclusion: A 3 Gbps Cloud HA VPN cannot meet this SLA. The architecture mandates either a 50 Gbps Partner Interconnect or Dedicated Interconnect.

---

6. Worker Egress Caps, Cloud NAT, and Firewall Rules

Distributed data processing engines (Dataflow, Dataproc) deploy fleets of virtual machine worker instances that perform heavy network I/O during pipeline execution.

Compute Engine & Dataflow Egress Bandwidth Caps

• Virtual machine network throughput is bounded by the instance's vCPU allocation and networking configuration. Standard VMs receive 2 Gbps per vCPU up to a maximum of 10–16 Gbps.
• For bandwidth-intensive ingestion pipelines shuffling hundreds of gigabytes across workers, selecting instances with higher vCPU counts or configuring Tier 1 High Bandwidth Networking (unlocking 50 to 100 Gbps per VM) is critical to prevent network throttling.

Cloud NAT for Outbound Internet Egress

• When Dataflow workers run with --no-use-public-ips, Private Google Access allows them to reach Google APIs privately. However, if the pipeline must pull data from an external third-party SaaS REST API or an external public web service, Private Google Access cannot route that traffic.
• Solution: Deploy Cloud NAT (Network Address Translation) on the Cloud Router serving the worker subnet. Cloud NAT provides managed, highly available outbound internet connectivity for instances lacking external IPs without allowing unsolicited inbound connections from the internet.

# Deploy Cloud NAT for private Dataflow worker internet egress
gcloud compute routers create nat-router \
    --network=vpc-data-prod \
    --region=us-central1

gcloud compute routers nats create worker-nat-gateway \
    --router=nat-router \
    --region=us-central1 \
    --auto-allocate-nat-external-ips \
    --nat-all-subnet-ip-ranges

Essential Firewall Rules for Dataflow and Dataproc Clusters

Distributed pipelines communicate internally between worker nodes during shuffle and coordination phases. Misconfigured firewall rules are the leading cause of pipeline execution timeouts:

• Dataflow Internal Shuffle Communication: When Dataflow executes without Streaming Engine (or during heavy batch operations), workers communicate directly across TCP ports 12345 and 12346. The VPC must have an ingress firewall rule allowing internal TCP traffic on ports 12345-12346 between worker instances tagged with dataflow.
• Dataproc Internal Communication: Apache Spark and Hadoop require open ingress across all internal cluster nodes for HDFS NameNode/DataNode communication (ports 8020, 9870, 9866) and YARN resource management (ports 8088, 8030-8032).

# Firewall rule allowing internal shuffle communication for Dataflow workers
gcloud compute firewall-rules create allow-dataflow-internal-shuffle \
    --network=vpc-data-prod \
    --action=ALLOW \
    --direction=INGRESS \
    --target-tags=dataflow \
    --source-tags=dataflow \
    --rules=tcp:12345-12346

---

7. Concrete Exam Scenarios & Architecture Pitfalls