4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE

4.1 Batch and Online Inference: Agent Platform, Model Garden, Cloud Run and GKE

In production machine learning systems, model serving represents the critical interface between trained mathematical artifacts and downstream business applications. Deploying models in Google Cloud requires ML engineers to balance stringent latency service level agreements (SLAs), throughput demands, infrastructure costs, and security perimeters. Google Cloud's Vertex AI Prediction suite offers two primary serving paradigms: Online Prediction for synchronous, real-time, low-latency inference, and Batch Prediction for asynchronous, high-throughput, offline scoring over massive datasets.

---

1. Vertex AI Online Prediction: Real-Time Serving Architecture

Vertex AI Online Prediction is engineered for interactive applications (e.g., e-commerce recommendation feeds, real-time fraud scoring, mobile app APIs) where client applications submit inference requests and expect immediate responses within milliseconds.

+---------------------------------------------------------------------------------------------------------+
|                                 VERTEX AI ONLINE PREDICTION TOPOLOGY                                    |
+---------------------------------------------------------------------------------------------------------+
|                                                                                                         |
|   [ Client App ]                                                                                        |
|          |                                                                                              |
|          | (HTTPS REST / gRPC)                                                                          |
|          v                                                                                              |
|   +-------------------------------------------------------------------------------------------------+   |
|   | Vertex AI Managed Endpoint (Single DNS / IP)                                                    |   |
|   |   - SSL Termination & Cloud IAM Authentication                                                   |   |
|   |   - Traffic Routing & Splitting Engine (e.g., 90% Model v1, 10% Model v2)                       |   |
|   +-------------------------------------------------------------------------------------------------+   |
|          |                                                                                              |
|          +-------------------------------+-------------------------------+                              |
|                                          |                               |                              |
|                                          v                               v                              |
|                         +---------------------------------+  +---------------------------------+        |
|                         | Deployed Model 1 (e.g., v1)     |  | Deployed Model 2 (e.g., v2)     |        |
|                         |  - Worker Pool: n1-standard-4   |  |  - Worker Pool: g2-standard-8   |        |
|                         |  - Accelerators: 1x NVIDIA T4   |  |  - Accelerators: 1x NVIDIA L4   |        |
|                         |  - Autoscaler: [2 .. 10 nodes]  |  |  - Autoscaler: [1 .. 4 nodes]   |        |
|                         +---------------------------------+  +---------------------------------+        |
+---------------------------------------------------------------------------------------------------------+

Core Mechanics of Managed Endpoints

To serve online predictions, trained models must be registered in the Vertex AI Model Registry and subsequently deployed to a Vertex AI Endpoint:

1. Decoupled Architecture: An Endpoint is a persistent, managed HTTPS gateway with a stable resource URI. A single Endpoint can host multiple deployed models simultaneously, enabling seamless traffic splitting, canary rollouts, and multi-model routing without altering client client-side API configurations.
2. Hardware & Accelerator Co-location: When deploying a model to an Endpoint, engineers specify the dedicated machine type (e.g., n1-standard-4, n1-highmem-8, a2-highgpu-1g) and optional hardware accelerators (e.g., NVIDIA T4, L4, V100, A100 GPUs). Vertex AI provisions dedicated compute instances where compute cores, memory, and accelerator cards are co-located in the same physical rack.
3. Autoscaling Configuration: Online endpoints support dynamic horizontal autoscaling governed by two mandatory parameters:
   • min_replica_count: The minimum number of compute nodes maintained at all times. Setting min_replica_count >= 1 is standard for production workloads to eliminate cold-start latency. (A standard deployment must keep min_replica_count >= 1, so dedicated nodes remain provisioned and billed continuously. A deployment can instead be enrolled in the separate Scale To Zero feature by setting min_replica_count = 0, at the cost of the first request after an idle period being rejected with 429 - Model is not yet ready for inference while the endpoint scales back up.)
   • max_replica_count: The maximum ceiling of compute nodes the autoscaler can provision under peak query-per-second (QPS) load.
   • Autoscaling Metric Target: The underlying horizontal pod autoscaler monitors average CPU utilization or GPU duty cycle (typically targeting 60%–80% utilization) and scales replicas up or down accordingly.
4. Private Endpoints and Enterprise Networking:
   • By default, Vertex AI Endpoints expose public HTTPS endpoints secured by Google Cloud IAM OAuth2 tokens.
   • For sensitive enterprise workloads requiring zero-trust network isolation, Vertex AI supports Private Endpoints deployed directly into a Virtual Private Cloud (VPC) via VPC Network Peering or Private Service Connect (PSC). This prevents inference traffic from traversing the public Internet, reduces network hops for sub-10ms latency SLAs, and complies with strict corporate egress policies.

---

2. Vertex AI Batch Prediction: High-Throughput Offline Serving

When inference requests do not require immediate sub-second responses, Batch Prediction offers an asynchronous, serverless, and highly cost-optimized alternative.

Batch Prediction Execution Model

Batch prediction is optimized for processing millions or billions of records in scheduled offline pipelines (e.g., nightly churn scoring, weekly marketing affinity updates, bulk catalog embedding generation):

+-----------------------+      Batch Prediction Job Spec       +------------------------+
|   Input Data Source   |  ==================================> | Vertex AI Worker Pool  |
| (Cloud Storage JSONL/ |    - Machine Type (e.g., c2-std-16)  | (Serverless Autoscaling|
|  CSV/TFRecord or BQ)  |    - Optional GPUs (e.g., 4x L4)     |  Dynamic Data Shards)  |
+-----------------------+    - Dedicated / Preemptible VMs     +------------------------+
                                                                            |
                                                                            | Direct Distributed Writes
                                                                            v
                                                               +------------------------+
                                                               |   Output Destination   |
                                                               | (Cloud Storage Bucket  |
                                                               |  or BigQuery Table)    |
                                                               +------------------------+

Key Architectural Differences from Online Serving

• Serverless Provisioning & Scale-to-Zero: Batch prediction jobs automatically provision temporary distributed worker clusters, shard input datasets across workers, execute inference in parallel, write results directly to the designated destination, and immediately terminate all compute resources. Users are billed strictly for the node-seconds consumed during execution.
• Input and Output Formats: Supports reading directly from BigQuery tables (via the BigQuery Storage Read API) and Cloud Storage files formatted as JSON Lines (.jsonl), CSV, or TensorFlow TFRecords (.tfrecord). Outputs are written back to BigQuery tables or partitioned GCS bucket prefixes.
• Preemptible and Spot VMs: To minimize compute costs for fault-tolerant offline jobs, batch prediction jobs can be configured to run on Preemptible or Spot VM instances, achieving up to 60%–91% cost savings compared to on-demand pricing.
• No Persistent Endpoint Required: Batch prediction jobs run directly against a registered model artifact in the Model Registry or a GCS model directory path without deploying to an active Vertex AI Endpoint.

---

3. Custom Prediction Routines (CPR) vs. Custom Serving Containers

When deploying trained models, ML engineers must choose the runtime environment that handles HTTP serialization, request payload parsing, feature transformations, and tensor execution.

+---------------------------------------------------------------------------------------------------------+
|                                 SERVING CONTAINER EXTENSION OPTIONS                                     |
+------------------------------------+------------------------------------+-------------------------------+
|        PRE-BUILT CONTAINERS        |  CUSTOM PREDICTION ROUTINES (CPR)  |   CUSTOM SERVING CONTAINERS   |
+------------------------------------+------------------------------------+-------------------------------+
| * Standard TF, PyTorch, Scikit,    | * Custom Python pre/post logic     | * Complete control over OS,   |
|   XGBoost serving images           | * User implements Predictor class  |   web server, C++ runtimes    |
| * Fast deployment, zero Docker     | * Google builds container image    | * Supports ONNX, TensorRT,    |
| * Rigid input/output expectations  | * Balance of speed and flexibility |   Triton, C++ dependencies    |
+------------------------------------+------------------------------------+-------------------------------+

1. Pre-built Prediction Containers

Google Cloud provides fully managed, optimized container images for popular ML frameworks (TensorFlow Serving, PyTorch TorchServe, Scikit-learn, XGBoost). These containers require strict adherence to standard model serialization formats (e.g., TensorFlow SavedModel, Scikit model.joblib, XGBoost model.bst) and expect JSON payloads matching standard signature definitions.

2. Custom Prediction Routines (CPR)

In real-world applications, raw client request payloads often require feature transformations (e.g., tokenizing text, scaling numerical inputs, looking up categorical embeddings) before reaching the model, and raw model logits require post-processing (e.g., softmax calibration, JSON formatting).

Custom Prediction Routines (CPR) bridge the gap between pre-built containers and custom Dockerfiles. CPR allows engineers to define a Python class extending the Predictor interface with custom load(), preprocess(), predict(), and postprocess() methods:

# Custom Prediction Routine (CPR) Implementation
import joblib
import numpy as np
from google.cloud.aiplatform.prediction.predictor import Predictor

class CustomChurnPredictor(Predictor):
    def __init__(self):
        self._model = None
        self._scaler = None

    def load(self, artifacts_uri: str) -> None:
        """Loads model weights and preprocessing artifacts from Cloud Storage."""
        self._model = joblib.load(f"{artifacts_uri}/model.joblib")
        self._scaler = joblib.load(f"{artifacts_uri}/scaler.joblib")

    def preprocess(self, prediction_input: dict) -> np.ndarray:
        """Parses raw JSON payload and applies feature normalization."""
        raw_instances = prediction_input["instances"]
        features = np.array([[item["tenure"], item["monthly_charges"]] for item in raw_instances])
        scaled_features = self._scaler.transform(features)
        return scaled_features

    def predict(self, instances: np.ndarray) -> np.ndarray:
        """Executes model inference."""
        return self._model.predict_proba(instances)

    def postprocess(self, prediction_results: np.ndarray) -> dict:
        """Formats probabilities into client-ready response payload."""
        probabilities = prediction_results[:, 1].tolist()
        return {"predictions": [{"churn_probability": p, "risk_level": "HIGH" if p > 0.7 else "LOW"} for p in probabilities]}

Vertex AI SDK compiles this CPR class into an optimized container image automatically, pushing it to Artifact Registry without requiring manual Dockerfile maintenance.

3. Custom Serving Containers: The HTTP Contract Specification

For complex architectures requiring non-Python runtimes, specialized C++ libraries, NVIDIA Triton Inference Server, TensorRT optimization, or gRPC streaming, engineers build Custom Serving Containers. To run reliably on Vertex AI Prediction, the custom container must strictly adhere to the Vertex AI Serving HTTP Contract:

1. Listening Port: The web server must listen for HTTP requests on 0.0.0.0 at the port defined by the AIP_HTTP_PORT environment variable (defaults to port 8080).
2. Health Check Endpoint (GET /health or GET /v1/models/{model_name}):
   • Vertex AI routinely sends HTTP GET requests to verify container responsiveness.
   • The server must return an HTTP status code 200 OK with an empty or diagnostic body.
   • If the container fails to return 200 OK within the configured timeout (typically 4–10 seconds), Vertex AI restarts the container replica.
3. Prediction Endpoint (POST /predict or POST /v1/models/{model_name}:predict):
   • Receives client prediction requests containing a JSON body structured with an "instances" array.
   • Must return an HTTP status code 200 OK containing a JSON body with a "predictions" array.
4. Environment Variables Injected by Vertex AI:
   • AIP_STORAGE_URI: The Cloud Storage path where the model artifacts reside (e.g., gs://my-bucket/models/v1).
   • AIP_HTTP_PORT: The network port on which the web server must bind (e.g., 8080).
   • AIP_HEALTH_ROUTE: The HTTP path for health checks (e.g., /health).
   • AIP_PREDICT_ROUTE: The HTTP path for prediction requests (e.g., /predict).

[!IMPORTANT] Payload Size Constraints: Vertex AI Online Prediction enforces a maximum request payload size of 1.5 MB for standard REST predictions and 10 MB for Vertex AI standard RPC/gRPC calls. For payloads exceeding these limits (e.g., high-resolution video streams, multi-gigabyte point clouds), client applications must upload the raw data to Cloud Storage and pass the GCS URI in the prediction request, or utilize Batch Prediction.

---

4. Multi-Model Endpoints vs. Dedicated Endpoints

When managing a fleet of production models, ML architects must choose between deploying models to separate dedicated endpoints or co-locating them on a single multi-model endpoint:

---

5. Architectural Decision Matrix: GCP Model Serving Platforms

Selecting the optimal serving infrastructure across Google Cloud Platform is heavily tested on the Professional ML Engineer certification: