12.2 Serving Beyond Agent Platform Endpoints: Model Garden, Cloud Run & GKE
Key Takeaways
- Model Garden serves Google models through managed APIs, partner and open models through MaaS, and open or custom-weight models through one-click self-deployment to dedicated endpoints.
- Cloud Run services can attach one NVIDIA L4 (24 GB) or RTX PRO 6000 Blackwell (96 GB) GPU per instance, and GPU instances can scale to zero.
- Cloud Run GPU instances with preinstalled drivers start in about 5 seconds, which suits spiky or low-traffic inference.
- GKE inference supports model servers such as vLLM, TensorFlow Serving, and Triton, with GKE Inference Gateway, Horizontal Pod Autoscaling, and Inference Quickstart.
- Choose GKE when you need Kubernetes-level control, portability, or shared accelerator pools, and Agent Platform endpoints when you want managed ML serving features such as Model Registry versions and Model Monitoring.
The exam guide names Agent Platform, Model Garden, Cloud Run, and GKE as serving options. Picking among them depends on the model source, traffic pattern, operations appetite, and the ML-specific features you need.
Serving Through Model Garden
| Model source | Serving path | Operations |
|---|---|---|
| Google models (Gemini, Veo, embeddings) | Managed API with PayGo, Provisioned Throughput, or batch | None |
| Partner and open models via MaaS | Serverless API ("API Service" model cards) | None |
| Open models with your weights or tuning | One-click self-deployment to a dedicated endpoint in your project, using prebuilt serving containers (vLLM, Hex-LLM, SGLang, TGI, TensorRT-LLM) or a custom vLLM container | You choose machine type and accelerators and pay for compute while deployed |
Self-deployed endpoints support Agent Platform features such as dedicated or private networking and the Chat Completions API for Model Garden models. Model Garden also offers batch inference for supported models. Before deploying, use the SDK's list_deploy_options() to see verified machine and accelerator configurations.
Serving on Cloud Run
Cloud Run runs stateless containers with request-based autoscaling, and it can scale to zero.
| Capability | Detail |
|---|---|
| GPUs | NVIDIA L4 (24 GB) or RTX PRO 6000 Blackwell (96 GB). One GPU per instance |
| Minimum resources | L4: at least 4 CPU and 16 GiB memory. RTX PRO 6000: at least 20 CPU and 80 GiB |
| Startup | Instances with GPU drivers preinstalled start in about 5 seconds |
| Scaling | On-demand GPUs with no reservation needed, and scale to zero when idle |
| Regions | Limited to specific regions for each GPU type |
Cloud Run fits well when:
- Traffic is spiky or low, and idle cost should be near zero.
- The model fits on one GPU or runs on CPU (small LLMs, embedding models, scikit-learn, XGBoost).
- The team already deploys containerized web services and wants the same workflow for models.
- You need a custom HTTP API around the model (authentication, business logic).
Trade-offs: you build and maintain the serving container, and cold starts include loading model weights (so keep images and weights small, or load weights from Cloud Storage efficiently). Cloud Run can split traffic between revisions, but ML-specific features such as built-in Model Monitoring, Model Registry version deployment, and explanations aren't provided the way Agent Platform endpoints provide them. You manage model versioning yourself.
Serving on GKE
GKE runs model servers as Kubernetes Deployments.
| Component | Role |
|---|---|
| Model servers | vLLM, JetStream, TensorFlow Serving, Triton Inference Server, or a custom Python server |
| GKE Inference Gateway | Load balancing and routing tailored to gen AI inference |
| Horizontal Pod Autoscaler (HPA) | Scale pods on CPU, latency, or custom metrics |
| GKE Inference Quickstart | Recommended accelerators, model servers, scaling, and storage configurations based on performance and cost goals |
| Autopilot vs. Standard | Autopilot manages nodes, including accelerators. Standard gives full node-pool control |
| Ecosystem | Kueue for queueing, Ray for distributed compute, LeaderWorkerSet for multi-host serving, LoRA adapters |
GKE supports models from about 8 billion to 671 billion parameters on GPUs and TPUs, with dashboards for popular servers such as vLLM.
GKE fits well when:
- The organization runs Kubernetes already and wants one platform for apps and models.
- Portability across environments matters.
- You need fine-grained control: custom schedulers, multi-host serving of very large models, or sharing accelerator pools across many models.
- Very high, steady volume makes hand-tuned serving cost-effective.
Trade-offs: you operate clusters, upgrades, security, and scaling policies.
Choosing a Serving Platform
| Requirement | Best fit |
|---|---|
| Managed ML serving with model versions, traffic splits, Model Monitoring, and explanations | Agent Platform endpoints |
| Use Gemini or a partner/open model with no infrastructure | Model Garden API or MaaS |
| Serve your own weights for an open LLM with managed endpoint features | Model Garden self-deployment |
| Spiky traffic, scale to zero, single-GPU or CPU models, container-native team | Cloud Run |
| Kubernetes-standardized org, portability, shared GPU/TPU pools, multi-host giant models | GKE |
| Batch scoring of data in BigQuery | BigQuery ML or Agent Platform batch inference |
Operational Checklist for Any Serving Platform
| Concern | Agent Platform endpoint | Cloud Run | GKE |
|---|---|---|---|
| Versioned rollout | Traffic split across deployed models | Revision traffic split | Deployment strategies or Gateway routing |
| Autoscaling | Min/max replicas with CPU, GPU, and request metrics | Request concurrency, scale to zero | HPA on custom metrics |
| Private access | Private Service Connect or private endpoints | Internal ingress, VPC connectors | Internal load balancers |
| Prediction logging for monitoring | Built-in request-response logging | Build it yourself | Build it yourself |
| Model Monitoring | Built in | Custom | Custom |
Worked Scenarios
- An internal document classifier gets 200 requests a day during business hours, with a small fine-tuned transformer on one L4. Cloud Run with an L4 GPU scales to zero overnight, so idle cost is minimal.
- A bank serving a fraud model needs canary rollouts between versions, drift monitoring, and private networking. Agent Platform endpoint (Private Service Connect) with traffic splitting and Model Monitoring.
- A platform team serves 40 open LLMs and adapters to many product teams on shared H100 pools with custom routing. GKE with vLLM, Inference Gateway, and HPA.
- A startup wants a Llama chat model with no GPU management. MaaS from Model Garden.
An internal tool calls a small fine-tuned transformer about 150 times a day, only during business hours, and leadership wants near-zero cost when idle. The model fits on one L4 GPU. Which serving option fits best?
A platform team must serve dozens of open LLMs and LoRA adapters for many product teams on shared GPU pools, with portable Kubernetes manifests and custom routing. Which platform fits best?
A company wants to use a partner LLM from Model Garden for a new feature with no GPU provisioning and pay-per-use billing. Which serving path should it choose?