12.1 Batch vs. Online Inference on Agent Platform
Key Takeaways
- Batch inference reads from Cloud Storage (JSON Lines, TFRecord, CSV, or file lists) or BigQuery and writes results to Cloud Storage or BigQuery without deploying the model to an endpoint.
- Batch inference jobs don't autoscale: they use starting_replica_count and ignore max_replica_count.
- Google recommends sizing batch jobs to run at least 10 minutes on CPUs and at least 20 minutes on GPUs, because replicas take about 5 and 10 minutes to start.
- For batch jobs, scaling out with more replicas of the smallest workable machine type improves throughput more predictably than using larger machines.
- Batch inference input, output, and the model must be in the same region or multi-region.
The exam guide lists deploying models for batch and online inference using appropriate services (for example, Agent Platform, Model Garden, Cloud Run, and GKE). This section covers Agent Platform Inference (formerly Vertex AI Prediction). Section 12.2 covers Model Garden, Cloud Run, and GKE.
Batch vs. Online at a Glance
| Dimension | Batch inference | Online inference |
|---|---|---|
| Pattern | Asynchronous job over a known dataset | Synchronous request → response |
| Endpoint needed | No. Request batch inference directly from the model | Yes. Deploy the model to an endpoint |
| Latency | Minutes to hours for the whole job | Milliseconds to seconds per request |
| Scaling | Fixed starting replica count for the job | Autoscaling between minimum and maximum replicas |
| Cost profile | Pay only while the job runs | Pay for deployed replicas the whole time the model is deployed |
| Typical uses | Nightly scoring, backfills, weekly propensity lists, offline evaluation | Fraud checks at checkout, recommendations on page load, chat |
Batch Inference
Inputs and outputs
| Input format | Notes |
|---|---|
| JSON Lines | One instance per line in Cloud Storage |
| TFRecord | Instances stored as TFRecords |
| CSV | Header row plus instances |
| File list | A list of Cloud Storage file URIs, such as images |
| BigQuery | A table as input. For registered BigQuery ML models, set the instance type to object |
Output goes to Cloud Storage or BigQuery. Agent Platform converts inputs to JSON instances before sending them to the container. For PyTorch prebuilt containers, each instance is wrapped in a data field because TorchServe's default handlers expect it. You can also filter or transform input fields (for example, exclude an ID column from the features while keeping it in the output).
Rules and limits
- Same location: input, output, and model must be in the same region or multi-region.
- No autoscaling: the data is partitioned across replicas at the start.
starting_replica_countis used andmax_replica_countis ignored. - Partitioning: Agent Platform automatically partitions BigQuery, file list, and JSON Lines input. It doesn't partition CSV, so Google advises against CSV for throughput-sensitive jobs. For TFRecord, split the data into at least as many files as replicas and pass a wildcard URI.
- Model load timeout: loading a model for batch inference times out after about 40 minutes. Very large models may need a smaller variant.
- Custom service accounts apply to the model server, not the batch client that reads and writes Cloud Storage and BigQuery.
- Monitoring: Model Monitoring can analyze batch inference inputs for drift (Chapter 19).
Sizing replicas
Google's guidance:
- Prefer more replicas of a small machine type over bigger machines. Throughput scales more linearly and predictably.
- Aim for jobs of at least 10 minutes on CPUs, because billing is per replica node hour and each replica takes about 5 minutes to start. GPU machines take about 10 minutes to start, so target at least 20 minutes.
- Rough starting points: tens of replicas for thousands of instances, hundreds for millions.
- Estimate: replicas ≈ N / (T × (60 / Tb)), where N is the number of batches, T is the target minutes, and Tb is seconds per batch.
Worked example: 1,000,000 instances at batch size 100 is 10,000 batches. With a 10-minute target and 1 second per batch, 10,000 / (10 × 60) ≈ 16.7, so about 17 replicas.
Other batch options
| Option | Best for |
|---|---|
BigQuery ML ML.PREDICT | Models living in BigQuery, SQL consumers |
| Gemini batch inference | Large prompt backlogs at a discount (Chapter 4) |
Dataflow RunInference | Batch or streaming pipelines that combine preprocessing and scoring |
Online Inference
Flow
- Upload the model to Model Registry with its serving container (prebuilt or custom).
- Create an endpoint (dedicated public, shared public, Private Service Connect, or private; Chapter 14).
- Deploy the model to the endpoint with a machine type, optional accelerators, and minimum and maximum replica counts.
- Send requests. The request body is
{"instances": [...], "parameters": {...}}, and the response is{"predictions": [...]}.
APIs
| API | Use |
|---|---|
predict | Standard JSON instances and predictions |
rawPredict | Send an arbitrary HTTP payload to the container, such as a non-standard format |
streamRawPredict | Streaming responses (dedicated and PSC endpoints), which suits gen AI |
explain | Predictions plus feature attributions, where configured |
What drives online latency
- Model size and hardware (Chapter 14)
- Feature retrieval before inference (Feature Store online serving)
- Pre- and postprocessing inside the container
- Network path (dedicated and private endpoints optimize it)
- Cold capacity: set minimum replicas so traffic never waits for new replicas to start
Hybrid Patterns
| Pattern | Example |
|---|---|
| Precompute + lookup | Batch-score product affinity nightly, store results in a low-latency store, and serve lookups online |
| Candidate generation + online ranking | A batch job retrieves top 500 items per user. An online model re-ranks them with session context |
| Online with batch fallback | If the endpoint is unavailable, serve yesterday's batch score |
Explanations and Monitoring for Each Mode
- Batch: you can request feature attributions in the batch job when the model is configured for explanations (keep the Explainable AI deprecation in mind, Chapter 18). Model Monitoring can analyze batch inputs for drift against training data.
- Online: enable request-response logging to BigQuery on the endpoint so predictions can be joined with later ground truth and monitored for skew and drift (Chapter 19).
Decision Checklist
- Are all inputs available before the prediction is needed? → Batch is possible.
- Is a fresh prediction needed within seconds of an event? → Online or streaming.
- Is traffic sparse and spiky? → Batch, or online with careful minimum replicas and scale-down.
- Is cost the main constraint and a delay acceptable? → Batch.
A team runs a batch inference job over 50 million records and sets starting_replica_count to 5 and max_replica_count to 200, expecting the job to scale out as it runs. What actually happens?
A retailer needs product-affinity scores for 20 million customers once a night for email campaigns, and nobody queries the scores in real time. Which approach is most cost-effective on Agent Platform?
A batch job must score 600,000 instances at batch size 100, each batch takes 2 seconds, and the team wants the job to finish in about 20 minutes. Using Google's estimation formula, about how many replicas should they start with?