12.3 Prebuilt & Custom Serving Containers

The exam guide lists packaging and serving models from different frameworks (for example, PyTorch and XGBoost) using prebuilt and custom containers. Many questions reduce to: does a prebuilt container support this model as-is, or do I need custom code?

Prebuilt Inference Containers

Google maintains HTTP inference servers per framework and version, stored in Artifact Registry. Like training images, they follow a support schedule with end-of-patch and end-of-availability dates, and you must use the latest tag.

Upload the artifact directory to Cloud Storage, register the Model with the matching prebuilt image, and deploy. The framework version in the image should match the training version.

Optimized TensorFlow runtime

The optimized TensorFlow runtime is a drop-in serving image that uses Google's proprietary and open-source technologies to cut inference cost and latency compared with open-source TensorFlow Serving containers. It's backward compatible, so you switch images with no code changes.

When Prebuilt Isn't Enough

Custom Container Contract

A custom container must run an HTTP server that follows these rules:

Build the image, push it to Artifact Registry, upload a Model that references the image and routes, then deploy.

Custom Prediction Routines (CPR)

CPR gives you custom logic without writing a web server or Dockerfile:

1. Implement a Predictor with load (read artifacts), preprocess, predict, and postprocess.
2. Optionally implement a Handler for raw request access, custom headers, or non-JSON payloads. Google recommends keeping web logic in the Handler and ML logic in the Predictor.
3. Build the container with the SDK: LocalModel.build_cpr_model(src_dir, image_uri, predictor=..., handler=..., requirements_path=...).
4. Test locally with deploy_to_local_endpoint(...) before pushing, which speeds up iteration.
5. Push, upload, and deploy like any custom container.

class ChurnPredictor(Predictor):
    def load(self, artifacts_uri):
        prediction_utils.download_model_artifacts(artifacts_uri)
        self._model = joblib.load("model.joblib")
        self._scaler = joblib.load("scaler.joblib")
    def preprocess(self, request):
        return self._scaler.transform(request["instances"])
    def predict(self, instances):
        return self._model.predict_proba(instances)[:, 1]
    def postprocess(self, scores):
        return {"predictions": [{"churn_prob": float(s), "tier": "high" if s > 0.7 else "low"} for s in scores]}

Putting preprocessing in the serving container keeps training and serving transformations consistent when they share code (Chapter 15).

Testing Containers Before Deployment

1. Run locally with the same environment variables Agent Platform sets (AIP_HTTP_PORT, AIP_HEALTH_ROUTE, AIP_PREDICT_ROUTE, AIP_STORAGE_URI), or use CPR's deploy_to_local_endpoint.
2. Send sample requests in the exact instances format your clients will use, including edge cases such as missing fields.
3. Measure startup time so health checks and minimum replicas are set realistically.
4. Load-test a staging endpoint to choose machine type and autoscaling targets (Chapter 14).
5. Scan the image in Artifact Registry for vulnerabilities, and pin base image versions.

Choosing a Container Strategy

Exam Traps

• Naming artifacts incorrectly, such as model.pkl for a scikit-learn prebuilt container that expects model.joblib, or a PyTorch archive not named model.mar.
• Returning 200 OK on the health route before the model finishes loading. Requests then fail during startup.
• Choosing a full custom container when a prebuilt container plus CPR would do.
• Sending large payloads, such as base64 images, to a shared public endpoint over the 1.5 MB limit. Use dedicated or private endpoints, or pass Cloud Storage URIs instead.