2.7 Prototyping with Model Garden Foundational and Open Models in Notebooks

2.7 Prototyping with Model Garden Foundational and Open Models in Notebooks

Blueprint reference: Section 2.2, "Using a variety of foundational and open-source models in Model Garden to create model prototypes in notebook environments."

The previous chapter covered selecting a model. This bullet is about the mechanics of the comparison itself, and the exam's angle is almost always cost or reproducibility rather than modelling.

Two Very Different Prototyping Paths

Path A — managed API models. Gemini, other first-party models, and partner models as a service are called through the SDK. There is nothing to deploy, billing is per token, and a notebook can compare three of them in minutes for a few cents.

from vertexai.generative_models import GenerativeModel   # SDK package name is unchanged
model = GenerativeModel("gemini-2.5-flash")
response = model.generate_content(prompt)

Path B — open-weight checkpoints. Gemma, Llama-family, and other open models in Model Garden are deployed: the notebook provisions an endpoint with an accelerator, uploads or references the checkpoint, and waits for the deployment to become ready. Only then can it send requests.

The exam's favourite consequence: that endpoint bills for accelerator hours from the moment it is ready until it is undeployed, regardless of whether the notebook sends a single request. A prototype endpoint left up over a weekend is a well-known and entirely avoidable cost incident. The correct habit is to undeploy the model and delete the endpoint at the end of the comparison, and to build that teardown into the notebook itself rather than trusting memory.

# Always paired with the deploy call, ideally in a try/finally
endpoint.undeploy_all()
endpoint.delete()

A second consequence is quota. Deploying a large open model requires accelerator quota in the region, and "the deployment never becomes ready" is usually a quota problem rather than a model problem.

Structuring a Fair Comparison

A prototype that cannot be defended is not a decision. The elements of a defensible comparison:

1. A fixed eval set. The same 100–300 representative inputs for every candidate, versioned so the comparison can be repeated. Include the hard cases deliberately; an eval set of easy examples separates nothing.
2. Identical prompts. If one candidate gets a carefully engineered prompt and another gets a first draft, the comparison measures prompt effort, not model quality.
3. Quality scoring. Task metrics where ground truth exists; an LLM-as-a-judge rubric where it does not, with the rubric written down.
4. Latency and cost recorded on the same runs. p50 and p95 latency, tokens consumed, and — for self-deployed models — the accelerator configuration and hourly rate.
5. Everything logged to Experiments. Each candidate is a run with parameters (model name, temperature, prompt version) and metrics (score, latency, cost). This turns scrollback into a comparable record.

import vertexai
vertexai.init(experiment="model-selection-support-router")
with vertexai.start_run(run="gemma-open-endpoint-a2"):
    vertexai.log_params({"model": "gemma", "accelerator": "1xL4", "prompt_v": 3})
    vertexai.log_metrics({"judge_score": 0.86, "p95_ms": 410})

Lightweight Customization in a Notebook

Full-parameter fine-tuning of a large open model is a training-job workload, not a notebook workload. What is practical in a notebook is parameter-efficient tuning — training a small set of adapter weights on top of a frozen base model. The base checkpoint stays unchanged, the trained artifact is small, and multiple adapters can be maintained against one base model.

This gives a prototyping loop that is genuinely fast: try prompting first, then few-shot, then a small adapter, and only escalate to full tuning or a larger model if the adapter fails to close the gap. It is the same escalation ladder as the Gemini tuning decision, applied to open models.

Prototype to Production

The handoff points the exam cares about:

• Register the winner. Upload the chosen model to the Model Registry with a version and a description of why it won, so serving and governance work from a catalogued artifact rather than a notebook variable.
• Keep the eval set. It becomes the regression suite for future model upgrades. When a base model version changes, rerunning the same eval set is the only way to know whether quality moved.
• Move the loop into a component. A comparison worth repeating belongs in a pipeline so that the next model release is evaluated automatically rather than by whoever remembers.
• Record the negative results. Which models were rejected and why is the part of the record that prevents the same comparison being run again in six months.

Exam Traps

• Leaving a prototype endpoint deployed. The single most common cost mistake in this area.
• Comparing models on different prompts or different eval data. The comparison measures nothing.
• Assuming open models are cheaper. Free weights, paid accelerators.
• Full fine-tuning inside a notebook kernel. Use a training job.
• Discarding the eval set after the decision. It is the regression suite.