10.3 Fine-Tuning Foundation Models & When to Tune
Key Takeaways
- Google recommends finding the best prompt first and tuning only when prompting can't reach the quality, consistency, or cost target.
- Gemini supervised tuning jobs report training and validation loss and fraction_of_correct_next_step_preds so you can spot overfitting.
- Tuning jobs with fewer than 10 epochs save about one checkpoint per epoch, and longer jobs save about 10 evenly spaced checkpoints plus the final one.
- Continuous tuning keeps tuning an already tuned Gemini model or checkpoint with more epochs or new examples.
- For open models, QLoRA uses about 75% less peak GPU memory than LoRA, while LoRA tunes about 66% faster and costs up to 40% less.
The exam guide lists fine-tuning foundational models from Agent Platform and Model Garden, and when tuning should be considered. Chapter 4 compared tuning methods (supervised, preference, reinforcement, distillation). This section covers the decision and the execution.
When Should Tuning Be Considered?
Google's guidance: start with prompting to find the best prompt, then tune if needed to boost performance or fix recurring errors.
| Signal | Tune? | Why |
|---|---|---|
| Well-written prompts still give inconsistent format, tone, or labels | Yes | Tuning teaches the behavior directly |
| Long few-shot prompts make every request slow and expensive | Yes | A tuned model needs shorter prompts, cutting tokens and latency |
| A task with domain-specific syntax or rules (claim codes, internal query language) | Yes | The examples encode patterns that are hard to explain in a prompt |
| Answers need current or changing facts | No, ground instead | Tuned knowledge goes stale, while retrieval stays current |
| Only a handful of examples exist | Not yet | Supervised tuning works best with a sizable, high-quality set (think 100+ examples, usually hundreds) |
| Requirements change weekly | Probably not | Each change means rebuilding data and re-tuning |
| The base model already meets the quality bar | No | Tuning adds cost and lifecycle work |
Before tuning, analyze failures. If errors come from missing context, fix retrieval. If they come from ambiguous instructions, fix the prompt. Tune when the model needs to learn a pattern.
Running a Gemini Supervised Tuning Job
1. Prepare data
- Training dataset: JSON Lines in Cloud Storage. Each example holds the conversation (
userinput and idealmodeloutput), optionally with a system instruction. Multimodal examples reference files by URI. - Validation dataset (optional but recommended): separate examples from the same distribution. This enables validation metrics.
- Match production prompt formats exactly, and remove PII and duplicates.
2. Configure the job
| Setting | Effect |
|---|---|
| Base model | A Gemini version that supports supervised tuning |
| Epochs | Full passes over the training data. Leave unset to use the recommended value |
| Adapter size | Number of trainable parameters in the adapter. Larger sizes can learn more complex tasks but need more data and compute |
| Learning rate multiplier | Scales the recommended learning rate |
| Export last checkpoint only | Skip intermediate checkpoints |
| Region | Tuning and tuned-model serving run only in the regions each model supports |
3. Monitor tuning metrics
| Metric | Meaning |
|---|---|
/train_total_loss, /eval_total_loss | Loss on the tuning and validation data |
/train_fraction_of_correct_next_step_preds, /eval_fraction_of_correct_next_step_preds | Token-level accuracy against the ground truth |
/train_num_predictions, /eval_num_predictions | Tokens predicted per step |
If training loss keeps falling while validation loss rises, the model is overfitting.
4. Use checkpoints
- Fewer than 10 epochs: about one checkpoint per epoch.
- More than 10 epochs: about 10 evenly spaced checkpoints, plus the final checkpoint.
- Intermediate checkpoints are deployed to endpoints as tuning progresses. You can pick the best checkpoint before overfitting as the default.
5. Deploy and evaluate
The tuning job produces a tuned model with an endpoint for inference. Tuned-model inference shares quota with the base model. Evaluate the tuned model against the base model on the same evaluation set with Gen AI evals (Chapter 7) before switching traffic. For thinking-capable models, set the thinking level to MINIMAL (Gemini 3 and later) or a thinking budget of 0 (Gemini 2.5 and earlier) on tuned tasks.
6. Continue tuning when needed
Continuous tuning starts from an already tuned model or checkpoint and adds epochs or new examples. It helps when a model underfits, when new data arrives, or when you want further customization, without starting over from the base model.
Tuning Open Models from Model Garden
Open models (such as Gemma or Llama) can be tuned in several ways:
- The model card's fine-tuning pipeline (runs on Agent Platform Pipelines) or Open notebook option.
- Custom training jobs with your own code.
- Supervised modes: full fine-tuning (all weights, highest potential quality and cost) or LoRA (parameter-efficient adapters). Distillation tunes a smaller student on a larger teacher's outputs.
LoRA vs. QLoRA
| Consideration | Recommendation |
|---|---|
| Least GPU memory | QLoRA: about 75% smaller peak GPU memory |
| Fastest tuning | LoRA: about 66% faster |
| Lowest cost | LoRA: up to 40% less expensive |
| Longer max sequence length or bigger batches | QLoRA (less memory per example) |
| Accuracy improvement | About the same for both |
Example from Google's guidance: tuning openLLaMA-7B on one L4 GPU fails with out-of-memory at LoRA batch size 1, while QLoRA runs at batch size 12. If you only have smaller GPUs, QLoRA makes tuning possible.
Tuned open-model weights are self-deployed from Model Garden to dedicated endpoints (billed for compute), with prebuilt serving containers such as vLLM that can serve adapters.
Cost and Latency Effects of Tuning
- Tuning cost is charged per tuning run, based on the tokens processed during tuning. Budget for several iterations, not one.
- Inference savings come from shorter prompts. Removing a 3,000-token few-shot block from millions of requests often outweighs the tuning cost.
- Smaller tuned models (Flash-Lite, or distilled open models) can match a larger prompted model on a narrow task at lower latency.
- Self-deployed open models trade per-token pricing for endpoint compute cost, which pays off at steady, high volume.
Tuning Lifecycle Checklist
- Base model version and its retirement date recorded.
- Training and validation datasets versioned in Cloud Storage.
- Tuning job configuration logged as an experiment run.
- Best checkpoint chosen from validation metrics.
- Evaluation results against the base model stored with the model in Model Registry.
- Re-tuning planned before the base model retires.
A legal team wants a Gemini assistant to answer questions about regulations that change monthly. A developer proposes supervised fine-tuning on this month's regulations. What is the better approach?
During a Gemini supervised tuning job, /train_total_loss keeps falling, but /eval_total_loss reaches its lowest point at epoch 3 and then rises through epoch 8. What should the team do?
A team must tune a 7B open model on a single L4 GPU, and LoRA fails with out-of-memory even at batch size 1. Which change is most likely to make tuning feasible?