16.3 Monitor, Optimize, and Promote Fine-Tuned Models

Monitor, Optimize, and Promote Fine-Tuned Models

Quick Answer: Train in non-production. Watch train_loss versus full_valid_loss and token accuracy; deploy a checkpoint if the last epoch overfits. Evaluate with quality and safety evaluators, then host on a Developer deployment (24-hour life, no SLA, no data-residency, no hourly hosting fee) for cheap candidate tests. Promote manually to Standard (regional) or provisioned throughput (PTU) in production, pin the version, monitor quality and hourly hosting plus tokens, and roll back to the base model. Promotion is not automatic.

The last Domain 5 fine-tuning bullet is monitor and optimize fine-tuned model performance and manage a fine-tuned model from development through production. Training success is not the same as production readiness. Foundry will not promote a job for you when train_loss looks pretty.

Train in non-production, then read the curves

Submit the job from a dev Foundry project or a non-prod Azure OpenAI resource. Use a training type that matches residency (section 16.1). After submit, jobs may queue. Training takes minutes to hours depending on the base model and dataset size.

Monitor these series on the job’s Monitor pivot or in the attached results.csv:

• train_loss: loss on the training batch. Each step is one forward and backward pass. Should trend down.
• full_valid_loss: validation loss at the end of each epoch. Should trend down if you provided a validation file.
• train_mean_token_accuracy and full_valid_mean_token_accuracy: share of tokens predicted correctly. Should trend up when training is healthy.

If training loss keeps falling while validation loss rises, you are overfitting. Microsoft’s documented response is fewer epochs or a smaller learning-rate multiplier — not “add 10,000 more synthetic rows and hope.” You can pause a running OpenAI job after at least one step while it is in Running; pause creates a deployable checkpoint after safety evaluations. Pause is not documented as a universal managed-compute feature.

Each epoch writes a checkpoint (ftchkpt-...). A checkpoint is a fully functional model: you can deploy it or use it as the base for a later fine-tune. When the job finishes, the three most recent versions are available to deploy. The final epoch is the fine-tuned model; the previous two epochs remain as checkpoints. If epoch 3 overfits, deploy epoch 2. Checkpoints can be copied across resources and subscriptions with the REST API.

results.csv columns include step, train_loss, train_mean_token_accuracy, valid_loss, validation_mean_token_accuracy, full_valid_loss, and full_valid_mean_token_accuracy. The portal plots loss and token accuracy with training and validation on the same chart when you supplied a validation file.

Loss is not product quality. A model can have beautiful curves and still fail groundedness, schema validity, or safety. Always run Foundry evaluators on a held-out set (section 16.2).

Evaluate quality and safety before any production SKU

Use the same evaluation discipline as Domain 4, pointed at the candidate and at the base model on the same dataset:

• Quality: groundedness, relevance, coherence, fluency, task adherence, and tool-call correctness.
• Risk and safety: hate and fairness, self-harm, violence, jailbreak, protected material.

If the fine-tune wins schema accuracy but loses a safety evaluator, you do not promote. You fix data or prompts and train again. Foundry Users may train; Foundry Owner (or a custom role with Microsoft.CognitiveServices/accounts/deployments/write) deploys. Role names recently changed from Azure AI User / Owner; IDs did not.

At inference, the system message must match training. Microsoft documents that a different system message is a reason the custom model will not behave as expected. Pin the system message in the same Git repo as the JSONL.

Continuous fine-tuning (OpenAI models only) is the path when production traces show new failure modes: take the previous base.ft-{jobid} as the new base, train on new disjoint JSONL, eval again, then promote a new pinned deployment. Do not silently overwrite the production deployment name with an unevaluated job.

Developer deployments for 24-hour evaluation

Do not pay Standard hourly hosting just to run an eval suite.

Developer inference deployments (DeveloperTier in the comparison table, developertier in ARM examples, --sku-name "Developer" in Azure CLI) exist only to evaluate fine-tuned candidates:

• Pay-per-token only — no hourly hosting fee
• No availability SLA
• No data-residency guarantee (processing can leave the geography)
• Fixed 24-hour lifetime, then auto-deleted regardless of traffic
• Not for production, not for latency-sensitive tests that need an SLA
• Listed for GPT-4.1, GPT-4.1-mini, GPT-4.1-nano, and o4-mini from all Azure OpenAI regions in Microsoft’s current tables (confirm the model list on the deployment-types page; it moves)

Deleting the Developer deployment does not delete the customized model. Redeploy at any time. If even your test environment requires residency or an SLA, use Standard for that test — not Developer.

Control-plane deploy uses the Azure Resource Manager deployments API (or az cognitiveservices account deployment create), not the data-plane Chat Completions API. Example SKU payload: "sku": {"name": "developertier", "capacity": 50}.

Promote to Standard or PTU — manually

Fine-tuned models store at rest at no extra hosting cost until you deploy. Each deployed custom model except Developer incurs hourly hosting whether or not anyone calls it. Token charges apply on top when you infer.

Microsoft documents Global Standard fine-tune hosting as preview in the deploy guide, with regional Standard available in listed regions (for example East US 2, North Central US, Sweden Central — check the current table). Regional PTU for some fine-tunes is listed for GPT-4o and GPT-4o-mini in North Central US and Sweden Central, and GPT-4.1 PTU in Sweden Central. Fine-tunes share the same regional PTU quota as base models: if you already purchased regional PTUs, you can host the custom model on that capacity in supported regions.

Automatic deployment after a successful training job is optional and OpenAI models only. It still requires Owner. It deploys according to the type you picked in the wizard. It is not a production promotion gate and it does not wait for quality or safety evaluators. There is no platform feature that watches eval scores and flips production traffic. You promote: eval pass → new Standard or PTU deployment with a pinned model ID → application or gateway routes traffic → previous deployment stays up for rollback.

Pin the version. Fine-tuned model IDs look like gpt-4.1-2025-04-14.ft-b044a9d3cf9c4228b5d393567f693b83. You may instead deploy a checkpoint ID (ftchkpt-...). Put that ID in Bicep or Azure CLI. Do not point production at “whatever the latest job produced.”

Cross-region and cross-subscription deploy is supported when the destination region supports fine-tuning and the identity that mints the token can read the source account and write the destination. The portal cannot do cross-region deploy; use SDK or REST. Cross-tenant deploy needs two tokens and the x-ms-authorization-auxiliary header.

Prompt caching is available on select fine-tuned models and can cut latency and cost when the prefix of the prompt is identical. It is an inference optimization, not a reason to skip eval.

Monitor in production, then roll back

Once the candidate is on Standard or PTU:

• Quality: sample production logs into Foundry continuous evaluation (Chapter 14). Watch groundedness, relevance, schema-parse fail rate, and tool-error rate versus the baseline you recorded at promotion.
• Cost: hourly hosting (custom models bill even at zero queries per second) plus token usage. A forgotten Standard fine-tune in an unused subscription is a standing charge. Developer is the exception: tokens only, then it vanishes at 24 hours.
• Inactivity (Standard / Global custom deployments): if the deployment is older than 15 days and no chat completions or Responses API calls arrive for a continuous 15-day period, Azure deletes the deployment. The customized model remains and can be redeployed. Developer deployments die at 24 hours even if busy.
• Rollback: keep the base model (and the previous fine-tune) deployed or instantly redeployable. Point the app to the base deployment name. Rollback is a routing change, not a platform “undo fine-tune” button. You do not retrain to roll back.
• Cleanup: delete the deployment before you delete the model. Foundry will not delete a fine-tuned model that still has a deployment. Delete unused training, validation, and results.csv files when policy requires.

Exam scenario: You fine-tuned gpt-4.1-mini in a dev project with Global training. A Developer deployment beats the base model on schema validity and passes safety evaluators on the held-out gold set. Finance requires in-region inference. You deploy Standard in Sweden Central, set the API model parameter to that deployment name, keep the base gpt-4.1-mini deployment, and alert on hosting hours plus parse-fail rate. A week later parse-fail spikes; you route traffic back to the base deployment in minutes without deleting the custom model.

Common trap: Enabling automatic deployment to Standard in production from the training wizard, skipping held-out eval, and discovering a safety regression only after users complain. Automatic deploy fires after successful training, not after successful evaluation. A second trap: leaving a Standard fine-tune deployed in an unused environment — you pay hourly until you delete it or it ages out after 15 days of inactivity. A third trap: treating Developer as a weekend demo environment; it will disappear at 24 hours even if the demo is still running.

Promotion checklist

Work this list in order. None of the boxes is optional for production:

• Trained in non-production; training type matches residency policy.
• Validation loss is not diverging from training loss; checkpoint chosen if the last epoch overfits.
• Quality and safety evaluators passed versus the base model on a held-out gold set the generator never saw.
• System message at inference matches training.
• Developer 24-hour eval complete (or Standard test if residency is required even in test).
• Production SKU is Standard or PTU — never Developer.
• Fine-tuned model ID or checkpoint ID is pinned in IaC and app configuration.
• Base model or prior fine-tune remains available for rollback.
• Hosting-cost, token, latency, and quality monitors are on.
• No one assumes Foundry will promote the job because training succeeded.

If a box is unchecked, the model stays a candidate. Candidates live on Developer or in storage. Production traffic stays on the pinned previous deployment.