16.2 Create and Manage Synthetic Data for Fine-Tuning

Create and Manage Synthetic Data for Fine-Tuning

Quick Answer: Use Microsoft Foundry data generation (preview) to expand a scarce labeled set into JSONL Chat Completions files. Generators take one reference file (a document for Simple Q&A, an OpenAPI JSON file for Tool use), emit 50–1,000 samples, and can split 80/20 train/validation. Never put personally identifiable information (PII) in the seed. Human-review, deduplicate, and never train on the evaluation set. Synthetic data can leak PII, amplify bias, and teach the model its own errors.

Domain 5 also measures create and manage synthetic data for fine-tuning. When you have 40 excellent gold conversations and the job wants hundreds, you do not scrape Slack into JSONL. You grow the gold set with a generator, then a human (or a stronger model plus a human) edits the output. Preview features change; the exam still expects the operating rules: seed hygiene, review, split discipline, and risk.

Why synthetic data exists

Microsoft lists four reasons Foundry generates synthetic training data:

• Real labeled conversations are scarce.
• You can preserve privacy if you generate from a cleaned policy or API specification instead of production transcripts that still contain member IDs.
• You can encode business-specific structure (your JSON schema, your tools, your confirmation policy).
• Generation is cheaper than paying annotators for every row.

It is still preview. Formats, limits, and the portal wizard can change. Always validate outputs before a production fine-tune. Synthetic data is a force multiplier for a good seed, not a substitute for a seed.

Foundry generators

In the portal, open Data > Synthetic Data Generation and select Generate data. The wizard asks for a task type, a reference file, a sample count, a generator model, and an optional split.

You also choose:

• Sample count between 50 and 1,000 (you cannot ask for 10 rows or 50,000 in this wizard).
• The model that writes the synthetic turns. If that model is not deployed, Foundry creates a deployment — which requires Foundry Owner (or equivalent deployment rights). Foundry User can run generation when a generator model already exists.
• Optional 80/20 train-validation split of the generated dataset.
• Optional suffix on the job name.

Output is JSONL ready for Foundry fine-tuning. Use Use this dataset in the preview pane or download the files for post-processing (PII scan, dedup, mixing with gold traces). Cost metrics appear on the completed job.

Supported regions for generation at the time of Microsoft’s July 2026 article include eastus2, eastus, westus, northcentralus, southcentralus, swedencentral, germanywestcentral, francecentral, uksouth, uaenorth, japaneast, and australiaeast. Confirm the current Learn page before you design a landing zone; preview region lists move.

Minimum role to open the wizard: Foundry User or Foundry Project Manager on the Foundry resource. Deploying the generator model is a separate, higher bar.

Seed-file hygiene

The preview accepts one reference file. If you have six policy PDFs, concatenate them into one file before upload. Keep formatting clean:

• Strip navigation chrome, cookie banners, “click here,” and repeated headers that are not subject matter.
• Put the target domain in the file. A generator cannot invent your claims SOP from a blank page.
• Do not include PII, account numbers, secrets, or production credentials. A generator will happily rewrite employee identifiers into hundreds of plausible fakes — or, worse, copy the real ones into assistant answers.
• For tool use, the OpenAPI specification is the curriculum. Incomplete paths, missing required properties, or hallucinated operation IDs in the spec become incorrect tool calls in every synthetic dialogue.

Simple Q&A from a tax statute produces system / user / assistant rows with a generic helpful-assistant system message unless you replace it. After download, overwrite that system message with the production prompt you will train and serve (section 16.1). Tool-use output already includes a long policy system message plus a tools array; still review it against your real agent policy.

Risks: PII, bias, and self-taught errors

Synthetic data is not free quality. Three failure modes show up on AI-300-style scenarios and in production postmortems.

1. PII leakage. If the seed PDF is a poorly redacted export of case notes, generated Q&A can restate names, member IDs, and diagnoses. Run PII detection (Azure AI Language or an equivalent Presidio-style scan) on both the seed and the output. Drop or rewrite hits. Fine-tuning on leaked PII can surface that PII at inference, which is a privacy incident, not a model-quality ticket.
2. Bias amplification. If 90 percent of seed examples show one demographic, one dialect, one claim outcome, or one tool path, the generator will over-produce that pattern. The fine-tuned model then treats the skew as policy. Balance the seed before you scale to 1,000 rows. After generation, check intent and demographic (or product-line) histograms, not only BLEU-like overlap.
3. Teaching the model its own mistakes. A generator that invents a statute citation, a fake flight number, or a tool argument the API will reject will produce hundreds of confident, wrong assistant turns. Fine-tuning cements those errors into LoRA weights. Always sample-review. For tool-use data, parse tool_calls and validate arguments against the OpenAPI schema or a sandbox API. Delete or repair rows that would book, cancel, or pay without the confirmation policy you require.
4. Distribution shift. Synthetic Q&A is cleaner than angry customers. Mix real, privacy-cleared production traces with synthetic rows so the model sees interruptions, typos, and “that is not what I asked.” A model trained only on generator-clean turns often fails the first messy production utterance.

Human review, deduplication, and the evaluation split

Treat generation as a factory, not a publish button.

Review workflow:

• Spot-check a stratified sample — every intent, every tool, every document section — not just the first 20 lines of the JSONL file.
• Score samples with the same quality evaluators you will use later (groundedness, relevance, task adherence) and risk and safety evaluators (hate, self-harm, violence, jailbreak, protected material). Chapter 13 is the evaluator catalog; this chapter is about not poisoning the training set those evaluators will judge.
• Have a domain expert accept or reject. Send rejects back as few-shot repairs or delete them. Do not “fix” a wrong citation by generating 200 more rows from the same dirty seed.

Deduplicate:

• Exact-line JSONL duplicates waste epochs and overweight that conversation.
• Near-duplicates (same user question, lightly paraphrased) collapse diversity. Hash normalized user turns and cap copies.
• If you generated from the same PDF twice, do not concatenate both dumps blindly.

Never train on the evaluation set.

• If you enable the portal’s 80/20 split, keep that validation file out of training_file. Validation loss is a training-time curve, not a product gate.
• Hold out a third gold set that the generator never saw: real tickets, real tool traces, real schema failures. Validation loss on a synthetic-only holdout is optimistic.
• Evaluation datasets you built in Domain 4 (groundedness and relevance mapping) must not be copied into JSONL training rows. Training on the test is how you ship a model that “passes eval” and fails users.

Seed quality beats volume. Fifty expert-edited dialogues plus 200 reviewed synthetic expansions beat 1,000 unreviewed generator dumps. Microsoft’s own fine-tuning article warns that a large unpruned internal dump can hurt relative to the base model. The same warning applies to synthetic volume.

Exam scenario: You have 60 gold tool-calling traces for a booking agent and a valid OpenAPI spec. You generate 800 more rows, enable 80/20, and fine-tune. Offline eval on the synthetic validation split looks excellent. Production users report the agent calling cancel_reservation without confirmation. Root cause: the generator omitted the “summarize then confirm” policy that existed only in the 60 gold traces, and you never mixed gold with synthetic. Fix: prepend the gold traces, put the confirmation policy in the system message on every row, regenerate, human-review confirmation turns, validate tool arguments against the spec, and evaluate on a held-out real transcript set.

Common trap: Training on the same generated file you later upload as the “eval set” in Foundry Evaluations. You are measuring memorization of synthetic phrasing, not generalization. Split once, pin file IDs in Git, and refuse jobs whose training_file identity equals the eval mapping file.

Operate the dataset as an MLOps asset

Synthetic data is a versioned artifact, the same way a component or an environment is:

• Version seed files, generator job IDs, generator model name, sample count, long-versus-short setting, and split ratios in Git or a data catalog.
• Prefer local upload or the SDK when Blob import would require public network access on the storage account. Microsoft requires public access for the Blob import path; many enterprises cannot allow that.
• After human edits, re-run JSONL validation (UTF-8 BOM, chat schema, 512 MB cap from section 16.1).
• Record what must never be generated: PII fields, secrets, internal hostnames, competitor names if policy forbids them.
• When the OpenAPI spec or the policy PDF changes, regenerate — do not keep training on last quarter’s synthetic dump.

Review gates before a fine-tune job may consume the files:

• Seed has no PII or secrets.
• Generator region is allowed for the workload.
• Sample count is in the 50–1,000 window.
• Human stratified review is complete and recorded.
• Exact and near-duplicate JSONL lines are removed.
• Training, validation, and held-out gold eval files are disjoint.
• Safety evaluators have scored a sample of assistant turns.
• Production system message is present on every training row.
• Gold traces are mixed in if they exist.

If any gate fails, you do not “make up for it” with more epochs. You fix the data. Section 16.3 will not save a model that learned the generator’s hallucinations.