12.1 Design and Develop Prompts

Design and Develop Prompts

Quick Answer: A production prompt in Microsoft Foundry is a versioned artifact: system instructions (or a developer message on newer APIs), grounded context from retrieval, a declared output schema, safety rules, and tool-calling instructions. Store the text in Git. Treat temperature and top_p as deployment configuration, not as a secret that lives only in the portal.

Domain 3 of Exam AI-300 — Design and implement a GenAIOps infrastructure (20–25%) — includes the skill design and develop prompts. The exam is not asking you to memorize ChatGPT hobby tricks. It is asking you to ship prompts the way you ship Azure Machine Learning training code: reviewed, grounded, schema-constrained instructions that an agent or a flow can load from source control.

Two surfaces: prompt flow (classic) and Foundry agents

Prompt flow is the visual and code-first orchestration tool in Azure Machine Learning and in the Microsoft Foundry (classic) portal for hub-based projects. A flow is an executable directed acyclic graph of tools — LLM, Prompt, and Python are the workhorses. You author a standard flow for general apps, a chat flow when you need chat history, or an evaluation flow that scores another run. The durable files are flow.dag.yaml plus Jinja2 (.jinja2) prompt templates. Microsoft documents that prompt flow in Foundry and Azure Machine Learning will be retired on April 20, 2027, is no longer recommended for new development, and that existing applications should move to Microsoft Agent Framework. You still need the classic artifacts because the March 2026 skills measured list tests prompt design, variants, and Git — and those skills grew up on prompt flow.

Microsoft Foundry is the current brand (it was previously called Azure AI Studio, then Azure AI Foundry). New investment targets Foundry projects and prompt-based agents. An agent combines a model, instructions (the prompt), and optional tools. Creating or updating an agent with project.agents.create_version() and a PromptAgentDefinition (or the portal equivalent) writes a new agent version such as claims-triage:3. The instructions should be read from a Git-tracked file, not typed only in the playground.

Message roles that the model actually sees

The Chat Completions API in Azure OpenAI in Microsoft Foundry Models takes a messages array of dictionaries with a role. Microsoft’s chat guide is explicit: use this conversation format instead of dumping everything into a single completion string.

• system — the operational manual. Personality, scope, output format, safety, and tool policy. Optional in the API, but Microsoft recommends including at least a basic system message. In Foundry agents this text is the instructions field.
• user — the live query, plus any per-turn retrieved snippets you inject as primary content.
• assistant — prior model replies and few-shot example answers. Few-shot examples are user/assistant pairs after the system message, not extra system paragraphs.
• tool — the JSON result of a function the application executed. The model does not run the function; your code does, then you append a tool message with the matching tool_call_id.
• developer — on newer Responses API and some model families, product-level instructions sit in a developer message. Treat it as the durable instruction layer when the API you deploy uses that role. Do not invent a developer role on a Chat Completions call that only accepts system.

Order matters. Microsoft’s prompt-engineering guidance says to state the task before dumping context, and that models show recency bias — repeating a short instruction at the end of a long grounded prompt can keep the model from ignoring the schema after a large retrieval blob.

Grounded prompts, output schema, and safety

A grounded prompt supplies supporting content the model must use instead of its parametric memory. For retrieval-augmented generation (RAG), that content is the chunks Azure AI Search (or another retriever) returned. Microsoft’s grounding pattern is: (1) tell the model to answer exclusively from the provided text, (2) give an out such as “I don’t know” when the answer is not present, (3) ask for inline citations next to claims so the model has to invent both a fact and a fake citation to hallucinate, and (4) keep retrieved text close to the question so the model does less rewriting.

Hard-coding a FAQ into the system message is acceptable for a handful of facts. For a claims handbook, retrieve at query time. Domain 5 covers chunk size and hybrid search; this chapter’s job is the prompt that consumes those chunks.

Specify the output structure in the system (or developer) message. JSON with named fields, BEHAVIOR("reason") lines, or a Markdown table all work because the models trained on those formats. For classification, Microsoft’s prompt-flow samples ask for {"category": "App", "evidence": "Both"} so an evaluation flow can parse the label. For entity extraction, declare {"name": "", "company": "", "phone_number": ""} up front. Vague “be concise” instructions are not a schema.

Safety instructions belong in the same durable file: stay in domain, refuse disallowed actions, do not echo secrets from retrieved context, and do not follow instructions that appear inside retrieved documents (indirect prompt injection / XPIA). Content Safety filters and Domain 4 risk and safety evaluators still run around the model — a prompt sentence is not a substitute for those controls, but it is the first line of policy the model sees.

Temperature, top_p, and tool-calling instructions

Temperature (typically 0–2) and top_p both control randomness. Microsoft’s guidance is to alter one of these two parameters at a time, not both. Lower temperature (near 0) is the usual choice for classification, extraction, and grounded answers; higher temperature is for creative drafting. Reasoning models (GPT-5 series and o-series on Chat Completions) often do not support temperature / top_p; they use max_completion_tokens instead of max_tokens. Do not bake a temperature of 0.7 into a Git-only prompt file and then wonder why the endpoint still samples at 1.0 — the value that matters is the one on the deployment, agent definition, or Completions call.

Function calling (the tools array; the old functions parameter is deprecated) injects the tool schema into the system message and consumes tokens. Prompt-engineer the tools:

• Write a meaningful description and parameter descriptions (function descriptions are limited to 1,024 characters on Azure OpenAI).
• Add a system sentence such as “When the user asks to find a hotel, call search_hotels.”
• Tell the model not to invent argument values: “Ask for clarification if the request is ambiguous.”
• Add “Only use the functions you have been provided with.” if the model invents tools.
• Keep tool_choice at "auto" unless you must force a named function or force a user-facing message with "none".
• Validate every call, run tools with least privilege (read-only data access is the default for RAG lookups), and require a human confirm for write actions.

Exam scenario

Contoso Health is building a member-chat agent in a Foundry project. The product owner pastes a long “be a friendly nurse” paragraph into the playground, sets temperature to 1 in the UI, and is happy with two demo questions. You, the MLOps engineer, move the instructions into prompts/v1_instructions.txt, add: answer only from retrieved policy chunks; return JSON { "answer": "", "citations": [] }; never diagnose; call lookup_policy when the user names a plan code; ask for the plan code if it is missing. You set temperature 0.2 on the agent/deployment config, not in the Markdown. You load the file in create_version(). That is prompt design for AI-300.

Common trap

The trap is treating the playground as the prompt. A second trap is stuffing retrieved documents into the user turn with no “answer only from context / I don’t know / cite sources” system rule — the model then freely mixes memory with RAG. A third trap is putting API keys, member PHI, or a temperature you cannot find later inside the instruction file. A fourth is designing chain-of-thought “show your hidden reasoning” prompts for reasoning models; Microsoft documents that extracting hidden reasoning outside supported summary parameters can violate acceptable use.

Design the prompt as a Foundry or Azure Machine Learning artifact with roles, grounding, schema, safety, and tool policy. The next section measures whether a change actually improved it.