Security, Networking, Monitoring, and Responsible AI
Key Takeaways
- Production AI-103 solutions should prefer Microsoft Entra ID, managed identity, and RBAC over embedded API keys.
- Private endpoints, private link, and managed networking reduce public exposure for Foundry projects, AI Search, Storage, APIs, and other dependencies.
- Content filters address harmful content categories, while Prompt Shields target direct and indirect prompt-injection attacks.
- Monitoring and tracing are separate but related: monitoring tracks production health and quality trends, while tracing captures request-level execution evidence.
- Responsible AI planning includes evaluators, safety evaluations, provenance metadata, approval workflows, tool constraints, and user-feedback loops.
Security, Networking, Monitoring, and Responsible AI
AI-103 treats security and responsible AI as design requirements, not cleanup tasks. A Foundry app might pass a quick demo with an API key, public endpoints, and no telemetry, but that design will not satisfy production scenarios. The exam favors keyless authentication, scoped authorization, private connectivity, safety filters, observability, and human oversight where actions are sensitive.
Identity and Access
Use Microsoft Entra ID and managed identity when Azure-hosted compute calls Foundry Models, Azure OpenAI, Azure AI Search, Storage, Key Vault, or internal APIs. A managed identity is an Azure-managed service principal, so code can request tokens without storing secrets. Then use role-based access control (RBAC) to grant only the permissions required at the correct scope.
| Requirement | Better design | Risky design |
|---|---|---|
| App calls model deployment | Managed identity with Azure AI or Azure OpenAI data-plane role | API key hard-coded in source or app settings with broad access |
| Project stores reusable service access | Project connection using Entra ID or managed identity where possible | Shared admin key pasted into multiple flows |
| Developer needs project access | Azure AI User or project-scoped role | Subscription Owner for day-to-day development |
| Secret still required | Store in Key Vault and retrieve with managed identity | Store in repository, prompt flow file, or front-end code |
Networking and Isolation
Use private networking when data sensitivity, compliance, or exfiltration risk matters. Private endpoints and private link create private IP paths to services such as Foundry resources, Storage, Azure AI Search, API Management, or private tool endpoints. In stricter designs, disable public network access and make sure every dependency has a private route, not just the model endpoint.
For agents, also constrain tools. Register only the tools the agent needs, authenticate tool calls independently, and require approval before high-impact actions such as sending external messages, updating production records, executing code, or calling privileged APIs.
Safety Controls
Content safety has multiple layers. Azure AI Content Safety content filters classify both prompts and completions across four harm categories: hate, sexual, violence, and self-harm. Each category is scored on a severity scale, and Azure OpenAI exposes configurable thresholds of safe, low, medium, and high so you choose how aggressively to block. The default filter blocks medium and high severity; a regulated workload may tighten to block low and above, while an internal research tool may loosen it.
Prompt Shields look for direct jailbreak attempts and indirect prompt-injection attacks hidden in retrieved documents, web pages, emails, images, or uploaded files. Protected-material filters detect known copyrighted text or code in generated output, and groundedness detection flags ungrounded (fabricated) claims in RAG responses.
| Control | Main purpose | Exam clue |
|---|---|---|
| Content filter (4 categories, 4 severities) | Block or annotate unsafe input and output | Harmful prompt or generated response |
| Prompt Shields | Detect jailbreak and indirect prompt injection | Retrieved content tries to override system instructions |
| Protected material detection | Flag known copyrighted text or code | Output reproduces lyrics, articles, or licensed code |
| Groundedness detection / evaluator | Check whether an answer is supported by sources | RAG bot invents facts or ignores citations |
| Approval workflow (human-in-the-loop) | Keep humans in control of sensitive actions | Agent can spend money, change data, or contact users |
Note the difference between a real-time filter (blocks or annotates a single request) and an evaluator (scores quality offline or on sampled traffic). Both can check groundedness, but only the filter intervenes in-line; the evaluator measures and reports.
Monitoring, Tracing, and Evaluation
Monitoring answers: "Is production healthy?" Foundry monitoring integrates with Azure Monitor and Application Insights to surface token consumption, latency, exceptions, quality scores, safety events, and response trends. Use alerts when cost spikes, latency rises, groundedness drops, or harmful-output metrics breach thresholds.
Tracing answers: "What happened in this one request?" Distributed traces record spans for prompt construction, retrieval, model calls, tool invocations, agent decisions, and downstream services. Traces support debugging, audit review, and provenance because they show the evidence trail behind a response.
Evaluation answers: "Is this system good enough?" Run pre-production evaluation on labeled datasets and use continuous evaluation on sampled production traffic. Include quality metrics, risk and safety evaluators, groundedness, relevance, task completion, and custom business checks. A reliable AI-103 solution combines all three: monitoring for trends, tracing for detail, and evaluation for measurable quality.
Foundry ships built-in evaluators you should be able to match to a goal. Groundedness and relevance check RAG answer quality against retrieved context; coherence and fluency check readability; similarity and F1/BLEU/ROUGE compare output to ground-truth labels; retrieval evaluators score whether the right chunks were fetched; and risk and safety evaluators score for violence, hate, sexual, self-harm, and protected-material content. For agents specifically, evaluate intent resolution, task adherence, and tool-call accuracy.
Tracing in Foundry is built on OpenTelemetry semantic conventions for generative AI, so spans are portable to Application Insights and other backends.
Scenario: Diagnosing a Bad RAG Answer
Users report the support bot occasionally cites the wrong policy. The correct triage order uses all three pillars. Monitoring shows that groundedness scores dipped after a knowledge-base reindex (a trend). Tracing for one failing request shows the retrieval span returned the wrong document chunk before the model ever ran (the detail). Evaluation on a labeled set then confirms the fix: after tuning the index and hybrid-search settings, the retrieval and groundedness evaluators recover.
The exam-correct lesson is that a wrong answer is usually a retrieval problem, surfaced by monitoring, pinpointed by tracing, and proven fixed by evaluation, not a reason to swap to a bigger model.
A RAG-powered agent uses managed identity and private endpoints, but users report that retrieved documents sometimes contain hidden instructions telling the agent to ignore its system prompt. Which control most directly addresses this risk?
An Azure-hosted Foundry app currently authenticates to its model deployment with a hard-coded API key in app settings. Which change best matches AI-103 production guidance?