8.1 Microsoft Foundry Custom Models & Foundry Tools Selection

Microsoft Foundry Custom Models & Foundry Tools Selection

Quick Answer: The Azure AI Foundry Model Catalog serves as the unified control plane for discovering, evaluating, and deploying frontier foundation models (OpenAI GPT-4o, o1, o3-mini), open-weights architectures (Meta Llama 3.1/3.3, Mistral Large), and Microsoft Phi Small Language Models (Phi-3.5, Phi-4). When architecting agent capabilities within the Foundry Agent Service, architects select from four core tools: Code Interpreter (isolated Python sandbox for deterministic math, data analysis, and chart generation with zero network egress), File Search (managed chunking, vector indexing, and hybrid retrieval with citation annotations), Function Calling (strict OpenAPI 3.0 schemas for external API integration), and Bing Grounding (real-time public web search). Multi-tool pipelines dynamically route execution graphs across these engines to handle complex enterprise workflows.

Naming currency — read this before the exam. Microsoft has consolidated and rebranded this platform, and the official AB-100 skills outline uses the new names exclusively. Map them as you read:

Previous brand	Current brand used on the exam
Azure AI Studio / Azure AI Foundry	Microsoft Foundry
Azure AI Services	Foundry Tools
Azure AI Foundry Agent Service	Microsoft Foundry Agents Service
Azure AI Foundry model catalog	Foundry Models catalog

Foundry also now distinguishes two agent form factors that appear in design scenarios: a declarative prompt agent (instructions, a model and attached tools, hosted and run by Foundry with no application code or containers to maintain) and a code-first hosted agent (your own container built on a framework such as Microsoft Agent Framework, LangGraph or Semantic Kernel, run by Foundry with a managed endpoint, scaling, identity and observability). This guide uses Azure AI Foundry and Microsoft Foundry interchangeably because both names remain in wide circulation in documentation and tooling, but exam item stems will say Microsoft Foundry, Microsoft Foundry Tools and Microsoft Foundry Agents Service.

Enterprise agentic solutions require a careful balance between cognitive reasoning capacity, execution latency, token cost, and tool integration boundaries. A solution architect designing an agent for financial reconciliation cannot rely solely on language model generation for numerical arithmetic without risking hallucinations. Similarly, attempting to ingest complex, multi-megabyte PDF policy documents directly into the prompt context window rapidly exhausts token quotas and inflates operational expenses. Azure AI Foundry addresses these architectural trade-offs through an integrated model catalog and a purpose-built suite of managed agent tools.

---

1. Azure AI Foundry Model Catalog & Model Endpoints Architecture

The Azure AI Foundry Model Catalog provides a unified model plane that abstracts the underlying infrastructure required to deploy, fine-tune, and invoke foundation and open-source models.

                    AZURE AI FOUNDRY MODEL TAXONOMY

  Frontier Foundation Models       Open-Weights Models           Small Language Models (SLMs)
  +------------------------+    +------------------------+    +-----------------------------+
  | OpenAI GPT-4o          |    | Meta Llama 3.1 / 3.3   |    | Microsoft Phi-3.5 / Phi-4   |
  | - Multimodal (Vision)  |    | - 8B, 70B, 405B        |    | - Mini, Medium, Vision, MoE |
  | - General Reasoning    |    | - Weight Sovereignty   |    | - Low Latency / High QPS    |
  | OpenAI o1 / o3-mini    |    | Mistral Large / NeMo   |    | - Edge & On-Premises        |
  | - Deliberate Reasoning |    | - Multilingual & Code  |    | - Cost-Optimized Extraction |
  +------------------------+    +------------------------+    +-----------------------------+
              |                              |                               | 
              +------------------------------+-------------------------------+
                                             |
                                             v
                              DEPLOYMENT & HOSTING MODES
     +--------------------------------------------------------------------------------+
     | Serverless APIs (MaaS): Pay-as-you-go per token, zero cluster management       |
     | Managed Compute: Dedicated GPU clusters (NDv4/NCv3), VNet injection, fine-tune|
     | Provisioned Throughput (PTU): Reserved throughput, predictable latency SLAs    |
     +--------------------------------------------------------------------------------+

1.1 Model Tier Selection Strategy

1. Frontier Foundation Models (GPT-4o, o1, o3-mini):

   • GPT-4o: The flagship omni-model combining high-speed multimodal reasoning across text, vision, and audio. It is the primary orchestrator model for agents requiring complex dialogue, visual UI interpretation, and dynamic tool calling.
   • o1 and o3-mini (Reasoning Models): Engineered for deliberate, multi-step chain-of-thought deduction. These models spend internal compute tokens reasoning before emitting their final response. They excel in complex mathematical proofs, algorithmic code synthesis, actuarial risk analysis, and multi-variable logic problems where standard foundation models frequently hallucinate intermediate reasoning steps.

2. Open-Weights Models (Meta Llama 3.1/3.3, Mistral Large):

   • Enterprises subject to strict regulatory compliance, data sovereignty, or intellectual property constraints often mandate open-weights models. Deploying Llama 3.1 (70B or 405B) allows organizations to maintain full governance over model checkpoints, apply custom parameter-efficient fine-tuning (PEFT/LoRA), and guarantee that model weights can be migrated across private cloud enclaves.

3. Microsoft Phi Family (Phi-3.5, Phi-4 Small Language Models):

   • The Phi series demonstrates that high-quality, synthetic-data-trained SLMs can rival larger models on specific tasks. Phi-4 is engineered for edge deployment, high-throughput batch classification, structured JSON entity extraction, and conversational summarization at a fraction of the token cost and latency of frontier models.

1.2 Deployment Modes: Serverless MaaS vs. Managed Compute vs. PTU

---

2. Comprehensive Selection and Configuration of Foundry Tools

The Azure AI Foundry Agent Service provides a modular tool runtime. Agents register tools via standardized schemas, allowing the underlying model to generate structured invocations during reasoning.

                    FOUNDRY AGENT SERVICE TOOL ARCHITECTURE

                               +------------------+
                               |  Foundry Agent   |
                               |  Runtime Engine  |
                               +------------------+
                                        |
         +-----------------+------------+------------+------------------+
         |                 |                         |                  |
         v                 v                         v                  v
  +--------------+  +--------------+          +--------------+   +--------------+
  |     Code     |  |     File     |          |   Function   |   |     Bing     |
  | Interpreter  |  |    Search    |          |   Calling    |   |  Grounding   |
  +--------------+  +--------------+          +--------------+   +--------------+
  | - Python 3.x |  | - Managed    |          | - OpenAPI 3  |   | - Real-Time  |
  | - Sandboxed  |  |   Vector RAG |          | - Strict JSON|   |   Public Web |
  | - Egress: 0  |  | - Embeddings |          | - External   |   | - Domain     |
  | - Visuals    |  | - Citations  |          |   REST APIs  |   |   Filtering  |
  +--------------+  +--------------+          +--------------+   +--------------+

2.1 Code Interpreter: Secure Sandboxed Python Execution

Code Interpreter provides agents with a dedicated, isolated Python runtime environment. Instead of attempting mental math or probabilistic estimation, the agent writes Python code, executes it inside the sandbox, and incorporates the standard output and generated files into its answer.

• Core Capabilities: Deterministic mathematical computation, statistical analysis, data munging using pandas and numpy, dynamic chart visualization via matplotlib and seaborn, and file transformation (e.g., converting uploaded JSON payloads into Excel spreadsheets).
• Sandbox Security Architecture:
  • Network Egress Isolation: The container runtime has zero outbound network access. It cannot establish socket connections, execute HTTP requests, or exfiltrate enterprise data to public endpoints.
  • Ephemeral Lifecycle: Sandbox file systems are short-lived and scoped to the active execution thread. Once the conversation thread terminates, the container and all intermediate files are permanently destroyed.
  • Compute Quotas: Enforces hard caps on CPU execution time (typically 60–120 seconds maximum runtime per script) and memory consumption to prevent denial-of-service loops caused by recursive code.

2.2 File Search: Managed Vector Search and RAG

File Search equips agents with turnkey Retrieval-Augmented Generation (RAG) over corporate documentation without requiring the architect to manually provision an external Azure AI Search cluster or vector pipeline.

• Ingestion and Vector Pipeline: Automatically ingests files (PDF, DOCX, XLSX, TXT, MD, HTML), executes semantic chunking, generates embeddings using text-embedding-3-large, and indexes the vectors into an internal vector store.
• Hybrid Search & Semantic Reranking: Queries combine vector similarity with lexical keyword matching, passing candidates through a semantic cross-encoder reranking tier to maximize retrieval relevance.
• Footnote Citations: The tool automatically returns citation markers and character offsets, allowing client applications to render inline reference links pointing back to exact source document pages.

2.3 Function Calling / Custom Tools: OpenAPI 3.0 Integration

Function Calling bridges the gap between cognitive reasoning and external enterprise transaction systems (e.g., SAP, Salesforce, Dataverse, custom microservices).

• OpenAPI 3.0 Contracts: Tools are defined using JSON Schema contracts describing function names, operational descriptions, and input parameters.
• Strict Schema Enforcement ("strict": true): Ensures that model-generated tool arguments conform 100% to the defined JSON Schema. Under strict mode, the model runtime guarantees that required fields are never omitted, property types match the schema exactly, and hallucinated parameter keys are eliminated.
• Execution Models:
  • Client-Side Dispatch: The model emits the function call and parameters to the client application, which executes the local code/API and returns the result back to the agent thread.
  • Agent Service Server-Side Dispatch: The Foundry Agent Service directly invokes the authenticated external REST webhook endpoint on behalf of the agent, processes the JSON response, and continues execution seamlessly.

2.4 Browser Automation & Web Search: Bing Grounding

When agents must operate on live, publicly available information (e.g., current currency exchange rates, breaking regulatory announcements, market competitor intelligence), internal vector stores are insufficient.

• Bing Grounding Integration: Enables agents to execute targeted web searches against Bing's index.
• Enterprise Controls: Architects can restrict query domains (e.g., grounding exclusively on sec.gov or learn.microsoft.com), enforce SafeSearch policies, specify geographic locales, and ingest real-time source URLs as citations.

---

3. Comparative Architectural Matrix of Foundry Tools

---

4. Composing Multi-Tool Execution Pipelines & Routing Logic

Sophisticated enterprise agents rarely rely on a single tool. The Foundry Agent Service includes an orchestration engine that interprets compound user goals, evaluates registered tool contracts, and builds a multi-step execution graph.

                  MULTI-TOOL EXECUTION PIPELINE WORKFLOW

               User Request: "Analyze Q3 variance in the attached
               PDF, calculate the 3-year CAGR, and update SAP ERP"
                                        |
                                        v
                   +-----------------------------------------+
                   | 1. Orchestration Engine (Model Reasoning)|
                   +-----------------------------------------+
                                        |
                                        v
                   +-----------------------------------------+
                   | 2. File Search Tool                     |
                   | - Extracts Q3 revenue tables from PDF   |
                   +-----------------------------------------+
                                        |
                                        v
                   +-----------------------------------------+
                   | 3. Code Interpreter Tool                |
                   | - Computes CAGR formula via Python      |
                   | - Generates variance chart (PNG)        |
                   +-----------------------------------------+
                                        |
                                        v
                   +-----------------------------------------+
                   | 4. Custom API Tool (Function Calling)   |
                   | - Strict JSON validation                |
                   | - Posts financial ledger update to SAP  |
                   +-----------------------------------------+
                                        |
                                        v
                   +-----------------------------------------+
                   | 5. Final Grounded Response Synthesis    |
                   | - Returns answer with citations & PNG   |
                   +-----------------------------------------+

4.1 Execution Routing Walkthrough

1. Step 1 (Document Extraction): The agent identifies that the user query references an unstructured report. It invokes File Search with semantic queries targeting revenue tables. File Search parses the PDF, extracts the relevant tables, and returns raw text chunks with citations.
2. Step 2 (Statistical Computation): The agent inspects the extracted numbers. Rather than performing mathematical operations in its internal transformer layers, it dispatches the historical figures to Code Interpreter. The Python sandbox calculates the exact Compound Annual Growth Rate (CAGR) and renders a variance chart.
3. Step 3 (Enterprise Mutation): With the validated calculation in hand, the agent targets the enterprise backend. It constructs a strictly validated JSON payload conforming to the SAP finance tool schema (UpdateLedgerEntries) and dispatches the Function Call.
4. Step 4 (Synthesis & Citation): The agent synthesizes a cohesive response summarizing the findings, embedding the citation references to the PDF, providing a download link for the generated chart artifact, and confirming the SAP transaction ID.

4.2 Tool Choice Strategies

In the Foundry Agent Service, developers control tool routing behavior using the tool_choice parameter:

• tool_choice = "auto": The model dynamically decides whether to answer directly from parametric memory or invoke one or more tools.
• tool_choice = "none": Forces the model to generate a direct response without invoking any tools.
• tool_choice = "required": Forces the model to call at least one tool before completing its turn.
• tool_choice = {"type": "function", "function": {"name": "UpdateRecord"}}: Forces the model to execute a specific designated function, ensuring deterministic routing for specialized agent nodes.

---

5. Real-World Architectural Case Scenario: High-Volume Actuarial Portfolio Analysis & Ledger Integration

The Incident

A multinational pension fund developed an autonomous actuarial risk assessment agent in Azure AI Foundry. Actuaries required the agent to ingest quarterly actuarial valuation filings (150-page PDFs), recalculate mortality-weighted liability projections against changing benchmark yield curves, and post reserve adjustments into Dynamics 365 Finance. The initial prototype attempted to perform all actuarial math using GPT-4o's direct prompt completions and used standard (non-strict) function calling to update Dynamics 365.

During quarterly close, the system failed with critical discrepancies:

1. Compounding Arithmetic Hallucinations: The model hallucinated discount cash-flow compounding factors, leading to a $42 million understatement of projected liabilities.
2. Downstream API Deserialization Failures: Approximately 18% of ledger update payloads sent to Dynamics 365 failed because the model omitted optional enum parameters or passed string representations of dates when ISO-8601 timestamps were expected.
3. Token Context Window Exhaustion: Uploading 150-page PDF tables directly into prompt context consumed 110,000 tokens per call, causing Time-to-First-Token latency to exceed 35 seconds.

Root Cause Analysis (RCA)

The architecture team failed to establish clear tool boundaries. Large Language Models are probabilistic token predictors, not deterministic arithmetic engines; relying on internal model weights for multi-period actuarial math guarantees errors. Furthermore, omitting strict schema enforcement allowed non-deterministic argument generation, while direct PDF prompt injection bypassed vector indexing optimizations.

The Architectural Remediation Pattern

The lead solution architect restructured the agent pipeline across three Foundry tools:

1. File Search Indexing: Actuarial valuation PDFs were attached to a vector store managed by File Search, enabling the agent to retrieve targeted liability balance tables with exact page citations in under 800ms.
2. Code Interpreter Sandboxing: Liability projection formulas were offloaded entirely to Code Interpreter. The agent generated Python scripts utilizing numpy_financial and scipy to execute deterministic discounted cash-flow calculations, exporting matplotlib sensitivity curve plots.
3. Strict Schema Function Calling: The Dynamics 365 finance action was configured with "strict": true on its OpenAPI 3.0 contract. This forced the model runtime to guarantee 100% type compliance and mandatory parameter inclusion, reducing API deserialization errors to zero.

---

6. Architectural Exam Tips & Implementation Pitfalls

[!IMPORTANT] Code Interpreter Egress Myth: A frequent distractor on the AB-100 exam asserts that Code Interpreter can be used to scrape private internal intranets or call REST APIs using the Python requests library. This is architecturally impossible. Code Interpreter runs with complete network egress isolation; all external network traffic is blocked at the hypervisor. To query external APIs, you must use Function Calling.

[!TIP] Strict Schema Enforcement: When building enterprise function calling tools, always enable "strict": true. Non-strict schemas allow models to pass optional fields with unexpected types or hallucinate extra parameters when edge-case user inputs occur. Strict schema validation enforces JSON Schema draft-7 compliance, eliminating downstream deserialization failures.

[!WARNING] Reasoning Model Token Quotas: OpenAI o1 and o3-mini utilize reasoning tokens that count toward overall token-per-minute (TPM) limits and billing metrics. These reasoning tokens are generated internally during deliberate problem decomposition and are not exposed in the final completion payload. Architects must account for reasoning token overhead when sizing throughput limits.