11.1 Serverless API Endpoints and Managed Compute for Foundation Models

Serverless API Endpoints and Managed Compute for Foundation Models

Quick Answer: Microsoft Foundry puts a catalog foundation model into production with Serverless API (preferred for Foundry Models sold by Azure and selected partner models; pay-per-token Standard, plus Global, Data Zone, Batch, and Developer types, or reserved provisioned throughput units) or managed compute (dedicated GPU or CPU SKUs billed hourly). These are not Azure Machine Learning managed online endpoints from Domain 2.

Domain 3 of Exam AI-300 asks you to deploy foundation models by using serverless API endpoints and managed compute options. Chapter 10 stood up the Foundry resource, project, identity, and network. This section is the first production surface those projects hit: a live inference API over a catalog model.

Microsoft Foundry is the current brand. Older material still says Azure AI Studio or Azure AI Foundry; the exam measures the Foundry catalog and its two hosting paths. The catalog holds more than 10,000 models and publishes roughly 50 new ones each month. It splits commercially into two buckets, and that split drives support, billing, and which deployment option you get.

1. Foundry Models sold by Azure. Microsoft hosts and sells them under Microsoft Product Terms. Every Azure OpenAI model sits here, plus selected models from providers such as Meta, Mistral, Cohere, and DeepSeek that Microsoft integrates and supports. Billing uses first-party Azure meters. Enterprise service-level agreements (SLAs) and Microsoft support apply. Some of these models also offer fungible provisioned throughput, so the same PTU pool can back more than one supported model.
2. Foundry Models from partners and community. This is most of the catalog: Hugging Face hub models, many open weights, and partner families such as Anthropic Claude. The model provider chooses how you deploy and sets license terms. Hugging Face support is the Hugging Face forum; Anthropic in Foundry still routes operational help through Microsoft Support. Azure Marketplace meters often apply for partner offers.

You are responsible for lawful use, for reading the model card and provider terms, for picking a model that fits the use case, and for applying content safety so the workload matches Microsoft's Acceptable Use Policy. Foundry does not pick the model for you.

Two deployment options, not four

The exam mixes Domain 2 and Domain 3 on purpose. Keep the product map small.

Serverless API is the preferred deployment option in Foundry. Use it whenever the model card lists it. Use managed compute when the card offers only that path, or when you must host custom weights on dedicated accelerators.

Instant access (preview) lets you call some supported models by name without creating a Serverless API or managed compute deployment. Treat it as a try-it path, not a production contract. The exam still expects you to know how to create a deployment.

If you create the deployment from an Azure OpenAI resource, the catalog shows only Azure OpenAI models. Upgrade to a Foundry resource when you need the rest of Foundry Models sold by Azure. Deployments are Microsoft.CognitiveServices/accounts/deployments resources; Azure Policy can deny a SKU name such as GlobalStandard if the organization forbids global processing.

Serverless API: types, data residency, and billing

A Serverless API deployment provisions an inference endpoint. Microsoft hosts the weights. You send prompts and pay for usage. Authentication is API keys or Microsoft Entra ID (recommended, keyless). Private networking follows the Foundry resource public network access flag; inbound traffic can use a private endpoint on the hub or Foundry resource. Hubs that received a private endpoint or serverless deployments before 11 July 2024 may need a new private endpoint and a new serverless deployment before networking settings apply.

Inside Serverless API you pick a deployment type (SKU). The type controls where prompts are processed and how you pay:

• Global Standard (GlobalStandard). Pay-per-token. Prompts may be processed in any Azure region where the model is hosted. Highest default quota, newest models first, usually the lowest price. Start here unless compliance forbids it.
• Data Zone Standard (DataZoneStandard). Pay-per-token. Processing stays inside a Microsoft data zone: United States, European Union (the Azure EU Data Boundary, which can include EFTA countries such as Norway and Switzerland), or Asia Pacific. Microsoft can add regions to a zone without prior notice.
• Standard (Standard). Pay-per-token, processing pinned to the deployment's Azure region. Lower default quota; last to receive new models; no guaranteed availability date.
• Global Provisioned, Data Zone Provisioned, and Regional Provisioned. Reserved provisioned throughput units (PTUs) billed hourly. Same residency story as the matching Standard SKU. Section 11.4.
• Global Batch and Data Zone Batch. Asynchronous jobs, separate enqueued-token quota, about 50 percent cheaper than the matching Standard SKU, 24-hour target turnaround, no real-time SLA.
• Developer (DeveloperTier). Fine-tuned evaluation only. Fixed 24-hour lifetime, then automatic delete. No SLA and no data-residency guarantee. Not production. Section 11.3.

Data at rest stays in the designated Azure geography for every type. Inferencing data (prompts and completions) is what Global, Data Zone, and Regional types relocate. New models become available in a fixed order: Global, then Data Zone, then single-region. Do not block a launch on Regional Standard if Global Standard already meets residency.

Content safety for Serverless language models runs synchronously as the service processes prompts. You can disable filters at deploy time or later on the deployment details page. Embedding models and time-series models do not get built-in filters. If you call a model through a path other than the Azure OpenAI SDK with the Azure OpenAI v1 APIs, filters may stay off unless you add Azure AI Content Safety yourself. Filters are billed separately.

Create Serverless deployments in the Foundry portal, with Azure CLI, or with Bicep (Chapter 10). The control-plane caller needs a role such as Cognitive Services Contributor on the Foundry resource.

Managed compute: dedicated SKUs, hourly bills

Managed compute deploys model weights onto dedicated accelerators. You need this when the catalog lists only managed compute (typical for many Hugging Face models), when you bring custom weights, or when you need a specific serving runtime or GPU family.

Classic hub-based path. Use a hub-based project in Foundry portal (classic). Models arrive through Azure Machine Learning registries (weights, containers, evaluation and fine-tune pipelines, sample datasets). You deploy to Azure Machine Learning managed compute and call a REST API. Billing is virtual-machine core hours. You need virtual-machine quota for the product that runs the model. Some models offer a temporary shared quota for testing. Content safety is a separate Azure AI Content Safety call, billed on its own meter.

New Foundry managed compute (preview). A managed GPU platform-as-a-service on the unified Foundry project endpoint. Foundry picks GPUs from families such as A100 (80 GB), H100 (80 GB), and MI300X (192 GB) and runs Microsoft-curated vLLM, SGLang, or NVIDIA NIM containers. You size in model-centric terms rather than picking a virtual-machine SKU. Billing is hourly per accelerator SKU. Quota is granted per accelerator SKU per region through Foundry and is not Azure virtual-machine quota. Auto-scale and an idle timeout that scales to zero exist in preview. Microsoft documents this preview as without an SLA and not recommended for production. Content filtering is not available in the public preview.

If the exam names a Hugging Face model and a hub-based project, pick managed compute. If it names gpt-4.1, gpt-5.2, Claude, or another model sold by Azure with a Serverless SKU, pick Serverless API.

Exam scenario

A European bank wants a chat copilot on gpt-4.1 with Microsoft support, an Azure SLA, and processing that never leaves the EU Data Boundary. Traffic is bursty and still low. Create a Serverless API deployment of type Data Zone Standard in an EU data-zone region. Do not stand up A100s. Do not deploy the catalog model to an Azure Machine Learning managed online endpoint as if it were a registered MLflow artifact.

Common trap

Treating a Serverless API deployment as a Domain 2 managed online endpoint. Serverless has no instance_type, no instance_count, no Azure Machine Learning traffic-split map, and no virtual-machine bill. Another trap: forcing a Hugging Face catalog model onto Global Standard when the model card offers only managed compute. A third: using Developer or instant access as the production SKU. A fourth: assuming Global Standard keeps prompts in the resource region — it does not; only Data Zone or Regional types constrain processing.