8.3 Multi-Model Endpoints (MME) & Multi-Container Endpoints (MCE)

Multi-Model Endpoints (MME) & Multi-Container Endpoints (MCE)

In enterprise machine learning deployments, organizations frequently face two distinct architectural challenges:

1. The Micro-Model Proliferation Problem: An organization needs to serve hundreds or thousands of specialized models (such as custom demand forecasts for 500 individual retail stores, or personalized recommendation models for 1,000 institutional clients). Deploying a dedicated real-time endpoint for every model creates astronomical compute costs and leaves provisioned instances underutilized.
2. The Multi-Step Pipeline Problem: Generating a prediction requires a sequence of distinct computational steps (e.g., step 1: data validation and scikit-learn feature encoding $\rightarrow$ step 2: XGBoost/PyTorch deep neural network inference $\rightarrow$ step 3: business rule post-processing and thresholding). Managing these as separate microservices introduces severe network latency and intermediate storage overhead.

Amazon SageMaker addresses these challenges with Multi-Model Endpoints (MME) and Multi-Container Endpoints (MCE) / Serial Inference Pipelines.

---

1. Multi-Model Endpoints (MME)

SageMaker Multi-Model Endpoints (MME) provide a scalable, cost-effective solution for deploying large numbers of models behind a single endpoint backed by a shared pool of compute instances.

+--------------------------------------------------------------------------------------------------+
|                             MULTI-MODEL ENDPOINT (MME) ARCHITECTURE                              |
|                                                                                                  |
|   [Amazon S3 Model Repository]                                                                   |
|   s3://my-bucket/models/                                                                         |
|   ├── store_001.tar.gz                                                                           |
|   ├── store_002.tar.gz                                                                           |
|   ├── store_003.tar.gz                                                                           |
|   └── ... (thousands of models under one S3 prefix)                                              |
|                                                                                                  |
|   Client Invocation:                                                                             |
|   runtime.invoke_endpoint(                                                                       |
|       EndpointName='retail-mme-endpoint',                                                        |
|       TargetModel='store_042.tar.gz',  <--- Specifies target model dynamically                  |
|       Body=payload                                                                               |
|   )                                                                                              |
|                                  |                                                               |
|                                  v                                                               |
|   +------------------------------------------------------------------------------------------+   |
|   |                      SHARED COMPUTE INSTANCE CLUSTER (MME)                               |   |
|   |                                                                                          |   |
|   |   +----------------------------------------------------------------------------------+   |   |
|   |   | Instance Memory (RAM / VRAM)                                                     |   |   |
|   |   | [Model 001 (Cached)]  [Model 042 (Cached)]  [Model 189 (Cached)]                 |   |   |
|   |   |                                                                                  |   |   |
|   |   | * Dynamic Model Loading: If TargetModel is not in RAM, downloads from S3         |   |   |
|   |   | * LRU Cache Eviction: When memory is full, evicts least recently used model      |   |   |
|   |   +----------------------------------------------------------------------------------+   |   |
|   +------------------------------------------------------------------------------------------+   |
+--------------------------------------------------------------------------------------------------+

1.1 How MME Works: Dynamic Loading & LRU Caching

1. S3 Model Storage: All model artifacts (model1.tar.gz, model2.tar.gz, ...) are stored under a common Amazon S3 bucket prefix (e.g., s3://ml-models-prod/retail-stores/).
2. Dynamic Invocations via TargetModel: When invoking the endpoint, the client passes the relative path of the desired model artifact in the HTTP request header TargetModel (or target_model='store_042.tar.gz' in the SDK).
3. On-Demand Loading:
   • Cache Hit: If the requested model is already resident in instance memory (RAM/VRAM), inference executes immediately with sub-second latency.
   • Cache Miss: If the model is not currently in memory, SageMaker dynamically downloads the .tar.gz artifact from Amazon S3 into local storage, loads weights into memory, and serves the prediction.
4. LRU Cache Eviction: When instance memory reaches saturation, SageMaker automatically unloads the Least Recently Used (LRU) model from memory to make room for new models.
5. Zero-Downtime Model Deployment: To deploy a new model or update an existing model, you simply upload the new artifact (store_501.tar.gz) to the S3 bucket prefix. No endpoint updates or CloudFormation redeployments are needed.

1.2 MME Hardware Support: CPU vs. GPU (NVIDIA Triton)

• CPU Multi-Model Endpoints: Supported across built-in and custom containers using SageMaker Multi-Model Server (MMS) or TorchServe (e.g., Scikit-learn, XGBoost, PyTorch, TensorFlow). All models hosted on a single CPU MME must use the same ML framework and container image.
• GPU Multi-Model Endpoints: Supported using the NVIDIA Triton Inference Server container on GPU instances (e.g., ml.g5, ml.p4d). Unlike CPU MME, Triton Inference Server on GPU supports multi-framework model serving (e.g., hosting PyTorch, ONNX, TensorRT, and TensorFlow models concurrently on the same GPU cluster) with dynamic GPU memory management.

---

2. Multi-Container Endpoints (MCE) & Serial Inference Pipelines

Multi-Container Endpoints (MCE) allow you to deploy up to 15 different Docker containers on a single SageMaker endpoint backed by shared compute instances.

+--------------------------------------------------------------------------------------------------+
|                       MULTI-CONTAINER ENDPOINTS (MCE) MODES                                      |
|                                                                                                  |
|   [MODE 1: DIRECT / INDEPENDENT INVOCATION]                                                      |
|   Client === (TargetContainer: 'container-B') ===> [Endpoint: Container A | Container B]         |
|   - Routes traffic directly to a specific container on the shared instance                       |
|                                                                                                  |
|   [MODE 2: SERIAL INFERENCE PIPELINE (SYNCHRONOUS CHAINING)]                                     |
|   Client (HTTP POST)                                                                             |
|        |                                                                                         |
|        v (Single Endpoint Call)                                                                  |
|   +------------------------------------------------------------------------------------------+   |
|   |                      SERIAL INFERENCE PIPELINE ENDPOINT                                  |   |
|   |                                                                                          |   |
|   |   +-------------------+       +-------------------+       +--------------------------+   |   |
|   |   | Container 1       |       | Container 2       |       | Container 3              |   |   |
|   |   | Scikit-learn      | ----> | XGBoost           | ----> | Business Post-Processor  |   |   |
|   |   | Feature Encoder   |       | ML Classifier     |       | Thresholding / Formatting|   |   |
|   |   +-------------------+       +-------------------+       +--------------------------+   |   |
|   |     (Raw CSV Input)           (Dense Vector)              (Probability Score)            |   |
|   +------------------------------------------------------------------------------------------+   |
|        |                                                                                         |
|        v                                                                                         |
|   Client <=== Returns Final Enriched Prediction (Sub-second, zero intermediate S3 writes)        |
+--------------------------------------------------------------------------------------------------+

2.1 Serial Inference Pipelines

A Serial Inference Pipeline is a specialized Multi-Container Endpoint where containers are chained sequentially in a defined linear graph (from 2 to 15 containers):

• Synchronous Execution: The client sends raw data to the endpoint. Container 1 processes the data and passes its output in-memory via local HTTP to Container 2. Container 2 performs model inference and forwards its output to Container 3, which generates the final response payload.
• Ultra-Low Latency: All container-to-container communication occurs over local UNIX sockets / localhost loops within the instance, eliminating intermediate Amazon S3 round-trips and external network hops.
• Separation of Concerns: Enables complete modularity between feature engineering code (e.g., Scikit-learn StringIndexer/OneHotEncoder), core model inference (e.g., XGBoost, PyTorch), and post-processing business logic.
• Deployment Simplicity: The entire multi-container sequence is treated as a single unified SageMaker PipelineModel, managed with a single endpoint name, a single endpoint configuration, and shared auto-scaling policies.

---

3. Python SDK Implementation Examples

3.1 Deploying a Multi-Model Endpoint (MME)

import boto3
import sagemaker
from sagemaker.multimodelmodel import MultiModelModel

session = sagemaker.Session()
role = sagemaker.get_execution_role()

# Common S3 prefix where all model.tar.gz artifacts reside
model_data_prefix = "s3://retail-ml-models-prod/store-forecasting/"

# Container image URI for XGBoost
xgb_image_uri = sagemaker.image_uris.retrieve(
    framework="xgboost",
    region=session.boto_region_name,
    version="1.7-1"
)

# Define the MultiModelModel
mme_model = MultiModelModel(
    model_data_prefix=model_data_prefix,
    image_uri=xgb_image_uri,
    role=role,
    sagemaker_session=session
)

# Deploy to shared instance cluster
mme_predictor = mme_model.deploy(
    initial_instance_count=2,
    instance_type="ml.m5.2xlarge",
    endpoint_name="retail-stores-mme-endpoint"
)

# Invoking a specific model via TargetModel
runtime = boto3.client("sagemaker-runtime")
response = runtime.invoke_endpoint(
    EndpointName="retail-stores-mme-endpoint",
    TargetModel="store_042.tar.gz",
    ContentType="text/csv",
    Body="45, 120.5, 0.8, 12"
)
print(response["Body"].read().decode("utf-8"))

3.2 Deploying a Serial Inference Pipeline

from sagemaker.pipeline import PipelineModel
from sagemaker.sklearn.model import SKLearnModel
from sagemaker.xgboost.model import XGBoostModel

# 1. Container 1: Pre-processing (Scikit-Learn Feature Transformer)
sklearn_preprocessor = SKLearnModel(
    model_data="s3://ml-artifacts/pipeline/preprocessor.tar.gz",
    role=role,
    entry_point="preprocessor.py",
    framework_version="1.2-1",
    sagemaker_session=session
)

# 2. Container 2: Model Inference (XGBoost Classifier)
xgb_model = XGBoostModel(
    model_data="s3://ml-artifacts/pipeline/xgboost_model.tar.gz",
    role=role,
    entry_point="inference.py",
    framework_version="1.7-1",
    sagemaker_session=session
)

# 3. Create Serial Inference Pipeline (Chaining Containers 1 and 2)
pipeline_model = PipelineModel(
    name="customer-churn-serial-pipeline",
    role=role,
    models=[sklearn_preprocessor, xgb_model],  # Executed in exact array order
    sagemaker_session=session
)

# Deploy as a single real-time endpoint
pipeline_predictor = pipeline_model.deploy(
    initial_instance_count=2,
    instance_type="ml.m5.xlarge",
    endpoint_name="churn-serial-pipeline-endpoint"
)

---

4. Master Decision Framework: Choosing the Right Hosting Architecture

+--------------------------------------------------------------------------------------------------+
|                          SAGEMAKER INFERENCE ARCHITECTURE DECISION TREE                          |
|                                                                                                  |
|   START: What is the primary operational requirement?                                            |
|                                                                                                  |
|   [Offline scoring over static datasets?]                                                        |
|         |---> YES: BATCH TRANSFORM (Ephemeral cluster, S3-in/S3-out, DataProcessing filters)     |
|                                                                                                  |
|   [Hundreds of models with similar frameworks & low individual traffic?]                         |
|         |---> YES: MULTI-MODEL ENDPOINT (MME) (Shared instances, S3 prefix, LRU cache)           |
|                                                                                                  |
|   [Chaining feature pre-processing, ML inference, and post-processing synchronously?]             |
|         |---> YES: SERIAL INFERENCE PIPELINE (2-15 containers, zero S3 intermediate hops)       |
|                                                                                                  |
|   [Intermittent/unpredictable traffic, payload < 4 MB, timeout < 60s, scale to zero?]            |
|         |---> YES: SERVERLESS INFERENCE (1-6 GB RAM, pay per ms, cold starts)                    |
|                                                                                                  |
|   [Payloads > 6 MB (up to 1 GB), processing times > 60s (up to 1 hr), scale to 0, SNS events?]    |
|         |---> YES: ASYNCHRONOUS INFERENCE (S3 request routing, managed SQS queue)                |
|                                                                                                  |
|   [Strict sub-second latency SLA (<100ms), 24/7 continuous traffic, payload <= 6 MB?]            |
|         |---> YES: REAL-TIME INFERENCE (Dedicated EC2/GPU instances, multi-AZ)                  |
+--------------------------------------------------------------------------------------------------+

[!TIP] Key Takeaway for MLA-C01:

• Whenever you see "hundreds of localized/personalized models" and "minimize compute costs on shared instances" $\rightarrow$ Select Multi-Model Endpoints (MME).
• Whenever you see "feature transformation + model inference in a single low-latency call without intermediate storage" $\rightarrow$ Select Serial Inference Pipeline.