8.1 Real-Time, Serverless, and Asynchronous Inference

Real-Time, Serverless, and Asynchronous Inference

Deploying machine learning models to production requires selecting an endpoint architecture that aligns precisely with business latency service level agreements (SLAs), payload data volumes, processing execution times, traffic volatility, and cost constraints. Amazon SageMaker provides three distinct online hosting options: Real-Time Inference, Serverless Inference, and Asynchronous Inference.

On the AWS Certified Machine Learning Engineer — Associate (MLA-C01) exam, you will be presented with architectural scenarios where you must evaluate trade-offs between persistent compute versus scale-to-zero compute, synchronous HTTP request-response patterns versus asynchronous decoupled queues, sub-second latency versus long-running (up to 1 hour) processing jobs, and small (6 MB) versus large (1 GB) payload limits.

---

1. Deep Dive: SageMaker Online Hosting Options

+--------------------------------------------------------------------------------------------------+
|                       SAGEMAKER ONLINE INFERENCE OPTIONS COMPARISON                              |
|                                                                                                  |
|   [1. REAL-TIME INFERENCE]                                                                       |
|   Client (HTTP POST) ===== Synchronous (<100ms) =====> [Persistent EC2/GPU Instances]           |
|   - Payload: <= 6 MB | Timeout: 60s | Persistent billing | Predictable ultra-low latency        |
|                                                                                                  |
|   [2. SERVERLESS INFERENCE]                                                                      |
|   Client (HTTP POST) ===== Synchronous (Managed) ====> [On-Demand Ephemeral MicroVMs]            |
|   - Payload: <= 4 MB | Memory: 1-6 GB | Auto-scale to 0 | Pay per ms | Cold starts on idle     |
|                                                                                                  |
|   [3. ASYNCHRONOUS INFERENCE]                                                                    |
|   Client ---> Uploads to S3 ---> Calls InvokeEndpointAsync ---> [Managed Internal Queue]        |
|                                                                         |                        |
|                                                                         v                        |
|   Client <--- SNS Notification <--- Output to S3 <--- [Worker Instances (Auto-Scale to 0)]       |
|   - Payload: <= 1 GB | Timeout: 15 min default, up to 1 hr | Queue | Scale-to-0 | CV/NLP        |
+--------------------------------------------------------------------------------------------------+

1.1 Real-Time Inference

Real-Time Inference is designed for workloads requiring immediate, sub-second response times and predictable low latency under steady or predictable continuous traffic.

• Compute Lifecycle: Deploys dedicated, persistent Amazon EC2 instances (e.g., ml.c6i.xlarge, ml.m5.2xlarge, ml.g5.2xlarge, ml.inf2.xlarge) managed by a SageMaker Endpoint Configuration.
• Invocation Protocol: Synchronous HTTP POST requests via the InvokeEndpoint API. The calling client holds the TCP connection open until the response payload is returned.
• Hard Architectural Limits:
  • Maximum Request Payload: 6 MB
  • Maximum Response Payload: 6 MB
  • Maximum Request Execution Timeout: 60 seconds
• High Availability & Auto-Scaling: Deploys instances across multiple Availability Zones (AZs). Endpoints scale dynamically based on CloudWatch metrics such as SageMakerVariantInvocationsPerInstance using Application Auto Scaling (Target Tracking, Step Scaling, or Scheduled Scaling).
• Cost Model: Billed hourly per instance provisioned, regardless of whether the endpoint processes requests or sits idle. Instances do not scale to zero.
• Best Suited For: Interactive consumer web/mobile applications, conversational AI chatbots, fraud detection scoring during financial checkout transactions, and real-time recommendation feeds.

1.2 Serverless Inference

Serverless Inference is a fully managed, compute-on-demand hosting option that automatically provisions compute capacity based on incoming traffic and scales down to zero instances when traffic ceases, eliminating idle infrastructure costs.

• Compute Lifecycle: Fully managed ephemeral micro-virtual machines. AWS handles provisioning, patching, load balancing, and scaling transparently.
• Invocation Protocol: Synchronous HTTP POST requests via InvokeEndpoint.
• Configuration Parameters:
  • Memory Size (MemorySizeInMB): Configurable in 1 GB increments: 1024 MB (1 GB), 2048 MB (2 GB), 3072 MB (3 GB), 4096 MB (4 GB), 5120 MB (5 GB), and 6144 MB (6 GB). Compute cores (vCPUs) and network bandwidth scale proportionally with the selected memory allocation.
  • Maximum Concurrency (MaxConcurrency): Specifies the maximum number of concurrent requests the serverless endpoint can process simultaneously (up to 200 per endpoint). Prevents unexpected billing spikes and manages downstream resource limits.
• Architectural Limits:
  • Maximum Request Payload: 4 MB
  • Maximum Response Payload: 4 MB
  • Maximum Request Timeout: 60 seconds
• The Cold-Start Latency Impact: When an endpoint transitions from an idle (zero instances) state to receiving an invocation, it experiences a cold start while AWS provisions container resources and loads model weights into memory. Cold-start latency can range from hundreds of milliseconds to tens of seconds depending on model artifact size.
• Cost Model: Billed per second of compute execution time (measured in milliseconds) plus data processed in GB. Zero cost incurred when idle.
• Best Suited For: Applications with intermittent or unpredictable traffic bursts, development/staging test environments, internal tools used only during business hours, and microservices with moderate latency tolerances.

1.3 Asynchronous Inference

Asynchronous Inference decouples request submission from inference processing by routing payloads through Amazon S3 and an internal SageMaker-managed queue (built on Amazon SQS). It is engineered for large payloads, long-running model processing, and cost-effective scale-to-zero processing of large jobs.

+--------------------------------------------------------------------------------------------------+
|                             ASYNCHRONOUS INFERENCE WORKFLOW                                      |
|                                                                                                  |
|   [1. Client]                                                                                    |
|        | 1. Uploads large payload (up to 1 GB)                                                   |
|        v                                                                                         |
|   [Amazon S3 Input Bucket] (s3://input/image_batch.raw)                                          |
|        ^                                                                                         |
|        | 2. Calls InvokeEndpointAsync(InputLocation='s3://input/...')                            |
|   [1. Client] ---------------------------------------------+                                     |
|                                                            |                                     |
|                                                            v                                     |
|                                             [SageMaker Managed Queue]                            |
|                                             (Queues & prioritizes requests)                      |
|                                                            |                                     |
|                                                            | 3. Auto-scales workers &            |
|                                                            |    pulls request from S3            |
|                                                            v                                     |
|                                             [Asynchronous Worker Fleet]                          |
|                                             (ml.g5.2xlarge - up to 15 mins)                      |
|                                                            |                                     |
|                                                            | 4. Writes model output              |
|                                                            v                                     |
|                                             [Amazon S3 Output Bucket]                            |
|                                             (s3://output/prediction.json)                        |
|                                                            |                                     |
|                                                            | 5. Emits completion event           |
|                                                            v                                     |
|   [Client / Lambda / App] <---------------- [Amazon SNS Topic]                                   |
+--------------------------------------------------------------------------------------------------+

Detailed Asynchronous Execution Lifecycle:

1. Payload Staging: The client uploads the request payload (up to 1 GB) to an Amazon S3 bucket.
2. Async Invocation: The client calls the InvokeEndpointAsync API, passing the InputLocation (S3 URI of the payload). The API immediately responds with an HTTP 202 Accepted status and an OutputLocation S3 URI.
3. Queue Ingestion: SageMaker places the request into an internal managed queue.
4. Worker Processing: Available worker instances pull requests from the queue, fetch the payload from S3, run model inference (default processing timeout of 15 minutes / 900 seconds, configurable up to 1 hour / 3,600 seconds per request via the InvocationTimeoutSeconds parameter), and write the resulting prediction to the specified S3 OutputLocation.
5. Completion Notification: Upon completion (or failure), SageMaker publishes an event notification to a configured Amazon Simple Notification Service (Amazon SNS) topic (e.g., SuccessTopic or ErrorTopic), allowing downstream microservices, AWS Lambda functions, or event-driven pipelines to react immediately.
6. Auto-Scaling to Zero: Asynchronous endpoints support scaling instance count down to 0 instances when no messages are present in the queue (ApproximateNumberOfMessagesVisible == 0). When a new asynchronous invocation arrives, SageMaker automatically scales the instance count from 0 to 1+ workers to process the queue.

---

2. Comprehensive Endpoint Architecture Comparison

---

3. SageMaker Inference Recommender

Selecting the right instance family and container configuration for real-time model hosting is often challenging due to complex interactions between model architecture, tensor concurrency, batching, and memory bandwidth.

SageMaker Inference Recommender is an automated benchmarking utility that deploys your model container into managed sandbox environments to simulate real-world inference load and recommend the optimal instance type based on your business requirements.

+--------------------------------------------------------------------------------------------------+
|                              SAGEMAKER INFERENCE RECOMMENDER                                     |
|                                                                                                  |
|   [Model Artifact + ECR Container Image + Sample Payload]                                        |
|                                 |                                                                |
|                                 v                                                                |
|                  [Inference Recommender Engine]                                                  |
|                                 |                                                                |
|        +------------------------+------------------------+                                       |
|        |                                                 |                                       |
|        v                                                 v                                       |
|   [Default Benchmark]                               [Advanced Benchmark]                         |
|   - Runs automated load test across                 - User specifies latency SLA (e.g. <50ms)    |
|     top candidate instance families                 - Custom traffic curves (P99, ramp-up)       |
|     (ml.c6i, ml.m6i, ml.g5, ml.inf2)                - Multi-model concurrency tuning             |
|        |                                                 |                                       |
|        +------------------------+------------------------+                                       |
|                                 |                                                                |
|                                 v                                                                |
|   [Recommendation Report: Latency (P95/P99) vs Throughput (RPS) vs Cost Per Inference ($/inv)]  |
+--------------------------------------------------------------------------------------------------+

Benchmarking Modes:

1. Default Recommendation Job: SageMaker automatically selects a curated matrix of instance families matching your model framework, executes load tests with sample payloads, and returns performance recommendations ranking instances by throughput, P95 latency, and cost per inference.
2. Advanced Recommendation Job: Allows ML engineers to provide explicit business constraints, including strict latency targets (e.g., P99 latency $\le 45\text{ ms}$), expected peak concurrency (e.g., 500 requests/sec), custom container environment variables, and traffic ramp-up schedules.

---

4. Python SDK Implementation Examples

Deploying Real-Time vs. Serverless vs. Asynchronous Endpoints

import boto3
import sagemaker
from sagemaker.pytorch import PyTorchModel
from sagemaker.serverless import ServerlessInferenceConfig
from sagemaker.async_inference import AsyncInferenceConfig

session = sagemaker.Session()
role = sagemaker.get_execution_role()

# Define the PyTorch Model
model = PyTorchModel(
    model_data="s3://ml-production-artifacts/resnet50/model.tar.gz",
    role=role,
    entry_point="inference.py",
    framework_version="2.1.0",
    py_version="py310",
    sagemaker_session=session
)

# Option 1: Deploy as Real-Time Endpoint (Persistent ml.g5.xlarge GPU)
realtime_predictor = model.deploy(
    initial_instance_count=2,
    instance_type="ml.g5.xlarge",
    endpoint_name="resnet50-realtime-endpoint"
)

# Option 2: Deploy as Serverless Endpoint (4 GB Memory, Max 50 Concurrency)
serverless_config = ServerlessInferenceConfig(
    memory_size_in_mb=4096,
    max_concurrency=50
)
serverless_predictor = model.deploy(
    serverless_inference_config=serverless_config,
    endpoint_name="resnet50-serverless-endpoint"
)

# Option 3: Deploy as Asynchronous Endpoint (S3 Queuing + Scale to Zero)
async_config = AsyncInferenceConfig(
    output_path="s3://ml-production-artifacts/async-output/",
    max_concurrent_invocations_per_instance=4,
    notification_config={
        "SuccessTopic": "arn:aws:sns:us-east-1:123456789012:inference-success-topic",
        "ErrorTopic": "arn:aws:sns:us-east-1:123456789012:inference-error-topic"
    }
)
async_predictor = model.deploy(
    initial_instance_count=1,
    instance_type="ml.g5.2xlarge",
    async_inference_config=async_config,
    endpoint_name="resnet50-async-endpoint"
)

[!TIP] Exam Rapid Decision Rules:

• If the question mentions sub-second latency SLA (<100ms) and interactive synchronous calls $\rightarrow$ Real-Time Endpoint.
• If the question mentions unpredictable/intermittent traffic, scale down to zero, memory < 6 GB, and cost optimization $\rightarrow$ Serverless Inference.
• If the question mentions payloads > 6 MB (up to 1 GB), processing times > 60s (default 15 min, up to 1 hour), SNS notifications, or scale-to-zero for large GPU workloads $\rightarrow$ Asynchronous Inference.