4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints

4.5 Deploying, Querying, and Splitting Traffic Across Real-Time Endpoints

Serverless Real-Time Endpoint Architecture

When a model from the Unity Catalog Model Registry is deployed to a serving endpoint, Databricks packages the model artifacts, its logged code environment, and its Python dependencies into managed serverless containers sitting behind an authenticated inference gateway.

+---------------------------------------------------------------------------------------------------+
|                         DATABRICKS REAL-TIME MODEL SERVING ARCHITECTURE                           |
|                                                                                                   |
|   EXTERNAL CLIENTS                                                                                |
|   (Web Apps, Mobile Backends, Microservices)                                                      |
|        |                                                                                          |
|        | HTTPS POST /serving-endpoints/{endpoint-name}/invocations                                |
|        | Headers: Authorization: Bearer <token>, Content-Type: application/json                    |
|        v                                                                                          |
|   +-------------------------------------------------------------------------------------------+   |
|   | MANAGED INFERENCE GATEWAY & ROUTER (Load Balancer & Auth Verification)                    |   |
|   +-------------------------------------------------------------------------------------------+   |
|        |                                                     |                                    |
|        | Traffic Split: 90% (Champion)                       | Traffic Split: 10% (Challenger)    |
|        v                                                     v                                    |
|   +---------------------------------------+     +---------------------------------------+         |
|   | SERVED ENTITY 1: @champion            |     | SERVED ENTITY 2: @challenger          |         |
|   | - Container Replicas (Auto-scaled)    |     | - Container Replicas (Auto-scaled)    |         |
|   | - Workload Size: Medium (CPU/GPU)     |     | - Workload Size: Small (CPU/GPU)      |         |
|   | - Scale-to-Zero when idle             |     | - Scale-to-Zero when idle             |         |
|   +---------------------------------------+     +---------------------------------------+         |
|        |                                                     |                                    |
|        +--------------------------+--------------------------+                                    |
|                                   | Asynchronous Payload Streaming                                |
|                                   v                                                               |
|   +-------------------------------------------------------------------------------------------+   |
|   | UNITY CATALOG INFERENCE TABLE (Delta Lake Managed Table)                                  |
|   | - request_payload | response_payload | latency_ms | model_name | model_version | timestamp  |   |
|   +-------------------------------------------------------------------------------------------+   |
+---------------------------------------------------------------------------------------------------+

Autoscaling & Scale-to-Zero

Databricks Model Serving dynamically monitors incoming request concurrency and latency metrics:

• Scale-Up: When request concurrency surges, the serving control plane provisions additional container replicas across available data plane capacity within seconds to maintain sub-50ms latency SLOs.
• Scale-to-Zero: When an endpoint experiences zero incoming traffic for a configurable cooldown window, the serving engine terminates active compute container instances, reducing compute cost to $0.00 while keeping the endpoint URL active. When a new request arrives, the endpoint automatically cold-starts.

Workload Sizing & Compute Configuration

When deploying a model, practitioners configure the workload size per served entity: Workload size sets the endpoint's provisioned concurrency — the number of requests it can process at once, since one unit of concurrency serves one request at a time:

When scale_to_zero_enabled is set, the lower bound of each range drops to 0 during idle periods. GPU-backed workload types are configured separately for deep learning frameworks such as PyTorch, TensorFlow, and Hugging Face Transformers.

---

Multi-Model Traffic Splitting & Rollout Strategies

Production deployments require rigorous risk mitigation when introducing new model versions. Databricks Model Serving enables multi-entity routing behind a single endpoint URL.

+---------------------------------------------------------------------------------------------------+
|                             PROGRESSIVE TRAFFIC ROLLOUT PATTERNS                                  |
|                                                                                                   |
|   PHASE 1: CANARY VALIDATION          PHASE 2: A/B EXPERIMENTATION        PHASE 3: FULL CUTOVER   |
|   +---------------------------+       +---------------------------+       +-------------------+   |
|   | Champion (@prod):     90% |       | Champion (@prod):     50% |       | New Model (@prod):100%|   |
|   | Challenger (@canary): 10% | ----> | Challenger (@exp):    50% | ----> | Old Model:          0%|   |
|   | Assess error rates/latency|       | Measure business KPIs     |       | (Decommissioned)  |   |
|   +---------------------------+       +---------------------------+       +-------------------+   |
+---------------------------------------------------------------------------------------------------+

Deployment Strategies

1. Canary Deployment: Route 5%–10% of live traffic to a newly registered candidate model while routing 90%–95% to the existing production baseline. Engineers monitor latency, memory consumption, and error rates before expanding traffic.
2. A/B Testing: Route 50% of traffic to Model A and 50% to Model B. Downstream business metrics (e.g., click-through rate, conversion rate) are tracked via Inference Tables to evaluate statistical superiority.
3. Champion / Challenger Rollout: Models are tagged in Unity Catalog using model aliases (@champion and @challenger). The endpoint configuration references these aliases directly, enabling zero-downtime weight adjustments.

Endpoint Configuration Example via Python SDK

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.serving import EndpointCoreConfigInput, ServedEntityInput

w = WorkspaceClient()

# Create or update a serving endpoint with 90/10 traffic splitting
w.serving_endpoints.create_and_wait(
    name="customer-churn-serving-endpoint",
    config=EndpointCoreConfigInput(
        served_entities=[
            ServedEntityInput(
                name="churn-champion",
                entity_name="prod_ml.customer_churn.churn_model",
                entity_version="4",  # Or Unity Catalog alias
                workload_size="Small",
                scale_to_zero_enabled=True
            ),
            ServedEntityInput(
                name="churn-challenger",
                entity_name="prod_ml.customer_churn.churn_model",
                entity_version="5",
                workload_size="Small",
                scale_to_zero_enabled=True
            )
        ],
        traffic_config={
            "routes": [
                {"served_model_name": "churn-champion", "traffic_percentage": 90},
                {"served_model_name": "churn-challenger", "traffic_percentage": 10}
            ]
        },
        auto_capture_config={
            "catalog_name": "prod_ml",
            "schema_name": "monitoring",
            "table_name_prefix": "churn_endpoint"
        }
    )
)

---

How Traffic Splitting Actually Works

A single endpoint can host multiple served entities — usually two versions of the same registered model. The endpoint's traffic_config assigns each route a percentage of incoming requests, and the gateway distributes calls accordingly.

traffic_config={
    "routes": [
        {"served_model_name": "churn-champion",   "traffic_percentage": 90},
        {"served_model_name": "churn-challenger", "traffic_percentage": 10},
    ]
}

Rules the exam tests:

• Percentages must sum to 100. A configuration totalling 90 or 110 is rejected.
• Splitting happens within one endpoint, behind one URL. Clients are unaware of it and need no change to participate in a canary.
• Shifting traffic is a configuration update, not a redeployment. Moving from 90/10 to 50/50 to 0/100 updates the endpoint config; the model containers are not rebuilt.
• Rollback is the same operation in reverse. Returning the champion to 100% is immediate, which is precisely why canary rollout is safer than an in-place swap.
• Attribution requires logging. To compare the two entities you must know which served the request; inference tables record that in request_metadata.

Canary vs. A/B vs. blue-green

A canary answers "is the new version broken?"; an A/B test answers "is the new version better?". The traffic percentage follows from which question is being asked.

---

REST API Invocation Formats & Client Integration

Databricks Model Serving endpoints accept standard JSON structures via HTTP POST requests.

Supported Request Formats

1. dataframe_split (Recommended for tabular data with column names):

{
  "dataframe_split": {
    "columns": ["age", "tenure", "monthly_charges", "support_tickets"],
    "data": [
      [34, 12, 65.50, 1],
      [52, 48, 110.00, 4]
    ]
  }
}

1. dataframe_records (Array of key-value objects):

{
  "dataframe_records": [
    {"age": 34, "tenure": 12, "monthly_charges": 65.50, "support_tickets": 1},
    {"age": 52, "tenure": 48, "monthly_charges": 110.00, "support_tickets": 4}
  ]
}

1. instances (List of feature vectors):

{
  "instances": [
    [34, 12, 65.50, 1],
    [52, 48, 110.00, 4]
  ]
}

1. inputs (Columnar dictionary of arrays or tensor dictionaries):

{
  "inputs": {
    "age": [34, 52],
    "tenure": [12, 48],
    "monthly_charges": [65.50, 110.00],
    "support_tickets": [1, 4]
  }
}

cURL Invocation Example

curl -X POST https://<databricks-instance>/serving-endpoints/customer-churn-serving-endpoint/invocations \
  -H "Authorization: Bearer dapi1234567890abcdef" \
  -H "Content-Type: application/json" \
  -d '{
    "dataframe_split": {
      "columns": ["age", "tenure", "monthly_charges", "support_tickets"],
      "data": [[45, 24, 89.99, 2]]
    }
  }'

Python Requests Client with Error Handling

import requests
import json
import time

DATABRICKS_HOST = "https://adb-123456789.databricks.com"
TOKEN = "dapi_your_auth_token_here"
ENDPOINT_NAME = "customer-churn-serving-endpoint"

url = f"{DATABRICKS_HOST}/serving-endpoints/{ENDPOINT_NAME}/invocations"
headers = {
    "Authorization": f"Bearer {TOKEN}",
    "Content-Type": "application/json"
}

payload = {
    "dataframe_split": {
        "columns": ["age", "tenure", "monthly_charges", "support_tickets"],
        "data": [[45, 24, 89.99, 2], [28, 6, 45.00, 0]]
    }
}

start_time = time.time()
response = requests.post(url, headers=headers, json=payload, timeout=5.0)
latency_ms = (time.time() - start_time) * 1000

if response.status_code == 200:
    predictions = response.json()["predictions"]
    print(f"Predictions: {predictions} (Latency: {latency_ms:.2f}ms)")
else:
    print(f"Invocation Failed [{response.status_code}]: {response.text}")

---

Unity Catalog Inference Tables (Request-Response Logging)

Inference Tables provide automated, asynchronous payload logging for all incoming requests and outgoing model responses.

Inference Table Architecture

When auto_capture_config is enabled on an endpoint, Databricks automatically provisions managed Delta Lake tables in the specified Unity Catalog schema:

• <catalog>.<schema>.<prefix>_payload: one row per request, with these columns:

• A batch of inputs sent in one call is logged as a single row, so unpacking the request JSON is required before per-record analysis.
• Logging is asynchronous and runs outside the inference path, so it does not add to client-visible latency.

Key Use Cases for Inference Tables

1. Data & Concept Drift Monitoring: Databricks Lakehouse Monitoring can be attached directly to the Inference Table to track statistical distribution drift between training and serving features over time.
2. Auditability & Compliance: Maintains an immutable historical record of exact inputs and model outputs for financial, healthcare, and legal audits.
3. Ground Truth Joining for Model Evaluation: Downstream ETL pipelines join inference table payloads with delayed ground-truth labels (e.g., actual customer churn 30 days later) to calculate live precision, recall, and ROC-AUC.

Querying Inference Tables with PySpark SQL

# Read and unpack JSON payloads from Unity Catalog Inference Table
raw_inference_df = spark.table("prod_ml.monitoring.churn_endpoint_payload")

# Unpack request and response fields
unpacked_df = raw_inference_df.selectExpr(
    "timestamp_ms",
    "date",
    "status_code",
    "execution_time_ms",
    "request_metadata",
    "from_json(request, 'struct<dataframe_split:struct<columns:array<string>,data:array<array<double>>>>') as req",
    "from_json(response, 'struct<predictions:array<double>>') as resp"
)

display(unpacked_df)

---

Security & Access Governance

Access to Databricks Model Serving endpoints is strictly governed through Unity Catalog and Workspace Access Control Lists (ACLs):

• CAN_QUERY Permission: Grants the ability to execute HTTPS POST inference requests against the endpoint URL. Given to client application service principals and BI tools.
• CAN_MANAGE Permission: Grants full administrative control to update endpoint configurations, modify served models, adjust traffic split percentages, resize compute, and delete the endpoint. Reserved for ML engineers and CI/CD service principals.
• CAN_VIEW Permission: Grants read-only visibility to inspect endpoint configuration, health status, and latency metrics in the Databricks UI.