4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving

4.1 Differences and Advantages of Batch, Streaming, and Real-Time Serving

Deploying a trained machine learning model into production requires selecting an inference architecture that aligns with business latency constraints, throughput requirements, compute budgets, and operational complexity. On the Databricks Lakehouse Platform, practitioners deploy models across three primary serving paradigms: Batch Inference (Offline Scoring), Streaming Inference (Near-Real-Time / Event-Driven), and Real-Time Model Serving (Synchronous REST API).

Choosing the incorrect inference pattern introduces severe production failure modes, such as excessive cloud infrastructure costs from over-provisioning real-time endpoints for non-interactive workloads, or unacceptable user-facing latency when batch pipelines are misapplied to interactive applications.

---

The Machine Learning Serving Spectrum

Inference paradigms exist along a continuum defined by latency requirements, data arrival characteristics, and invocation triggers:

+---------------------------------------------------------------------------------------------------+
|                             DATABRICKS ML INFERENCE PARADIGM SPECTRUM                             |
|                                                                                                   |
|   BATCH INFERENCE                 STREAMING INFERENCE               REAL-TIME SERVING             |
|   (Offline / Scheduled)           (Near-Real-Time / Micro-batch)    (Interactive / Synchronous)   |
|   +-----------------------+       +-------------------------+       +-------------------------+   |
|   | - Trigger: Schedule   |       | - Trigger: Event Stream |       | - Trigger: HTTP Request |   |
|   | - Latency: Hours/Mins | ----> | - Latency: Secs/Mins    | ----> | - Latency: 10 - 50 ms   |   |
|   | - Throughput: TBs/PBs |       | - Throughput: Millions  |       | - Throughput: 1-10k RPS |   |
|   | - Target: Delta Lake  |       | - Target: Delta / Bus   |       | - Target: App Client    |   |
|   | - Engine: Spark Job   |       | - Engine: Structured St.|       | - Engine: Serverless EP |   |
|   +-----------------------+       +-------------------------+       +-------------------------+   |
|   LOWEST COST PER ROW                                                       LOWEST REQUEST LATENCY|
+---------------------------------------------------------------------------------------------------+

---

In-Depth Analysis of Inference Paradigms

Batch Inference (Offline Scoring)

Batch inference operates on static, accumulated datasets at scheduled intervals (hourly, daily, weekly). It is the most common and cost-effective serving pattern in enterprise machine learning.

• Execution Mechanics: A Databricks Workflow executes a scheduled Python or SQL job. The job reads a Delta Lake table, broadcasts or distributes the registered MLflow model across Spark worker nodes, evaluates predictions in parallel using mlflow.pyfunc.spark_udf or vectorized Pandas UDFs, and writes output predictions into a target Delta Lake gold table.
• Latency SLA: Asynchronous; minutes to hours. Predictions are generated long before downstream consumers access them.
• Throughput Capacity: Massive. Scales horizontally across hundreds of Spark executor cores to process billions of records or petabytes of data.
• Cost Efficiency: Lowest cost per prediction. Workloads utilize ephemeral automated job clusters that spin up on demand, execute vectorized SIMD/Arrow operations, and terminate immediately upon completion. Spot/preemptible instances can be leveraged safely because Delta ACID transactions ensure complete, recoverable writes.
• Failure Recovery: Idempotent job retries. If a node fails, Spark recomputes failed partition tasks; if the job fails, the target Delta table remains unaffected until the atomic transaction commits.
• Canonical Use Cases:
  • Nightly customer churn risk scoring for CRM marketing campaigns.
  • Weekly customer lifetime value (LTV) recalculations.
  • Daily retail demand forecasting across store-SKU hierarchies.
  • Monthly credit limit adjustment calculations for banking portfolios.

Streaming Inference (Near-Real-Time / Micro-Batch)

Streaming inference operates on continuous, unbounded data streams. It bridges the gap between batch efficiency and real-time responsiveness.

• Execution Mechanics: Spark Structured Streaming continuously ingests streaming events from message brokers (Apache Kafka, AWS Kinesis, Azure Event Hubs) or Delta Lake Change Data Feed (CDF). The stream applies an MLflow scoring UDF on micro-batches (e.g., every 5 to 30 seconds) or continuous event streams, writing scored results to downstream Delta sinks, message queues, or operational databases.
• Latency SLA: Sub-second to several minutes depending on trigger configuration (Trigger.ProcessingTime("5 seconds") or Trigger.AvailableNow()).
• Throughput Capacity: High and elastic. Automatically adapts to variable streaming ingestion volume.
• Cost Efficiency: Moderate. Requires continuously running cluster resources unless scheduled incrementally with Trigger.AvailableNow().
• Failure Recovery: Checkpoint-driven state recovery. The stream persists processed offsets and metadata to cloud object storage via checkpointLocation, guaranteeing exactly-once fault tolerance upon restarts.
• Canonical Use Cases:
  • Real-time financial transaction fraud detection on continuous card swipe streams.
  • IoT industrial sensor anomaly detection and alert triggering.
  • Real-time clickstream intent classification for dynamic web personalization.
  • Live delivery driver and customer order matching feeds.

Real-Time Model Serving (Synchronous REST API)

Real-time model serving exposes models as low-latency, highly available HTTPS endpoints. Predictions are generated synchronously on-demand in response to individual client requests.

• Execution Mechanics: Databricks Model Serving hosts models on managed Serverless compute. Client applications (mobile apps, web servers, backend microservices) issue HTTPS POST requests containing JSON feature payloads. The serving gateway routes requests to containerized model replicas, which compute predictions and return synchronous JSON responses.
• Latency SLA: Sub-second (typically 10 ms to 50 ms).
• Throughput Capacity: Requests are processed individually or in small micro-payloads (1 to 100 records). Throughput scales horizontally by adding container instances based on incoming Requests Per Second (RPS) and concurrent connections.
• Cost Efficiency: Highest cost per individual prediction. Serverless architecture optimizes costs through automatic scale-to-zero, shutting down compute instances when no traffic is detected.
• Failure Recovery: Client-side retries, exponential backoff, circuit breakers, multi-region failover, and static fallback heuristic responses.
• Canonical Use Cases:
  • Point-of-sale instant credit underwriting and loan decisioning.
  • Interactive e-commerce search query re-ranking and real-time recommendations.
  • Live chatbot conversational intent parsing and sentiment extraction.
  • Real-time medical triage diagnostic screening.

---

Comprehensive Architectural Comparison Matrix

The following matrix summarizes the technical, operational, and financial dimensions across the three paradigms:

---

Hybrid Architectures: Precomputation vs. Dynamic On-Demand Scoring

In complex enterprise systems, machine learning engineers frequently combine paradigms to optimize latency and compute expenditure.

+---------------------------------------------------------------------------------------------------+
|                                 HYBRID SERVING ARCHITECTURES                                      |
|                                                                                                   |
|   PATTERN A: BATCH PRECOMPUTATION + ONLINE CACHE LOOKUP                                           |
|   +------------+     Spark Batch    +------------+    Sync Push    +------------+  Sub-5ms GET    |
|   | Delta Lake | -----------------> | ML Scoring | --------------> | Key-Value  | <------------   |
|   | Gold Table |    (Nightly Run)   | Predictions|                 | Redis/KV   |   Web Client    |
|   +------------+                    +------------+                 +------------+                 |
|                                                                                                   |
|   PATTERN B: REAL-TIME SERVING + ONLINE FEATURE STORE ENRICHMENT                                  |
|                     +---------------------------------------+                                     |
|                     | Databricks Serverless Model Endpoint   |                                     |
|   +------------+    |   1. Receive {user_id, cart_items}     |     +--------------------------+    |
|   | Web Client | -> |   2. Fetch user historical features   | <-> | Unity Catalog Feature    |    |
|   | (HTTP POST)|    |   3. Compute dynamic score            |     | Store (Online Store)     |    |
|   +------------+    |   4. Return JSON {risk_score: 0.82}   |     +--------------------------+    |
|                     +---------------------------------------+                                     |
+---------------------------------------------------------------------------------------------------+

Pattern A: Batch Precomputation with Low-Latency Cache Store

• Mechanism: Compute predictions for all possible entities (e.g., every active customer ID) during an offline batch job. Push the {entity_id: prediction_value} pairs into a high-throughput, low-latency NoSQL database or cache (e.g., Redis, DynamoDB, Cosmos DB, or Unity Catalog Online Table).
• Advantages: Delivers sub-5ms lookup latency at request time without incurring the compute latency and cost of executing model forward passes during live user interactions.
• Limitations: Only viable when the entity universe is bounded and known in advance (e.g., existing registered users). Cannot incorporate real-time session context, dynamic user queries, or unseen feature values.

Pattern B: Real-Time Dynamic Scoring with Feature Store Joins

• Mechanism: The client sends an HTTP request containing lightweight transient keys (e.g., user_id, current_session_click). The serving endpoint automatically enriches the payload by fetching precomputed historical features (e.g., 90-day purchase frequency) from the Unity Catalog Online Feature Store, combines them with live request-time features, and executes model inference dynamically.
• Advantages: Supports dynamic inputs, unseen entity combinations, and real-time contextual signals while retaining access to complex historical features.

---

Architectural Decision Framework

When designing an inference pipeline on Databricks, evaluate these four foundational questions:

1. Is the prediction required synchronously to unblock a human user or live service?
   • Yes: Real-Time Model Serving (Serverless Endpoint).
   • No: Proceed to Question 2.
2. Does the input feature set originate from a continuous streaming message broker with a sub-minute latency requirement?
   • Yes: Streaming Inference (Spark Structured Streaming).
   • No: Proceed to Question 3.
3. Is the entity space small and predictable enough to precompute all combinations in advance?
   • Yes: Batch Precomputation + Online Key-Value Store.
   • No: Batch Inference directly to Delta Lake.