4.3 Streaming Inference with Delta Live Tables

4.3 Streaming Inference with Delta Live Tables

Streaming Inference with Spark Structured Streaming

Structured Streaming applies MLflow UDFs incrementally against continuous data streams with micro-batch processing.

Implementation with Checkpointing

import mlflow.pyfunc
from pyspark.sql.functions import struct, col

# 1. Register distributed scoring UDF
model_uri = "models:/enterprise_ml.iot.turbine_anomaly@champion"
anomaly_udf = mlflow.pyfunc.spark_udf(spark, model_uri=model_uri, result_type="double")

# 2. Read streaming DataFrame from Bronze ingestion stream
streaming_raw_df = spark.readStream \
    .format("delta") \
    .table("enterprise_ml.iot.bronze_sensor_telemetry")

# 3. Apply inference transformation
streaming_scored_df = streaming_raw_df.withColumn(
    "anomaly_score",
    anomaly_udf(struct("vibration_level", "rpm", "temperature", "pressure"))
).filter("anomaly_score > 0.85")  # Alert threshold

# 4. Write stream to Gold Delta sink with mandatory checkpointing
query = streaming_scored_df.writeStream \
    .format("delta") \
    .outputMode("append") \
    .option("checkpointLocation", "dbfs:/checkpoints/turbine_anomaly_scoring") \
    .trigger(processingTime="10 seconds") \
    .toTable("enterprise_ml.iot.gold_critical_anomalies")

The Critical Role of checkpointLocation

In Spark Structured Streaming, the checkpointLocation directory stores the write-ahead transaction log, offset metadata, and state information in durable cloud storage. If an executor or cluster restarts, the stream resumes from the exact recorded offset without re-scoring duplicate records or dropping events, guaranteeing exactly-once processing when combined with Delta Lake sinks.

---

Declarative ML Pipelines with Delta Live Tables (DLT)

Delta Live Tables (DLT) simplifies production ETL and inference by replacing manual stream and cluster management with declarative pipeline definitions.

import dlt
import mlflow.pyfunc
from pyspark.sql.functions import struct

# Broadcast model UDF inside DLT pipeline context
model_uri = "models:/supply_chain.logistics.delivery_delay_model@champion"
predict_delay = mlflow.pyfunc.spark_udf(spark, model_uri, result_type="double")

@dlt.table(
    name="silver_cleaned_shipments",
    comment="Ingests and validates raw logistics telemetry"
)
@dlt.expect_or_drop("valid_shipment_id", "shipment_id IS NOT NULL")
@dlt.expect_or_drop("valid_distance", "distance_miles > 0")
def silver_cleaned_shipments():
    return dlt.read_stream("bronze_shipment_events")

@dlt.table(
    name="gold_predicted_delays",
    comment="Applies MLflow model to forecast delivery delays"
)
@dlt.expect_or_fail("valid_prediction", "predicted_delay_minutes >= 0")
def gold_predicted_delays():
    features = ["distance_miles", "weather_severity", "traffic_index", "carrier_historical_delay"]
    return dlt.read("silver_cleaned_shipments").withColumn(
        "predicted_delay_minutes",
        predict_delay(struct(*features))
    )

---

Why DLT Rather Than a Hand-Written Stream

Both approaches apply the same MLflow UDF to the same events. The difference is how much operational machinery you own.

This is why a scenario that mentions fluctuating event volume, dynamic resizing, and declarative pipeline management points to DLT, while one that mentions a single bespoke stream with custom state handling points to Structured Streaming.

Streaming Tables vs. Materialised Views

Scoring an unbounded event feed should read as a stream, so the model is applied once per event rather than re-applied to the entire history on every update. That difference dominates both cost and latency at scale.

Expectations on Predictions, Not Just Inputs

Quality constraints are as valuable on the output of a model as on its input:

@dlt.table(name="gold_scored_events")
@dlt.expect("score_in_range", "anomaly_score BETWEEN 0 AND 1")
@dlt.expect_or_drop("has_entity", "device_id IS NOT NULL")
def gold_scored_events():
    return dlt.read_stream("silver_events").withColumn(
        "anomaly_score", predict_udf(struct(*feature_cols))
    )

• @dlt.expect records violations as pipeline metrics but keeps the rows — the right choice for monitoring prediction drift.
• @dlt.expect_or_drop filters violating rows out of the target table.
• @dlt.expect_or_fail aborts the pipeline update, reserved for violations that must never reach downstream consumers.

Because the violation counts are recorded per expectation and per update, a rising "score out of range" rate becomes an early warning that the model or its inputs have shifted — without a separate monitoring job.

The Serving-Endpoint Alternative Inside a Pipeline

A pipeline can also call a Model Serving endpoint from a UDF rather than loading the model into the stream. That is the right design when the model is very large, when it runs on a GPU, or when several consumers must share one governed deployment. It is the wrong design for high-volume event scoring, where a per-event HTTPS round trip adds latency and cost that in-pipeline UDF scoring avoids entirely.

The Name Changed; the Objective Did Not

The exam guide dated March 1, 2025 words this objective as "Identify how streaming inference is performed with Delta Live Tables," and its own sample question describes tens of thousands of events per second with fluctuating volume and compute that must resize dynamically. The keyed answer there is a Delta Live Tables pipeline applying the algorithm as a Spark UDF — precisely the tradeoff above. The pipeline supplies the autoscaling and the managed update semantics, while applying the model in process avoids an HTTPS round trip per event. Calling a serving endpoint from the pipeline would add a network hop to every event and force two components to scale independently where one suffices.

Since the 2025 Data + AI Summit, Databricks has folded Delta Live Tables into Lakeflow; the documentation now calls these Lakeflow pipelines, and the declarative framework itself was open-sourced as Spark Declarative Pipelines. Databricks states that no migration is required — existing DLT code still runs — and the concepts carry over directly: streaming tables, materialized views, expectations, and pipeline updates all mean what they meant before. In current Python, import dlt is superseded by from pyspark import pipelines as dp, where @dp.table creates a streaming table, @dp.materialized_view creates a materialized view, and the old @dlt.view becomes @dp.temporary_view. Expect the exam itself to use the older Delta Live Tables vocabulary, because that is the language of the live exam guide.