7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion

7.4 Structured Streaming Fundamentals & Azure Event Hubs / Kafka Ingestion

DP-750 Exam Focus: Understand the core architecture of Spark Structured Streaming (the unbounded table model, readStream, writeStream, and checkpointing). Master configuring real-time ingestion from Azure Event Hubs using the Kafka interface, securing connection credentials via Databricks Secret Scopes, deserializing binary payloads (key, value) with from_json(), and managing offsets and rate limits (maxOffsetsPerTrigger).

---

1. Structured Streaming Processing Model

Apache Spark Structured Streaming is a scalable and fault-tolerant stream processing engine built upon the Catalyst query optimization and Photon execution engines.

Rather than treating streaming data as discrete low-level RDD batches (the legacy DStream model), Structured Streaming models a real-time data stream as an Unbounded Table to which incoming records are continuously appended.

+-----------------------------------------------------------------------------------+
|                     STRUCTURED STREAMING UNBOUNDED TABLE MODEL                    |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  Time   Incoming Stream Records                Unbounded Input Table              |
|  ----   -----------------------                ---------------------              |
|  t0:    [Row 1, Row 2]                  --->   [Row 1, Row 2]                     |
|  t1:    [Row 3]                         --->   [Row 1, Row 2, Row 3]              |
|  t2:    [Row 4, Row 5]                  --->   [Row 1, Row 2, Row 3, Row 4, Row 5]|
|                                                                                   |
|                                    |                                              |
|                                    v                                              |
|                   [ Incremental Query Execution Plan ]                            |
|                   - Catalyst Logical Optimization                                 |
|                   - State Store Management (RocksDB)                              |
|                                    |                                              |
|                                    v                                              |
|                   [ Result Table / Target Delta Sink ]                            |
+-----------------------------------------------------------------------------------+

The readStream and writeStream Pipeline Lifecycle

Every Structured Streaming query follows a standardized programmatic pattern:

1. spark.readStream: Defines the streaming DataFrame reader attached to an unbounded streaming source (e.g., Kafka, Event Hubs, Auto Loader, or Delta Lake table).
2. Transformations: Stateless transformations (projections, filters, column additions) or stateful operations (windowed aggregations, stream-stream joins).
3. df.writeStream: Configures the output sink, output mode (append, complete, update), trigger interval, and mandatory checkpointLocation.
4. .start(): Initiates the background streaming execution thread and returns a StreamingQuery control handle.

---

2. Ingesting from Azure Event Hubs via Kafka Interface

Microsoft Azure Event Hubs exposes an Apache Kafka 1.0+ compatible endpoint, allowing Azure Databricks to ingest real-time events using the built-in, highly optimized Spark Kafka connector (format("kafka")). This avoids third-party JAR dependencies and leverages native Spark streaming optimizations.

                     AZURE EVENT HUBS INGESTION FLOW

  +-------------------------------------------------------------------+
  | Azure Event Hubs Namespace: eh-telemetry-prod.servicebus.windows.net|
  | Event Hub (Topic): iot-device-events                              |
  +-------------------------------------------------------------------+
                                   |
                                   | (SASL_SSL / Port 9093)
                                   v
  +-------------------------------------------------------------------+
  | Azure Databricks Cluster (Spark Structured Streaming)             |
  | - format("kafka")                                                 |
  | - dbutils.secrets.get("azure-eh", "connection-string")            |
  | - Kafka JAAS Configuration                                        |
  +-------------------------------------------------------------------+
                                   |
                                   v
  +-------------------------------------------------------------------+
  | Bronze Delta Lake Table: bronze.iot.telemetry_raw                 |
  +-------------------------------------------------------------------+

Security & Connection Configuration

Connecting securely to Azure Event Hubs requires authenticating via SASL_SSL with a Shared Access Signature (SAS) connection string stored securely in a Databricks Secret Scope (backed by Azure Key Vault):

# Step 1: Retrieve Event Hubs connection details from Databricks Secret Scope
eh_namespace = "eh-telemetry-prod.servicebus.windows.net:9093"
eh_topic = "iot-device-events"
eh_conn_str = dbutils.secrets.get(scope="azure-keyvault-scope", key="eventhubs-conn-string")

# Step 2: Construct the Kafka JAAS configuration string
# Note: $ConnectionString is the required Kafka username for Azure Event Hubs
jaas_config = f'org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="{eh_conn_str}";'

# Step 3: Configure Spark Structured Streaming readStream
df_eventhubs_raw = (spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", eh_namespace)
    .option("subscribe", eh_topic)
    .option("kafka.security.protocol", "SASL_SSL")
    .option("kafka.sasl.mechanism", "PLAIN")
    .option("kafka.sasl.jaas.config", jaas_config)
    .option("startingOffsets", "latest") # Options: 'earliest', 'latest', or JSON offset map
    .option("maxOffsetsPerTrigger", 10000) # Rate limiting: max messages per micro-batch
    .option("failOnDataLoss", "false") # Resilient to Event Hubs message retention truncation
    .load())

Important Exam Fact: When connecting to Azure Event Hubs using the Kafka protocol, the username in the JAAS config is always literally "$ConnectionString", and the password is the complete Event Hubs Primary Connection String.

---

3. Message Deserialization & Binary Payload Processing

When reading from Kafka or Azure Event Hubs, Spark produces a DataFrame with a fixed binary schema:

Deserializing JSON Payloads

Because the value column is stored as raw binary bytes, data engineers must cast it to STRING and apply from_json() with a predefined StructType schema to unpack the attributes into structured columns:

from pyspark.sql.functions import col, from_json
from pyspark.sql.types import StructType, StructField, StringType, DoubleType, LongType, TimestampType

# Define payload schema
payload_schema = StructType([
    StructField("device_id", StringType(), True),
    StructField("temperature", DoubleType(), True),
    StructField("humidity", DoubleType(), True),
    StructField("reading_time", TimestampType(), True)
])

# Unpack binary payload and extract metadata
df_parsed = (df_eventhubs_raw
    .select(
        col("topic"),
        col("partition"),
        col("offset"),
        col("timestamp").alias("enqueued_time"),
        from_json(col("value").cast("string"), payload_schema).alias("data")
    )
    .select(
        "topic",
        "partition",
        "offset",
        "enqueued_time",
        "data.*"
    ))

---

4. Offset Management, Rate Limiting, & Backpressure

Offset Configurations on Startup

• startingOffsets = 'latest': Ingests only new messages that arrive after the query starts. Historical messages currently in the Event Hub partitions are skipped.
• startingOffsets = 'earliest': Ingests all available historical messages retained within the Event Hub's retention window (e.g., past 1 to 7 days), starting from offset 0.
• Specific Offset Map: Ingests from exact per-partition offsets provided in a JSON string (e.g., {"iot-device-events":{"0":14200,"1":18950}}).

Exam Note: startingOffsets only takes effect when the stream starts for the very first time without an existing checkpoint directory. If a checkpoint directory already exists, Spark resumes from the persisted offsets in the checkpoint, ignoring startingOffsets.

Backpressure & Rate Limiting

• maxOffsetsPerTrigger: Restricts the maximum number of records read across all partitions in a single micro-batch. Setting maxOffsetsPerTrigger = 50000 prevents large backlogs from causing driver out-of-memory errors and maintains predictable batch durations.
• minOffsetsPerTrigger: Specifies a minimum number of records required before triggering a micro-batch, avoiding micro-batches on low-volume streams.

---

5. Checkpointing & Exactly-Once Ingestion Pipeline

To achieve end-to-end exactly-once guarantees, Structured Streaming pairs offset tracking with Delta Lake's ACID transaction log:

# Production Pipeline: Writing parsed stream to Bronze Delta Table
query = (df_parsed.writeStream
    .format("delta")
    .outputMode("append")
    .option("checkpointLocation", "abfss://checkpoints@myadls.dfs.core.windows.net/iot_stream_cp")
    .trigger(processingTime="10 seconds")
    .toTable("bronze.iot.telemetry_events"))

Checkpoint Anatomy & State Recovery

• offsets/: Contains the starting and ending partition offsets for each micro-batch (e.g., Partition 0: offsets 100-200).
• commits/: Contains an atomic commit marker written only after the target Delta Lake table successfully commits the batch.
• Failure Recovery Protocol: If a cluster node crashes mid-batch, upon restart the driver inspects commits/. If the last batch in offsets/ lacks a matching entry in commits/, Spark re-executes that exact offset range. Because Delta Lake uses transactional idempotence, replaying the micro-batch overwrites uncommitted data, ensuring zero duplicates and zero data loss.