12.1 Streaming Architecture, Sources, and Sinks

Apache Spark's Structured Streaming engine revolutionized real-time data processing by integrating stream processing directly into the Spark SQL ecosystem. Rather than requiring developers to reason about low-level streaming abstractions (such as receivers, DStreams, and sliding RDD intervals), Structured Streaming introduces a declarative, high-level DataFrame API that treats real-time data ingestion as a continuous query running against an ever-growing, unbounded table.

For the Databricks Certified Associate Developer exam, you must master the fundamental mechanics of Structured Streaming: how queries are conceptualized against unbounded tables, how the underlying execution engines operate, how streaming data sources are configured and schema-enforced, and how processed records are dispatched to production storage sinks.

The Unbounded Table Model

The central architectural abstraction in Structured Streaming is the Unbounded Table. In this mental model, incoming data events are viewed as new rows continuously being appended to an input table with an infinite number of rows.

=========================================================================
                      STRUCTURED STREAMING MODEL
=========================================================================

   Incoming Stream: [Event 1] [Event 2] [Event 3] [Event 4] ...
                           |
                           v
+-----------------------------------------------------------------------+
|                         UNBOUNDED INPUT TABLE                         |
|-----------------------------------------------------------------------|
| Time | Device ID | Temperature | Status                               |
|------|-----------|-------------|--------------------------------------|
| 10:01| sensor-A  | 68.4        | OK                                   |
| 10:02| sensor-B  | 72.1        | OK                                   |
| 10:03| sensor-A  | 85.0        | WARNING   <-- Appended as new row    |
+-----------------------------------------------------------------------+
                           |
             [ Continuous Query Computation ]
                           |
                           v
+-----------------------------------------------------------------------+
|                             RESULT TABLE                              |
|-----------------------------------------------------------------------|
| Device ID | Max Temperature | Total Events                            |
|-----------|-----------------|-----------------------------------------|
| sensor-A  | 85.0            | 2                                       |
| sensor-B  | 72.1            | 1                                       |
+-----------------------------------------------------------------------+
                           |
                           v (Trigger Interval)
+-----------------------------------------------------------------------+
|                             OUTPUT SINK                               |
|   (Delta Lake / Kafka / Cloud Files / JDBC via foreachBatch)          |
+-----------------------------------------------------------------------+

Core Properties of the Model

1. Unified Batch and Streaming API: Developers write queries against streaming DataFrames using the identical DataFrame, Dataset, and SQL expressions used for static batch data. You can filter, project, aggregate, and join data using familiar operations like df.filter(), df.groupBy(), and df.select().
2. Catalyst Optimizer Integration: Streaming queries are parsed, analyzed, optimized, and compiled into physical execution plans by Spark's Catalyst Optimizer and run through Project Tungsten code generation. The engine determines whether transformations can be pipelined or require stateful tracking.
3. Incremental Execution: At each trigger interval, Spark checks the streaming source for newly arrived data, computes the delta against the previous state, updates the Result Table, and writes the output to the target sink.

Processing Engines: Micro-Batch vs. Continuous Processing

Structured Streaming supports two distinct execution engines under the hood:

1. Micro-Batch Processing Engine (Default)

The micro-batch engine is the default, battle-tested execution model in Apache Spark:

• Latency Profile: Can reach low latencies on suitable workloads, but trigger interval, source, state and sink costs determine actual end-to-end latency.
• How It Operates: The coordinator periodically queries the source for available offsets, batches newly arrived records together, constructs a standard Spark physical execution DAG, and executes the micro-batch across distributed executor task slots.
• Strengths: Maximum throughput, excellent resource utilization, robust dynamic autoscaling, and support for many relational and stateful operations, subject to streaming-specific restrictions on output modes, joins and chained stateful operations. Arbitrary batch SQL operations are not all supported on an unbounded stream.

2. Continuous Processing Engine (Low-Latency Specialized)

Introduced in Spark 2.3, the Continuous Processing engine can achieve latency as low as about 1 ms in suitable cases; this is not a guaranteed service level:

• Latency Profile: Low-latency long-running processing; measure actual latency.
• How It Operates: Rather than launching discrete tasks per micro-batch, the driver launches long-running tasks on executors that continuously poll the source and process records immediately. Epoch-based checkpointing (using an asynchronous Chandy-Lamport variant) coordinates snapshot commits at fixed intervals.
• Critical Limitations (High Exam Probability):
  • Supports only map-like operations: selections (select), projections, filters (where), and simple mapping functions.
  • Does NOT support aggregations (no groupBy, count, sum).
  • Does NOT support watermarking or windowing.
  • Does NOT support stream-stream joins or full outer joins.
  • Supports only a subset of sources (Kafka, rate) and sinks (Kafka, memory, console).

Streaming Sources (spark.readStream)

Streaming queries begin by invoking spark.readStream, which returns a DataStreamReader instance.

1. File Sources (JSON, CSV, Parquet, ORC, Delta Lake)

Spark can continuously monitor a cloud object storage directory (e.g., S3, ADLS Gen2, DBFS) or distributed file system for newly written files.

# Reading streaming JSON files with an explicit schema
from pyspark.sql.types import StructType, StructField, StringType, DoubleType, TimestampType

user_schema = StructType([
    StructField("device_id", StringType(), True),
    StructField("temperature", DoubleType(), True),
    StructField("timestamp", TimestampType(), True)
])

streaming_df = spark.readStream \
    .format("json") \
    .schema(user_schema) \
    .option("maxFilesPerTrigger", 10) \
    .load("/mnt/iot/incoming_json/")

Key File Source Behaviors:

• Schema Enforcement: By default, Spark refuses to infer schemas on streaming file sources to avoid the massive performance overhead of scanning entire directories. You must supply a schema via .schema(user_schema) or explicitly enable schema inference with spark.conf.set("spark.sql.streaming.schemaInference", "true").
• maxFilesPerTrigger: Controls rate limiting by capping the maximum number of new files processed in a single micro-batch. This prevents driver and executor memory exhaustion when pointing a stream at a directory containing a large backlog of historical files.
• Atomicity Requirement: Files placed in the monitored directory must appear atomically. If a file is written partially while Spark reads it, the stream will read corrupted data.
• Delta Lake as Source: When reading from Delta Lake (.format("delta")), schema inference is supported automatically because Delta maintains a transaction log (_delta_log) that defines the exact table schema. Delta also supports tracking row-level modifications via the Change Data Feed (CDF).

2. Apache Kafka Source

Kafka is the industry-standard distributed streaming source. Spark connects to Kafka using .format("kafka").

kafka_stream_df = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "broker1:9092,broker2:9092") \
    .option("subscribe", "telemetry-events") \
    .option("startingOffsets", "latest") \
    .option("failOnDataLoss", "false") \
    .load()

Kafka Connection Options:

• kafka.bootstrap.servers: Comma-separated list of host:port broker endpoints.
• subscribe: Comma-separated list of Kafka topics. Alternatively, use subscribePattern for regex matching or assign for specific topic-partitions.
• startingOffsets: Controls where reading begins upon the initial stream launch:
  • "latest": Read only new messages arriving after query startup (default for streaming).
  • "earliest": Read all available messages from the beginning of Kafka topic retention.
  • JSON string: Specify specific offsets per partition (e.g., """{"topicA":{"0":23,"1":-1}}""").
• failOnDataLoss: When set to "true" (default), the query fails if offsets have expired out of Kafka retention before Spark consumed them. Setting this to false can allow processing to continue despite missing offsets, at the cost of possible data loss. Keep the default unless that tradeoff is explicitly acceptable; fix retention and lag rather than hiding missing input.

Schema of the Kafka Source DataFrame:

Spark presents Kafka messages with a fixed internal schema:

from pyspark.sql.functions import col, from_json

parsed_df = kafka_stream_df \
    .select(from_json(col("value").cast("string"), user_schema).alias("payload")) \
    .select("payload.*")

3. Testing and Synthetic Sources

• Rate Source (.format("rate")): Generates synthetic data at a specified rate per second. Contains two columns: timestamp (TimestampType) and value (LongType starting at 0). Useful for performance profiling and benchmarking.
  • Options: rowsPerSecond (default 1), rampUpTime, numPartitions.
• Socket Source (.format("socket")): Reads UTF-8 text lines from an open TCP socket connection (host, port). Strictly for testing; it is not fault-tolerant because TCP sockets cannot replay historical data upon failure.

Streaming Sinks (df.writeStream)

A streaming query must write its results to an external sink using df.writeStream, which returns a DataStreamWriter.

query = processed_df.writeStream \
    .format("delta") \
    .outputMode("append") \
    .option("checkpointLocation", "/mnt/checkpoints/telemetry_gold") \
    .start("/mnt/tables/telemetry_gold")

Summary of Supported Sinks

1. Delta Lake Sink (.format("delta")): The primary recommended storage sink on Databricks. Supports ACID transactions, schema enforcement, schema evolution (mergeSchema), unified batch and streaming reading/writing, and transactional streaming writes with checkpoint-based replay protection; custom foreachBatch writes need their own idempotency design.
2. Kafka Sink (.format("kafka")): Publishes streaming output back to a Kafka topic. Requires a value column (string or binary) and an optional key column. Target topic is specified via .option("topic", "output-topic") or as a topic column in the DataFrame.
3. File Sink (.format("parquet"), .format("orc"), .format("json"), .format("csv")): Writes files to storage directories. Only supports Append output mode; does not support updates or overwrites. Requires a checkpoint directory.
4. Console Sink (.format("console")): Prints micro-batch output directly to the driver stdout/terminal. Options include numRows and truncate. Strictly for interactive debugging.
5. Memory Sink (.format("memory")): Stores output as an in-memory table on the Driver node registered under .queryName("my_table"). Allows querying with spark.sql("SELECT * FROM my_table"). Dangerous for large volumes because it can cause Driver OutOfMemory (OOM) exceptions.
6. Foreach and ForeachBatch Sinks: Custom output mechanisms for unsupported databases or complex write logic.

ForeachBatch vs. Foreach

• foreachBatch(func): Passes each micro-batch as a standard static DataFrame to a Python function with signature func(batch_df, batch_id). This is the most versatile sink in modern Spark. It enables:
  1. Writing to databases lacking native streaming sinks (e.g., writing to PostgreSQL or MySQL via .write.jdbc()).
  2. Fan-out writing to multiple destinations (e.g., saving to Delta and sending metrics to an API) from a single stream.
  3. Executing Delta Lake MERGE INTO (upserts) per micro-batch.

def upsert_to_delta(batch_df, batch_id):
    # Re-using DeltaTable batch API inside foreachBatch
    delta_table.alias("target").merge(
        batch_df.alias("source"),
        "target.device_id = source.device_id"
    ).whenMatchedUpdateAll().whenNotMatchedInsertAll().execute()

query = streaming_df.writeStream \
    .foreachBatch(upsert_to_delta) \
    .option("checkpointLocation", "/mnt/checkpoints/upsert_stream") \
    .start()

• foreach(writer): Row-level processing. Requires implementing an object with open(partition_id, epoch_id), process(row), and close(error). Rarely used in PySpark due to Python-to-JVM serialization overhead; foreachBatch is strongly preferred.

Delivery guarantee depends on the sink

Micro-batch checkpointing plus a replayable source and a correctly implemented transactional/idempotent sink can provide end-to-end exactly-once effects. The Kafka sink is at-least-once and can duplicate records. foreachBatch is also at-least-once unless the callback uses batch IDs or stable business keys to suppress retries. Continuous processing has at-least-once guarantees. An append output mode describes which result rows are emitted, not whether the destination can receive duplicates. A MERGE callback must be deterministic and idempotent under replay; writing two systems in one callback is not one cross-system atomic commit.