12.2 Output Modes, Triggers, and Checkpointing

When deploying a Structured Streaming query to production, configuring how results are emitted, when micro-batches execute, and where execution state is persisted is critical for application correctness and cost efficiency. Spark manages these behaviors through Output Modes, Triggers, and Checkpoint Locations.

Understanding the precise constraints of each output mode, the operational advantages of modern triggers like AvailableNow(), and the internal directory mechanics of checkpointing is essential for building resilient streaming pipelines and scoring highly on the certification exam.

---

The Three Output Modes

The output mode defines what data from the internal Result Table is written to the external sink at each trigger interval. Output mode is configured on the DataStreamWriter via .outputMode("append" | "update" | "complete").

+-------------------------------------------------------------------------+
|                           OUTPUT MODES SUMMARY                          |
+-------------------------------------------------------------------------+
| Mode     | What Gets Written to Sink?     | Aggregation Required?       |
|----------|--------------------------------|-----------------------------|
| Append   | Only brand-new rows added      | No (Allowed only with       |
|          | since the last trigger         | watermarking if aggregated) |
|----------|--------------------------------|-----------------------------|
| Update   | Only rows updated or added     | No (Behaves like Append     |
|          | since the last trigger         | if no aggregation)          |
|----------|--------------------------------|-----------------------------|
| Complete | The ENTIRE Result Table        | YES (Throws exception       |
|          | rewritten on every trigger     | if no aggregation)          |
+-------------------------------------------------------------------------+

1. Append Mode (.outputMode("append"))

In Append Mode, only the rows that were newly added to the Result Table since the previous micro-batch are output to the storage sink.

• Semantics: Only newly appended result rows are emitted; actual delivery semantics depend on the sink. Once written, rows are never modified or updated.
• Permitted Queries:
  • Simple transformational queries without aggregations (e.g., select, filter, map, flat map).
  • Aggregations only if an event-time watermark is defined. Spark holds back the output row until the watermark passes the window end time (watermark >= window.end), finalizing the retained result under the configured lateness policy; later data may still arrive and be discarded.
• Prohibited Queries: Streaming aggregations without a watermark are not supported in Append mode. Because un-watermarked aggregations can be updated indefinitely by future data, Spark cannot guarantee that an aggregate row will never change, and therefore refuses to emit it in Append mode.

2. Update Mode (.outputMode("update"))

In Update Mode, only the rows in the Result Table that were updated (or newly inserted) since the last trigger are emitted to the sink.

• Semantics: If an existing group's count or sum changes during a micro-batch, the updated record is output to overwrite or update downstream systems. Unchanged rows in the Result Table are not emitted.
• Permitted Queries:
  • Queries with aggregations (with or without watermarks).
  • Queries without aggregations: If a query has no aggregations, Update mode behaves identically to Append mode.
• Common Sinks: Sinks that support key-based in-place updates or upserts, such as Delta Lake (via foreachBatch), Kafka, MySQL/PostgreSQL via JDBC, or key-value NoSQL stores.

3. Complete Mode (.outputMode("complete"))

In Complete Mode, the entire updated Result Table is rewritten to the external sink every time a trigger fires, regardless of how many rows actually changed.

• Mandatory Requirement (Frequent Exam Trap): Complete mode strictly requires an aggregation (groupBy(), window(), etc.). If you attempt to start a streaming query in Complete mode on a DataFrame without aggregations, Spark immediately throws an AnalysisException:

  AnalysisException: Complete output mode not supported when there are no streaming aggregations on streaming DataFrames/DataSets
  

• Prohibited Sinks: Complete mode is incompatible with file sinks (Parquet, ORC, CSV, JSON). File storage cannot overwrite individual records or cleanly rewrite an entire directory per micro-batch without massive overhead. Complete mode is typically used with Delta Lake, console, or memory sinks.
• State Retention: Unlike Append mode, Complete mode does not drop old aggregation state even if watermarks are defined, because the entire table must be output on every iteration.

Output Mode Compatibility Matrix

---

Streaming Triggers

Triggers define the timing of streaming execution—dictating exactly when Spark checks sources for newly arrived data and executes micro-batch computations. Triggers are configured via .trigger(...) on the DataStreamWriter.

# Configuring an AvailableNow trigger
query = df.writeStream \
    .format("delta") \
    .outputMode("append") \
    .trigger(availableNow=True) \
    .option("checkpointLocation", "/mnt/checkpoints/etl_stream") \
    .start("/mnt/tables/etl_gold")

Available Trigger Types

1. Default Micro-Batch Trigger (Unspecified):
   • If no trigger is specified, Spark executes micro-batches in as-fast-as-possible mode. As soon as the previous micro-batch completes execution, the next micro-batch is scheduled immediately.
2. Fixed Interval Micro-Batch Trigger (.trigger(processingTime="interval")):
   • Executes micro-batches at discrete, user-defined clock intervals (e.g., .trigger(processingTime="10 seconds") or .trigger(processingTime="1 minute")).
   • Catch-up Behavior: If a micro-batch takes longer to process than the specified interval (e.g., a batch takes 16 seconds on a 10-second trigger), Spark starts the next micro-batch immediately upon completion without waiting for the next clock boundary.
3. Once Trigger (.trigger(once=True) - Legacy):
   • Ingests all available data from the source in a single micro-batch, commits offsets to the checkpoint directory, and terminates the streaming query cleanly.
   • AvailableNow was added in Spark 3.3; once=True is deprecated since Spark 3.4: If a substantial backlog of data has accumulated, .trigger(once=True) attempts to ingest all data in one giant batch regardless of maxFilesPerTrigger, frequently causing executor OutOfMemory (OOM) failures.
4. AvailableNow Trigger (.trigger(availableNow=True) - Spark 3.3+ Standard):
   • Replaces .trigger(once=True). Ingests and processes all outstanding data across multiple micro-batches, strictly honoring rate limits such as maxFilesPerTrigger or Kafka's maxOffsetsPerTrigger.
   • Once all backlogged data has been processed across sequential micro-batches, the query shuts down cleanly.
   • The Incremental Batch-as-Stream Pattern: This is the most cost-efficient pattern on Databricks. Data engineers can schedule a Databricks Job to spin up a compute cluster once per day or hour, execute .trigger(availableNow=True), process the backlog incrementally using streaming logic, and terminate the cluster to save compute costs.
5. Continuous Trigger (.trigger(continuous="1 second")):
   • Activates the low-latency continuous processing engine with asynchronous epoch-based checkpoint commits.

---

Fault Tolerance and Checkpointing

Structured Streaming guarantees end-to-end exactly-once processing through the combination of:

1. Replayable Sources: Sources (like Apache Kafka, Delta Lake, or cloud object files) that can replay data from specified offsets.
2. Deterministic Engine with Checkpointing: Spark tracks read progress and state changes in a durable write-ahead log.
3. Idempotent / Transactional Sinks: Sinks (like Delta Lake) that can detect duplicate writes or execute atomic commits.

Checkpointing is configured by passing an external storage path to .option("checkpointLocation", "/path/to/checkpoint").

query = streaming_df.writeStream \
    .format("delta") \
    .option("checkpointLocation", "/mnt/lakehouse/checkpoints/device_stream") \
    .start("/mnt/lakehouse/tables/device_events")

Internal Directory Structure of a Checkpoint

Inside the specified checkpoint location, Spark maintains four critical directories:

/mnt/lakehouse/checkpoints/device_stream/
├── offsets/         <-- Write-Ahead Log of ingested source offsets
│   ├── 0
│   ├── 1
│   └── 2
├── commits/         <-- Completion markers for finished micro-batches
│   ├── 0
│   └── 1
├── state/           <-- Serialized state store files (RocksDB or HDFS)
│   └── 0/
│       └── 0/
└── metadata         <-- Query UUID and configuration metadata

1. offsets/ (Write-Ahead Log): Before executing tasks in micro-batch $N$, the driver records the exact starting and ending source offsets (e.g., Kafka topic partition offsets or file paths) in a new file named N. This guarantees Spark knows exactly what data it intended to process.
2. commits/ (Completion Log): After all executor tasks in micro-batch $N$ successfully complete and the sink confirms persistence, the driver writes a versioned commit-log entry named N (not necessarily an empty file). If a micro-batch fails mid-execution, offsets/N will exist, but commits/N will be missing.
3. state/ (State Store): Houses intermediate key-value state for stateful operations (aggregations, windowing, deduplication, stream-stream joins). Implemented either as HDFS-backed state store (default memory/disk) or RocksDB State Store (recommended in Databricks for large state sizes).
4. metadata: Stores the persistent query ID. query.id remains stable across checkpoint recovery; query.runId changes at each start/restart. State-store metadata has separate compatibility rules.

Failure Recovery Protocol

When a crashed Structured Streaming application restarts pointing to the same checkpoint directory:

1. Spark reads the offsets/ and commits/ directories.
2. If the highest batch in offsets/ (e.g., batch 2) does not have a corresponding file in commits/, Spark knows batch 2 failed before completing.
3. Spark reads the exact offsets recorded in offsets/2, re-executes micro-batch 2 from the replayable source, and writes the output to the sink. If the sink is idempotent (such as Delta Lake, which checks the streaming query ID and batch ID in its transaction log), duplicate writes are prevented.
4. Once batch 2 commits, Spark proceeds to batch 3.

Official reference

Structured Streaming guide — source checked October 16, 2026. The examples state their runtime assumptions.