12.4 Streaming Deduplication With and Without Watermarks

Why per-batch deduplication is insufficient

A source can resend an event after a timeout, reconnect or upstream replay. If duplicate event_id values arrive in different micro-batches, applying distinct only to a batch DataFrame inside foreachBatch remembers nothing about the previous batch. A streaming deduplication operator instead maintains state across batches. That state must be checkpointed and bounded appropriately for the application.

Choose a key that describes event identity. customer_id would discard a customer's later legitimate events if the desired key is event_id. Conversely, using every payload field can fail to identify the same event when retry metadata or ingestion timestamps differ. Before selecting an API, write down which fields can legitimately change between copies and how long a duplicate may be delayed.

Without a watermark

# events is a streaming DataFrame with event_id and event_time
unique_forever = events.dropDuplicates(["event_id"])

This remembers seen event IDs across batches and suppresses later duplicates under the query's state history. There is no event-time bound for safe removal of an old key, because another duplicate could arrive arbitrarily far in the future. High-cardinality input can therefore produce growing state, increased checkpoint IO and pressure on memory/native storage. A larger cluster or RocksDB provider can change capacity but does not supply an expiry policy.

Do not assume a trigger interval bounds deduplication memory. A ten-second trigger schedules computation; it does not mean IDs expire after ten seconds. Likewise, a checkpoint path stores progress/state but does not impose business retention or a fixed state-size cap. Monitor retained state and define the intended duplicate horizon.

Traditional watermark-aware deduplication

bounded_exact_pairs = (events
    .withWatermark("event_time", "20 minutes")
    .dropDuplicates(["event_id", "event_time"]))

In this pattern, the watermark timestamp is included in the key so old state can be cleaned up. Equality now means the pair event_id/event_time, not event_id alone. If two copies of an event have the same event ID and the exact same event_time, they match. If the second has a timestamp one second later, the pair differs and it can survive this deduplication. That is a correctness difference, not merely a tuning detail.

Suppose two events are (A, 10:00:00) and (A, 10:00:01). The pair-based pattern treats them as different keys, even though a producer contract may call them retries of one event. Do not remove event_time from an old-API dedup key and assume watermark cleanup still behaves identically. Check the API's documented retention behavior for the supported Spark version.

Spark 3.5: identity within a watermark horizon

Spark 3.5 added dropDuplicatesWithinWatermark for streaming DataFrames with a watermark. It allows deduplication by stable identity without requiring the timestamp itself to be part of the identity key.

bounded_ids = (events
    .withWatermark("event_time", "20 minutes")
    .dropDuplicatesWithinWatermark(["event_id"]))

query = (bounded_ids.writeStream
    .format("parquet")
    .outputMode("append")
    .option("checkpointLocation", "/checkpoints/unique_events")
    .start("/output/unique_events"))

Choose a delay longer than the maximum event-time difference among copies you must reliably suppress, following the API's documented guarantee. This is not unlimited forever-deduplication: once old state expires, a much later copy is outside that retention contract. A bad future timestamp can advance event-time progress unexpectedly; validate event-time quality before choosing a horizon.

The writer path and checkpoint in this example are distinct. A query restart should reuse its own compatible checkpoint to recover identifiers and offsets. A new checkpoint creates a new history, potentially accepting source records previously seen by the old query. Concurrent queries must not share one checkpoint directory.

Watermarks are not universal timestamp filters

Watermark delay is measured relative to observed event-time progress, not the wall-clock age of a record when it reaches a worker. With a latest observed event at 12:45 and delay of twenty minutes, the candidate watermark is 12:25. Advancement is coordinated with micro-batch execution and operator semantics. withWatermark alone does not drop every row older than that timestamp.

For watermark-based operators, inclusion guarantees and late-data handling depend on the API and retained state. Do not generalize a window-aggregation cleanup rule into every deduplication algorithm. A low-traffic stream may not advance its event-time watermark promptly, so state can remain longer in wall-clock time than the nominal delay.

Input identity versus output delivery

Input deduplication suppresses repeated business events while its state is retained. Sink retry protection is a separate requirement. The Kafka sink can produce duplicate output on retry, and foreachBatch is at-least-once unless its writes are made idempotent. Deduplicating before a sink does not automatically make an external insert exactly-once. Use stable keys, batch IDs and appropriate transactional handling at the destination.

For a practical check, test copies in the same batch, in different batches, with identical timestamps, with differing timestamps and outside the chosen horizon. Also test recovery from a compatible checkpoint. Inspect query.lastProgress state-operator metrics and watermark progress rather than assuming a successful startup proves bounded state. Change dedup keys or state schema only through a planned compatible migration. See streaming deduplication and the watermark API.

Choose identity and retention together