1.4 Streaming Distribution, Replay & Processing State

1.4 Streaming Distribution, Replay & Processing State

Service comparisons are easier when you first model the flow semantics. A source produces records, a transport orders and retains them, consumers maintain progress, and sinks apply side effects. Most exam failures come from selecting a service with the right name but the wrong delivery or recovery behavior.

Fan-in and fan-out

Fan-in combines many producers or upstream streams into one processing path. Thousands of devices can write to a Kinesis stream, or multiple EventBridge rules can target one queue. The risk is a hot partition or a consumer that cannot match the combined rate.

Fan-out gives multiple independent consumers the same logical events. Kinesis enhanced fan-out gives registered consumers dedicated per-shard read throughput. Kafka consumer groups behave differently: consumers in the same group divide partitions, while separate groups each receive the topic independently. Amazon SNS can publish one message to several SQS queues, giving each downstream application its own backlog and retry policy.

Ordering scope

Ordering is almost never global. Kinesis orders records within a shard; Kafka orders within a topic partition; an SQS FIFO queue orders within a message group. Increasing parallelism by adding shards, partitions, or message groups increases throughput but divides the ordering boundary. Choose a partition key that preserves the business sequence—such as account ID—while spreading unrelated entities. A single constant key preserves broad ordering but creates a hot partition.

Replay is not retry

A retry repeats a failed delivery or invocation. A replay deliberately reads retained historical records again from an earlier offset, sequence, or timestamp. Kinesis can start consumers at trim horizon, latest, or a timestamp, subject to retention. Kafka consumer groups can reset offsets while the records remain in the topic. DynamoDB Streams retains records for only 24 hours, so it is a narrow recovery window.

Safe replay requires four controls:

1. Retain the source long enough to cover detection and repair.
2. Save checkpoints separately for each consumer application.
3. Make the sink idempotent with a source event ID, conditional write, merge key, or processed-event table.
4. Decide whether replay writes to the production sink, a shadow destination, or a corrected partition.

Without idempotency, replay can double-charge a card, duplicate a warehouse row, or send the same notification twice. Exactly-once claims usually describe a bounded service integration; end-to-end correctness still depends on the sink transaction.

Stateful and stateless processing

A stateless transform needs only the current record: parse JSON, redact one field, or convert a timestamp. It scales horizontally because any worker can process any record.

A stateful operation remembers prior events: a five-minute rolling count, stream-table join, session window, fraud sequence, or deduplication set. Managed Service for Apache Flink checkpoints operator state and source positions so a restarted job can recover consistently. Event time and watermarks control how long the system waits for late events; an aggressive watermark improves latency but can exclude late data.

State grows with key cardinality and retention. Use state time-to-live where valid, monitor checkpoint duration, and avoid a key that concentrates most state on one task.

Backpressure and rate limits

Backpressure appears when arrival rate exceeds processing rate. Iterator age, Kafka consumer lag, queue depth, and Firehose delivery freshness expose the symptom. Remedies include scaling consumers, increasing partitions, batching calls, compacting small records, or reducing work per event. Producers calling DynamoDB, RDS, or external APIs must respect target limits with bounded exponential backoff and jitter.

An unbounded retry loop is not resilience: it holds capacity, duplicates traffic, and increases the target's load. After bounded retries, move the failed unit to a durable failure destination with enough context to repair and replay it. Alert on backlog age, not only count, because a small backlog of very old records can be more serious than a brief burst.

Exam decision sequence

Ask: What is the ordering boundary? Does every consumer need every record? How long must records be replayable? Who owns the checkpoint? Is the operation stateful? Can the sink tolerate duplicates? Those answers select the architecture more reliably than latency alone.