Streaming Analytics with Amazon Kinesis and Amazon MSK
Key Takeaways
- Amazon Kinesis Data Streams is a replayable, shard-based stream with independent consumers; Amazon Data Firehose is a delivery service that buffers and lands data without a consumer GetRecords API.
- Amazon MSK is managed Apache Kafka: partitions, consumer groups, and the Kafka ecosystem. Choose it when Kafka compatibility matters; choose Kinesis when AWS-native consumers (Lambda, Firehose, Flink) and a simpler shard model matter.
- A Kinesis shard provisions 1 MB/s or 1,000 records/s write and 2 MB/s read (classic consumers share that read cap); Kafka partitions are the parallel ordered unit for MSK.
- Amazon Managed Service for Apache Flink runs stateful stream jobs (windows, joins) against Kinesis or MSK; it is not a drop-in replacement for an SQS worker.
- Use SQS for discrete work items that competing consumers delete; use a stream for clickstream and other ordered, multi-consumer, replay-within-retention workloads. Lambda MaximumBatchingWindowInSeconds is 0–300 seconds.
Why streams are not queues
SAP-C02 Task 2.5 asks you to design large-scale access patterns and to apply buffering as a performance pattern. A buffer might be Amazon SQS (Task 2.4 decoupling) or a stream. The Professional mistake is using a queue for clickstream that five teams must read, or using a stream for order capture that should be deleted after one successful worker. Independent SAP-C02 study material by OpenExamPrep treats Amazon Kinesis Data Streams, Amazon Data Firehose (formerly Kinesis Data Firehose), Amazon Managed Streaming for Apache Kafka (Amazon MSK), and Amazon Managed Service for Apache Flink as a decision set.
HarborCart's mobile app emits millions of navigation events per hour. Product, personalization, and data-science teams all want the same feed. Warehouse pick jobs remain on SQS FIFO. Those are different problems.
Kinesis Data Streams versus Data Firehose versus MSK
Amazon Kinesis Data Streams stores records in shards. Producers PutRecord/PutRecords. Consumers read with GetRecords (shared throughput) or enhanced fan-out SubscribeToShard (dedicated throughput). Records remain for a retention period: default 24 hours, extendable to 8,760 hours (365 days). Retention above 24 hours is billed extra. Multiple consumer applications can read the same shard independently; a crash recovers by checkpointing within retention. That is why clickstream belongs here.
Provisioned mode: you choose shard count. On-demand mode: AWS scales with traffic; you can configure warm throughput when you forecast a peak. AWS documents that on-demand aggregate read capacity scales with write throughput and recommends enhanced fan-out when more than one consumer application must read in parallel without sharing the 2 MB/s classic read budget.
Amazon Data Firehose is a delivery stream. Producers (or a Kinesis data stream, or an MSK topic) send records; Firehose buffers by size (MB) and interval (seconds), optionally invokes Lambda to transform, optionally converts to Parquet/ORC, and loads Amazon S3, Amazon Redshift, Amazon OpenSearch Service, Splunk, Snowflake, HTTP endpoints, and similar destinations. There is no consumer application API to replay Firehose the way you replay a Kinesis shard. If the only need is "land clickstream in the data lake," Firehose alone is enough. If product and fraud must react in seconds and data-science still wants S3, use Data Streams plus a Firehose consumer (or Flink) rather than Firehose alone.
Amazon MSK runs Apache Kafka. Topics split into partitions. A record key hashes to a partition; order is per partition. Consumer groups share work: each partition is consumed by one member of the group, while separate groups independently reread the topic (within Kafka retention). Use MSK when HarborCart already has Kafka clients, Kafka Connect, compacted topics, transactions, or a requirement to stay Kafka-portable. MSK Serverless reduces broker sizing; provisioned clusters expose Kafka configuration, storage, and throughput knobs. MSK Replicator and MSK Connect appear in hybrid and pipeline stems. Lambda, Firehose, EventBridge Pipes, and Flink all integrate with MSK.
| Need | Kinesis Data Streams | Amazon Data Firehose | Amazon MSK |
|---|---|---|---|
| Multiple independent real-time consumers | Yes | No GetRecords consumers | Yes (consumer groups) |
| Replay within retention | Yes (24 h–365 d) | Not a stream replay API | Yes (Kafka retention) |
| Managed load to S3/OpenSearch/Splunk | As a consumer of Streams | Primary job | Firehose integration or Connect |
| AWS Lambda event source mapping | Yes | Not as a stream ESM | Yes (MSK ESM) |
| Kafka protocol / Connect / compact topics | No | No | Yes |
| Simplest AWS-native clickstream | Often yes | Yes if lake-only | If Kafka is already the standard |
Shards versus partitions
A Kinesis shard is a capacity unit. AWS documents each shard as supporting up to 1,000 records per second for writes and 1 MB/s write (including partition keys), five read transactions per second, and 2 MB/s total read for classic consumers. Scale by resharding (split/merge) or by on-demand. The partition key chooses the shard; a hot key (all events with key web) concentrates on one shard and throttles (ProvisionedThroughputExceeded). HarborCart uses sessionId or shopperId as the partition key so clickstream spreads while one shopper stays ordered.
A Kafka partition on MSK is the ordered log segment. Throughput depends on broker sizing, partition count, replica I/O, and consumer lag—not on a single published "1 MB/s per partition" Kinesis-style number. Parallelism of a consumer group cannot exceed the partition count. Compacted topics and exact Kafka semantics are why MSK is not "Kinesis with a different name."
Enhanced fan-out on Kinesis gives each registered consumer up to 2 MB/s per shard dedicated, with AWS documenting up to 20 such consumers on a stream. Classic GetRecords consumers share the shard's 2 MB/s. If product Lambda and a Flink job both trail a busy stream, enhanced fan-out (or separate on-demand headroom) is the Task 2.5 answer.
Managed Service for Apache Flink
Amazon Managed Service for Apache Flink (the current name for the managed Flink service; older stems may still say Kinesis Data Analytics) runs Apache Flink applications. Use it for stateful streaming: event-time windows, joins between clickstream and order events, sessionization, and continuously updated aggregates. Sources include Kinesis Data Streams and Amazon MSK. Studio notebooks (Apache Zeppelin) help interactive analysis, then you deploy a durable application.
Flink is not an SQS replacement. If HarborCart needs "exactly one worker captures payment," that remains FIFO SQS. If HarborCart needs "unique shoppers in a 5-minute window per campaign," that is Flink (or a carefully designed Lambda with tumbling windows—Lambda's Kinesis ESM supports tumbling windows, but complex joins belong in Flink).
Queue versus stream
| Question | Queue (SQS) | Stream (Kinesis / MSK) |
|---|---|---|
| Who should process a message? | Usually one competing consumer | Many independent consumers |
| After success? | Delete the message | Advance a checkpoint; others still read |
| Replay last week? | Only if it is still within retention and not deleted (max 14 days) | Yes, within stream retention (Kinesis up to 365 days) |
| Ordering | FIFO per message group | Per shard (partition key) or per Kafka partition |
| Typical HarborCart use | Order capture, emails via SNS+SQS, warehouse jobs | Clickstream, inventory position ticks, fraud features |
Do not put clickstream on SQS expecting three analytics teams to share it: the first worker deletes the message. Do not put payment capture on Kinesis expecting the record to vanish after one consumer; every consumer sees it, and you still need idempotency.
Lambda batch windows and parallelization
Lambda event source mappings poll streams and queues and invoke the function with a batch.
AWS documents MaximumBatchingWindowInSeconds from 0 to 300 (one-second steps). For Kinesis, DynamoDB streams, and SQS, the default window is 0 seconds (invoke as soon as records are available). For Amazon MSK, self-managed Kafka, Amazon MQ, and DocumentDB, the default window is 500 ms. A window trades latency for fewer, fuller invocations—Task 2.5 buffering.
Batch size is the maximum records per invoke (Kinesis up to 10,000). Each Kinesis batch is from a single shard. ParallelizationFactor (1 default to 10) lets Lambda process multiple concurrent batches per shard while still preserving order per partition key. When IteratorAge is high and keys are diverse, raising parallelization (and shard count) is the scale lever. A single hot partition key still serializes.
On failure, stream mappings can bisect the batch, retry, and send to an on-failure destination. SQS mappings should use ReportBatchItemFailures so one poison message does not recycle the entire batch. FIFO SQS plus Lambda will not concurrently process two batches of the same message group; that preserves order and caps scale per group.
HarborCart's clickstream Lambda uses a 2-second batching window, batch size 500, parallelization factor 4, and enhanced fan-out. A Flink job on the same stream computes 5-minute unique shoppers. Firehose (as a stream consumer) writes Parquet to S3 for Athena. Order capture never uses this path.
HarborCart clickstream design
- Mobile and web producers put events to Kinesis Data Streams on-demand with partition key
shopperId. - Retention 7 days so a bad deploy can replay (additional cost above 24 hours).
- Lambda (enhanced fan-out) updates a hot personalization cache.
- Managed Service for Apache Flink computes session windows.
- Firehose delivers to S3.
- If a Kafka-based data-science platform must consume with existing Connectors, add MSK as a parallel ingest or replicate—do not pretend Kinesis speaks Kafka.
- Warehouse jobs stay on SQS FIFO; clickstream never shares that queue.
Traps
- Firehose when the stem needs multiple real-time consumers or replay.
- SQS for multi-team clickstream.
- Kinesis for single-worker payment capture that should be deleted.
- One hot partition key.
- Classic consumers starving each other when enhanced fan-out is the requirement.
- A 24-hour Lambda batch window (the maximum is 300 seconds).
- Assuming MSK and Kinesis have the same shard math.
- Using Flink for a three-step order workflow that belongs in SQS plus Step Functions.
HarborCart's website emits a clickstream that product, personalization, and data-science teams must process independently. They need to replay about a week after a bad consumer deploy, and they also want Parquet in Amazon S3. Which design matches Amazon Kinesis and Firehose roles?
HarborCart must capture payment for each order exactly through one worker path, and must also run clickstream analytics for several independent consumers. Which queue-versus-stream statement should guide the architecture?
HarborCart's Lambda function that reads clickstream from Amazon Kinesis shows rising IteratorAge during flash sales. Events use many shopperId partition keys. The team also needs 5-minute unique-shopper counts. Which change matches Lambda stream mapping and Flink roles?