1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning

1.1 Kinesis Data Streams vs. Firehose Architecture & Partitioning

Streaming Ingestion Fundamentals on AWS

In enterprise data engineering, streaming ingestion pipelines process high-velocity data generated continuously by thousands of distributed sources—such as IoT sensors, application log forwarders, clickstream trackers, and financial transaction feeds. AWS provides two foundational services for streaming data ingestion: Amazon Kinesis Data Streams (KDS) and Amazon Data Firehose (formerly Kinesis Data Firehose).

While both services ingest streaming payloads, they serve fundamentally different architectural paradigms: KDS is a low-latency, stateful event store built for real-time custom processing, whereas Amazon Data Firehose is a fully managed, serverless stream delivery service designed to load streaming data directly into storage sinks, analytical data stores, and third-party destinations.

---

Amazon Kinesis Data Streams (KDS) Architecture

Capacity Modes: Provisioned vs. On-Demand

Kinesis Data Streams operates in one of two capacity modes:

• Provisioned Mode: You explicitly specify the number of shards allocated to the stream. Each shard provides a fixed unit of read and write capacity. You are billed hourly per shard regardless of data throughput.
• On-Demand Mode: AWS automatically manages and scales stream capacity in response to traffic. Current choices include On-demand Standard and the account-level On-demand Advantage mode, which supports warm throughput for forecast peaks. Regional stream quotas vary and reach well beyond the older 200 MB/s default, so design from the current quota page instead of memorizing one obsolete stream-wide ceiling.

Shard Limits & Throughput Dynamics

A shard is the base throughput unit of a Kinesis Data Stream. Understanding exact shard throughput limits is critical for the AWS Data Engineer Associate exam:

If a producer attempts to write data that exceeds 1 MB/s or 1,000 records/sec on a single shard, KDS returns a ProvisionedThroughputExceededException. Producers must handle this using exponential backoff with randomized jitter, or scale shard capacity.

Partitioning & MD5 Hash Distribution

When a producer writes a record using the PutRecord or PutRecords API, it must supply a string-based Partition Key. KDS applies an MD5 hash algorithm to the partition key, mapping it to a 128-bit integer space ranging from 0 to 2^128 - 1 (340,282,366,920,938,463,463,374,607,431,768,211,455). Each shard in a stream is assigned a contiguous range of hash keys.

Partition Key ("user_8492") ---> MD5 Hash ---> 128-bit Hash Key Range ---> Assigned Shard ID

Hot Shard Mitigation Strategy: If partition keys are poorly chosen (e.g., using a constant value, region code, or timestamp rounded to the hour), a disproportionate percentage of hash values will map to a single shard. This causes a hot shard, triggering ProvisionedThroughputExceededException errors while adjacent shards remain underutilized.

• Best Practice: Use high-cardinality attributes as partition keys (e.g., UUID, transaction ID, device serial number).
• Explicit Hash Key Override: Producers can override partition key hashing by supplying an ExplicitHashKey parameter to directly target specific hash ranges during resharding operations (SplitShard or MergeShards).

Consumer Architecture: Standard vs. Enhanced Fan-Out (EFO)

1. Standard Consumers: Use short or long polling via HTTP GET requests (GetRecords). All standard consumers attached to a shard share the 2 MB/s egress throughput budget. If 5 consumers read from the same standard shard, each receives an average throughput of ~400 KB/s, increasing read latency (typically 200 ms to 1,000 ms).
2. Enhanced Fan-Out (EFO) Consumers: Introduced for sub-second, low-latency topologies. EFO uses HTTP/2 streaming (SubscribeToShard) to push data from KDS to consumers with typical latency under 70 ms. Each EFO consumer gets a dedicated 2 MB/s egress bandwidth pipe per shard without competing with other registered consumers.

---

Amazon Data Firehose Architecture

Fully Managed Delivery & Buffering Mechanics

Amazon Data Firehose is a serverless ingest service that eliminates custom consumer code, shard provisioning, and scaling management. Data Firehose buffers incoming streaming data before writing it to downstream destinations.

Buffering behavior is governed by two configurable parameters:

• Buffer Size: A destination-specific hint. Amazon S3 and Redshift destinations accept 1–128 MiB (default 5 MiB).
• Buffer Interval: Also destination-specific. Amazon S3 and Redshift accept 0–900 seconds (default 300 seconds). Zero buffering is unavailable when dynamic partitioning is enabled, and S3 backup uses a nonzero interval.

Firehose flushes buffered records to the destination sink as soon as either condition is satisfied first. For example, if Buffer Size is set to 64 MB and Buffer Interval is set to 300 seconds, Firehose will flush data immediately upon accumulating 64 MB of records, or every 5 minutes if data volume is low.

Destination Sinks & Native Integrations

Firehose natively delivers data to standard AWS and third-party endpoints:

• Amazon S3: Raw or transformed object delivery.
• Amazon Redshift: Automatically buffers data into S3 and executes an intermediate COPY command into Redshift staging tables.
• Amazon OpenSearch Service: Indexing log and metric streams.
• Third-Party Analytics Sinks: Direct HTTPS delivery to Snowflake, Datadog, Splunk, and New Relic.

Inline Transformations & Format Conversion

Firehose provides two powerful serverless data transformation capabilities prior to delivery:

1. AWS Lambda Transformation: Firehose invokes an AWS Lambda function to execute custom inline ETL (e.g., stripping PII, decoding base64 payloads, filtering events). Lambda returns processed records back to Firehose (Ok, Dropped, or ProcessingFailed).
2. Serverless Format Conversion: Firehose can convert incoming raw JSON payloads directly into columnar storage formats—Apache Parquet or Apache ORC—before writing to Amazon S3. Firehose references table schemas defined in the AWS Glue Data Catalog to perform this conversion automatically without spinning up EMR or Glue ETL jobs, significantly improving Athena query performance and lowering storage costs.

Dynamic Partitioning & S3 Prefix Formatting

Firehose supports Dynamic Partitioning, allowing records to be written to custom S3 bucket prefixes evaluated dynamically from record attributes or timestamp parameters. For example, log events containing a customer_id and event_type can be dynamically partitioned in S3:

s3://telemetry-data-lake/customer_id=!{partitionKeyFromQuery:customer_id}/year=!{timestamp:yyyy}/month=!{timestamp:MM}/

---

Architectural Decision Matrix: KDS vs. Data Firehose

---

Code Example: Kinesis Data Streams Boto3 Producer with Retry Logic

import json
import random
import time
import boto3
from botocore.exceptions import ClientError

kinesis_client = boto3.client('kinesis', region_name='us-east-1')
STREAM_NAME = 'telemetry-ingestion-stream'

def send_telemetry_event(device_id: str, payload: dict):
    # High-cardinality partition key avoids hot shards
    partition_key = f"device#{device_id}"
    data_bytes = json.dumps(payload).encode('utf-8')
    
    max_retries = 5
    base_delay = 0.1  # 100ms
    
    for attempt in range(max_retries):
        try:
            response = kinesis_client.put_record(
                StreamName=STREAM_NAME,
                Data=data_bytes,
                PartitionKey=partition_key
            )
            print(f"Record ingested successfully! ShardId: {response['ShardId']}, SequenceNo: {response['SequenceNumber']}")
            return response
        except ClientError as e:
            error_code = e.response['Error']['Code']
            if error_code == 'ProvisionedThroughputExceededException':
                # Exponential backoff with full jitter
                sleep_time = random.uniform(0, base_delay * (2 ** attempt))
                print(f"Throughput exceeded. Retrying in {sleep_time:.3f} seconds (Attempt {attempt + 1})...")
                time.sleep(sleep_time)
            else:
                raise e
    raise RuntimeError("Failed to ingest record after maximum retries due to throughput limits.")