1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns

1.2 Amazon MSK (Managed Streaming for Apache Kafka) & Serverless Patterns

Apache Kafka on AWS: Provisioned vs. Serverless

Amazon Managed Streaming for Apache Kafka (MSK) is a fully managed service that simplifies running Apache Kafka applications without needing to manage infrastructure. Kafka is widely adopted for event-driven microservices, log aggregation, and real-time data integration due to its high-throughput topic architecture and client ecosystem.

AWS offers two distinct deployment models for MSK:

1. MSK Provisioned: You explicitly configure broker nodes (e.g., kafka.m5.xlarge), specify the number of brokers across Availability Zones, and manage EBS storage volumes. You retain access to custom Kafka configuration parameters (such as auto.create.topics.enable, log.retention.hours, and unclean.leader.election.enable).
2. MSK Serverless: A fully serverless offering that automatically provisions and scales compute and storage capacity based on incoming traffic. MSK Serverless manages broker compute and storage capacity while the application team still creates topics and chooses partition counts and record keys. It supports up to 200 MB/s per cluster of write throughput and 400 MB/s per cluster of read throughput under the current service quotas, with throughput and storage charges.

---

Security & Authentication Architectures

Enterprise streaming pipelines require rigorous authentication and authorization controls. Amazon MSK supports multiple security mechanisms:

IAM Authentication (SASL/OAUTHBEARER)

AWS provides an open-source library (aws-msk-iam-auth) allowing Kafka client applications to authenticate against MSK using AWS Identity and Access Management (IAM) credentials.

• MSK Serverless Requirement: MSK Serverless strictly requires IAM Authentication. Custom SASL/SCRAM username/password mechanisms are not supported in MSK Serverless.
• IAM Policy Authorization: Fine-grained permissions can be specified using IAM policies to grant topic-level actions (e.g., kafka-cluster:Connect, kafka-cluster:WriteData, kafka-cluster:ReadData).

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:Connect",
        "kafka-cluster:DescribeTopic",
        "kafka-cluster:WriteData"
      ],
      "Resource": "arn:aws:kafka:us-east-1:123456789012:topic/orders-cluster/*"
    }
  ]
}

SASL/SCRAM & mTLS

For legacy or multi-cloud Kafka clients, MSK Provisioned supports:

• SASL/SCRAM (Salted Challenge Response Authentication Mechanism): User credentials stored securely in AWS Secrets Manager and integrated with MSK.
• Mutual TLS (mTLS): Certificate-based authentication integrated with AWS Private Certificate Authority (Private CA).

---

Kafka Partitioning Strategy & Consumer Group Scaling

Topic Partitions as Parallelism Units

In Apache Kafka, a topic is divided into one or more Partitions. Partitions represent the fundamental unit of scalability and parallelism. Records written to a partition are assigned a strictly increasing sequential ID called an offset.

Kafka producers compute partition assignments using the MurmurHash2 algorithm on the record key:

Partition = MurmurHash2(RecordKey) % NumberOfPartitions

• Records with the exact same key always land in the same partition, guaranteeing strict per-partition ordering.
• Partition Immutability: While you can add partitions to an existing Kafka topic, you can never decrease partition count. Increasing partitions alters the MurmurHash2 key distribution, meaning subsequent records for an existing key may map to a different partition.

Consumer Groups & Rebalancing Mechanics

Kafka consumers read data as part of a Consumer Group. The Kafka broker assigns partitions among consumers in the group to ensure parallel processing:

• Consumer Count <= Partition Count: If a topic has 12 partitions and a consumer group has 4 instances, each consumer reads from 3 partitions.
• Consumer Count > Partition Count: If a consumer group has 16 instances for a 12-partition topic, 4 consumer instances remain completely idle as backup standby nodes.
• Rebalancing Overhead: When a consumer joins or fails, Kafka initiates a group rebalance. Excessive rebalances disrupt processing throughput; MSK best practices recommend setting appropriate consumer heartbeat and timeout configurations (session.timeout.ms, max.poll.interval.ms).

---

Storage Management & Tiered Storage

MSK Local Storage vs. Tiered Storage

Traditional Kafka brokers store all partition log segments on attached local disks (EBS volumes). High message volumes require continuously resizing EBS volumes or adding expensive broker nodes solely for disk capacity.

Amazon MSK Tiered Storage solves this by separating compute from storage:

1. Primary Tier (Hot Data): Recent messages are stored in high-performance local EBS volumes (gp3) on MSK brokers for low-latency writes and immediate consumer reads.
2. Secondary Tier (Cold Data): Older log segments are automatically offloaded to Amazon S3. Tiered storage allows practically unlimited, cost-effective data retention (months or years) without scaling broker nodes.

Applications read seamlessly from Tiered Storage using standard Kafka APIs without code modifications; the MSK broker fetches segment data from S3 transparently.

---

Schema Governance with AWS Glue Schema Registry

Data quality and contract enforcement are major challenges in high-speed Kafka streaming. A single producer deploying an updated payload schema can break downstream analytics consumers—a phenomenon known as a poison pill message.

AWS Glue Schema Registry integrates with MSK to enforce schema evolution control:

• Supported Formats: Apache Avro, JSON Schema, and Protocol Buffers (Protobuf).
• Compatibility Modes: BACKWARD, FORWARD, FULL, and NONE.
  • BACKWARD (Default): Ensures new schemas can read data produced by the previous schema version.
  • FULL: Guarantees dual-way compatibility between old and new schemas.
• Execution Flow: Producers serialize records using the Glue Schema Registry client library. The library validates the schema against the Glue Registry before sending the record to MSK. If invalid, registration fails client-side before poisoning the Kafka topic.

---

Architectural Comparison Matrix: MSK Provisioned vs. MSK Serverless vs. Kinesis

---

Code Example: Kafka Java Producer using AWS IAM Authentication

package com.aws.dataengineer.msk;

import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;

public class MSKIamProducer {
    public static void main(String[] args) {
        String bootstrapServers = "b-1.msk-serverless.us-east-1.amazonaws.com:9098";
        
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        
        // AWS IAM Authentication Configuration for MSK Serverless
        props.put("security.protocol", "SASL_SSL");
        props.put("sasl.mechanism", "AWS_MSK_IAM");
        props.put("sasl.jaas.config", "software.amazon.msk.auth.iam.IAMLoginModule required;");
        props.put("sasl.client.callback.handler.class", "software.amazon.msk.auth.iam.IAMClientCallbackHandler");

        Producer<String, String> producer = new KafkaProducer<>(props);
        
        String topic = "order-events-topic";
        String key = "customer-9821";
        String value = "{\"order_id\": \"ord-5511\", \"amount\": 149.50}";

        ProducerRecord<String, String> record = new ProducerRecord<>(topic, key, value);
        
        producer.send(record, (RecordMetadata metadata, Exception exception) -> {
            if (exception == null) {
                System.out.printf("Message produced successfully! Topic: %s | Partition: %d | Offset: %d%n",
                        metadata.topic(), metadata.partition(), metadata.offset());
            } else {
                exception.printStackTrace();
            }
        });
        
        producer.close();
    }
}