2.1 Ingesting Batch and Streaming Data for ML

Ingesting Batch and Streaming Data for ML

High-performance machine learning models depend on clean, scalable, and resilient data ingestion pipelines. In enterprise AWS architectures, data originates from diverse sources—transactional databases, telemetry sensors, clickstream event streams, mobile applications, and third-party SaaS platforms. As an ML engineer preparing for the AWS Certified Machine Learning Engineer — Associate (MLA-C01) exam, you must know how to select, configure, and optimize AWS ingestion services based on throughput requirements, latency constraints, file formatting, and downstream training modalities.

---

1. Batch Data Ingestion Architecture with Amazon S3

Amazon Simple Storage Service (Amazon S3) serves as the primary object storage foundational layer for machine learning data lakes. Structuring batch data ingestion into S3 requires maximizing network throughput and handling multi-gigabyte or multi-terabyte raw datasets efficiently.

+-----------------------------------------------------------------------------+
|                   BATCH DATA INGESTION PATTERNS TO S3                       |
|                                                                             |
|   [On-Premises NAS / HDFS] ---> [AWS DataSync]         ---> [S3 Raw Bucket] |
|                                                                             |
|   [Remote Clients / Edge]  ---> [S3 Transfer Accel]    ---> [S3 Raw Bucket] |
|                                                                             |
|   [Large Objects >100 MB]  ---> [S3 Multipart Upload]  ---> [S3 Raw Bucket] |
|                                                                             |
|   [Hadoop / EMR Clusters]  ---> [S3DistCp]             ---> [S3 Data Lake]  |
+-----------------------------------------------------------------------------+

Key S3 Batch Ingestion Mechanisms

1. S3 Multipart Upload:

   • Mechanism: Breaks large files into discrete byte-range parts (from 5 MB up to 5 GB per part) and uploads them in parallel.
   • Threshold Rules: Mandatory for single objects exceeding 5 GB; strongly recommended for any object greater than 100 MB.
   • Resilience: If a network interruption occurs, only the failed part needs to be retransmitted rather than restarting the entire multi-gigabyte file.
   • Automated Lifecycle Cleanup: Always configure an S3 Lifecycle rule with AbortIncompleteMultipartUpload to automatically delete orphan, uncompleted parts after a set number of days (e.g., 7 days) to prevent unexpected storage costs.

2. S3 Transfer Acceleration (S3TA):

   • Mechanism: Enables fast, secure transfers of files over long geographical distances between client applications and S3 buckets.
   • Network Path: Utilizes Amazon CloudFront's globally distributed Edge Locations. Data enters the AWS edge network at the closest Point of Presence (PoP) and is routed over the congestion-free, optimized AWS private backbone to the destination S3 bucket.
   • Exam Indicator: Use S3TA when distributed global clients or remote IoT gateways upload batch training datasets directly into a centralized single-region S3 bucket.

3. S3DistCp (S3 Distributed Copy):

   • Mechanism: An open-source extension of Apache DistCp optimized to work with Amazon S3 on Amazon EMR clusters.
   • Use Case: Massively parallel data movement between Amazon S3 and HDFS, or between distinct S3 buckets across different accounts or regions. It uses MapReduce to distribute large-scale data copy operations across EMR worker nodes.

4. AWS DataSync:

   • Mechanism: An online data transfer service that automates moving data between on-premises storage (NFS, SMB, HDFS) and AWS storage (S3, EFS, FSx).
   • Features: Provides built-in data integrity verification (MD5/SHA256 checksums), bandwidth throttling, automatic encryption in transit (TLS), and up to 10 Gbps network utilization per agent task.

---

2. Streaming Ingestion Architectures: Kinesis vs. Firehose vs. MSK

Real-time machine learning workflows—such as fraud detection, real-time personalization, and predictive maintenance—require streaming architectures capable of ingesting continuous event streams with sub-second or near-real-time latency.

+-----------------------------------------------------------------------------+
|                STREAMING DATA PIPELINE DECISION MATRIX                      |
|                                                                             |
|   Continuous Event Stream                                                   |
|         |                                                                   |
|         +---> Need sub-second latency / multiple custom ML consumers?       |
|         |     ---> AMAZON KINESIS DATA STREAMS (with Enhanced Fan-Out)      |
|         |                                                                   |
|         +---> Need serverless direct-to-S3 delivery in Parquet format?      |
|         |     ---> AMAZON DATA FIREHOSE (Glue Catalog Conversion)           |
|         |                                                                   |
|         +---> Existing Kafka ecosystem / multi-topic / complex topologies?   |
|               ---> AMAZON MSK (with MSK Connect to S3)                      |
+-----------------------------------------------------------------------------+

Deep Dive: Amazon Kinesis Data Streams

• Sharding Mechanics:
  • In Provisioned Mode, throughput is determined by the number of active shards.
  • Ingest Capacity: Each shard provides 1 MB/second or 1,000 records/second for write operations.
  • Standard Consumer Capacity: Each shard provides 2 MB/second total read throughput shared across all standard consumers via GetRecords API.
• Enhanced Fan-Out (EFO):
  • Dedicated throughput of 2 MB/second per shard per registered consumer.
  • Data is pushed to consumers over an HTTP/2 connection using the SubscribeToShard API.
  • Reduces propagation latency from ~200 ms down to ~70 ms.
  • MLA-C01 Scenario: When multiple independent ML applications (e.g., a real-time fraud scoring model and an anomaly detection service) must read from the same stream simultaneously without throttling each other, use Enhanced Fan-Out.
• Partition Keys & Resharding:
  • A partition key groups data by shard. Poorly chosen partition keys (e.g., static string or low cardinality) cause hot shards, resulting in ProvisionedThroughputExceededException errors.
  • Dynamic scaling: Execute Shard Splitting to increase capacity for high-traffic keys, and Shard Merging to reduce costs during low-volume periods.

Deep Dive: Amazon Data Firehose (formerly Kinesis Data Firehose)

• Serverless Ingestion & Buffering:
  • Firehose buffers incoming streaming records before writing them to the destination.
  • Buffer Size: Configurable from 1 MB to 128 MB.
  • Buffer Interval: Configurable from 60 seconds to 900 seconds.
  • The buffer flushes whenever either the size or time limit is reached first.
• Inline Lambda Transformation:
  • Firehose can invoke an AWS Lambda function synchronously to transform raw JSON records, extract specific telemetry attributes, mask sensitive PII, or add metadata before delivering data to S3.
• Server-Side Format Conversion (JSON to Parquet/ORC):
  • Firehose can automatically convert incoming JSON data to columnar Apache Parquet or Apache ORC formats before storing it in Amazon S3.
  • Requires referencing an existing schema defined in the AWS Glue Data Catalog.
  • Exam Impact: This eliminates the need to run separate post-ingestion EMR or Glue Spark batch conversion jobs, cutting storage footprint by up to 80% and speeding up downstream Athena/SageMaker training queries.

+-----------------------------------------------------------------------------+
|           AMAZON DATA FIREHOSE SERVER-SIDE PARQUET PIPELINE                 |
|                                                                             |
|   [Incoming JSON Events]                                                    |
|             |                                                               |
|             v                                                               |
|   [Amazon Data Firehose]                                                    |
|             |                                                               |
|             +---> 1. Buffer Hints (e.g., 128 MB / 300s)                     |
|             +---> 2. Optional: Inline AWS Lambda (PII Scrubbing)            |
|             +---> 3. Query AWS Glue Data Catalog Schema                     |
|             +---> 4. Convert JSON to Columnar Apache Parquet                |
|             |                                                               |
|             v                                                               |
|   [Amazon S3 Training Data Lake] (Snappy-compressed Parquet, Partitioned)   |
+-----------------------------------------------------------------------------+

Deep Dive: Amazon Managed Streaming for Apache Kafka (Amazon MSK)

• Enterprise Kafka Compatibility: Fully compatible with open-source Apache Kafka APIs, allowing existing Kafka producers, consumer groups, and stream processing code (Kafka Streams, Apache Flink) to migrate to AWS without code refactoring.
• MSK Connect: A fully managed, serverless feature of Amazon MSK that runs Kafka Connect sink and source connectors.
  • To ingest Kafka topics into an S3 ML data lake without provisioning EC2 instances, deploy an open-source Amazon S3 Sink Connector inside MSK Connect with automatic scaling.

Deep Dive: Amazon Managed Service for Apache Flink

• Managed Stateful Stream Processing: Amazon Managed Service for Apache Flink runs Apache Flink applications (DataStream API, Table API, or Flink SQL) without managing clusters. It reads continuously from Amazon Kinesis Data Streams or Amazon MSK and applies stateful, exactly-once stream processing.
• ML Relevance: Use Flink for low-latency, stateful stream feature computation — sliding event-time windows (e.g., transactions per card per 5 minutes), sessionization, and real-time aggregations — before features reach an online feature store or a real-time inference endpoint. This goes beyond Amazon Data Firehose's record-at-a-time Lambda transformations.
• Exam Indicator: When a scenario requires stateful, event-time windowed stream computation or complex stream joins at sub-second latency feeding ML inference, choose Amazon Managed Service for Apache Flink. Choose Kinesis Data Streams + Lambda for simple per-record transforms, and Amazon Data Firehose for near-real-time delivery into S3.

---

3. Database Ingestion with AWS DMS and Glue Connectors

Relational transactional databases (Amazon RDS, Amazon Aurora, on-premises Oracle / SQL Server) store critical business features. Ingesting these datasets into ML storage requires continuous synchronization without impacting production database performance.

+-----------------------------------------------------------------------------+
|               DATABASE INGESTION & CHANGE DATA CAPTURE (CDC)                |
|                                                                             |
|   [Production Relational DB] (PostgreSQL / MySQL / Aurora / Oracle)         |
|                 |                                                           |
|                 | (Reads Engine Transaction Log: WAL / Binlog / Redo)       |
|                 v                                                           |
|   [AWS Database Migration Service (DMS) Replication Instance]               |
|                 |                                                           |
|                 +---> 1. Phase 1: Full Load (Historical baseline snapshot)  |
|                 +---> 2. Phase 2: Ongoing CDC (Continuous delta stream)     |
|                 |                                                           |
|                 v                                                           |
|   [Amazon S3 Landing Zone] (Raw CSV / Parquet with CDC Op Flags: I, U, D)   |
|                 |                                                           |
|                 v                                                           |
|   [AWS Glue ETL / SageMaker Processing] ---> [S3 Curated Feature Store]     |
+-----------------------------------------------------------------------------+

AWS Database Migration Service (AWS DMS)

• Two-Phase Ingestion:
  1. Full Load: Migrates all existing historical rows from source database tables into target S3 buckets.
  2. Change Data Capture (CDC): Reads source database transaction logs (e.g., PostgreSQL Write-Ahead Logs [WAL], MySQL Binary Logs [Binlog], Oracle Redo Logs) to capture inserts, updates, and deletes continuously.
• Target Configuration in S3:
  • DMS outputs CDC records with an operational metadata column (Op flag: I for Insert, U for Update, D for Delete) and a timestamp.
  • Supports direct output in Apache Parquet format with date-based partitioning.

AWS Glue Connectors and Incremental Bookmarks

• When batch querying transactional databases using AWS Glue ETL jobs, enable Job Bookmarks (--job-bookmark-option job-bookmark-enable).
• Job bookmarks maintain state across scheduled ETL runs, tracking previously processed primary keys or timestamp columns so that each run processes only newly added or modified rows.

---

4. S3 Partition Layouts & File Optimization for ML Training

How data is partitioned and structured in Amazon S3 directly dictates I/O efficiency, network saturation, and training epoch latency in SageMaker, Spark, and Athena.

+-----------------------------------------------------------------------------+
|                      HIVE PARTITIONING STRUCTURE                            |
|                                                                             |
|   s3://ml-feature-store-prod/customer_churn/                                |
|   ├── year=2026/                                                            |
|   │   ├── month=08/                                                         |
|   │   │   ├── day=16/                                                       |
|   │   │   │   ├── part-00000-c000.snappy.parquet (256 MB)                   |
|   │   │   │   └── part-00001-c000.snappy.parquet (256 MB)                   |
|   │   │   └── day=17/                                                       |
|   │   │       └── part-00000-c000.snappy.parquet (256 MB)                   |
+-----------------------------------------------------------------------------+

Best Practices for ML Dataset S3 Storage:

1. Hive-Style Partitioning:

   • Follow key-value folder naming conventions: /prefix/partition_key=partition_value/ (e.g., year=2026/region=us-east-1/).
   • Enables Partition Pruning: Query engines (Athena, Spark, Glue, SageMaker Data Wrangler) scan only the directories matching filter predicates in SQL queries (WHERE year=2026 AND month=08), avoiding full bucket scans and slashing I/O costs.

2. The "Small File Problem" and File Consolidation:

   • The Problem: Ingesting streaming data directly into S3 every few seconds generates millions of tiny files (10 KB – 500 KB). This causes massive metadata listing overhead, S3 API throttling (503 Slow Down), and degrades training throughput.
   • Target File Size: Target 128 MB to 512 MB per Parquet/ORC file (matching Hadoop/Spark split block sizes).
   • Remediation: Use Amazon Data Firehose buffer sizing (128 MB buffer hint) or run scheduled AWS Glue compaction jobs to coalesce small files into optimized Parquet blocks.

3. Columnar Formats with Compression:

   • Convert raw CSV/JSON to Apache Parquet with Snappy or ZSTD compression.
   • Columnar storage allows training algorithms to load only the specific feature columns needed for a model, reducing memory bandwidth utilization and I/O wait times.

[!TIP] Exam Architecture Pattern: When an exam question describes an IoT sensor stream generating high-volume JSON logs that must be stored in S3 for regular batch ML model training at the lowest operational overhead and cost, choose Amazon Data Firehose with built-in Parquet format conversion and AWS Glue Data Catalog integration with an S3 buffer size of 128 MB.