8.2 High-Throughput Offline Inference with Batch Transform
Key Takeaways
- SageMaker Batch Transform is designed for high-throughput offline inference across massive datasets in Amazon S3, provisioning ephemeral compute clusters that automatically terminate upon job completion to eliminate idle costs.
- `SplitType` determines how input files are partitioned for container ingestion (`None` sends whole files, `Line` splits CSV/JSONLines by newline, and `RecordIO` splits binary protobuf records).
- `BatchStrategy` controls mini-batch packaging (`MultiRecord` packs multiple records per HTTP request up to `MaxPayloadInMB` to maximize GPU/CPU utilization, whereas `SingleRecord` sends one record per request).
- `DataProcessing` parameters (`InputFilter`, `OutputFilter`, and `JoinSource='Input'`) enable filtering input features and automatically joining original record identifiers with model prediction outputs without writing custom Glue/Spark ETL scripts.
High-Throughput Offline Inference with Batch Transform
Not all machine learning applications require persistent, low-latency online endpoints. When predictions are generated over large, static datasets on a recurring schedule (such as nightly customer churn scoring, weekly sales forecasting, monthly portfolio risk valuations, or processing millions of archived documents), deploying persistent real-time endpoints results in massive compute idle waste and unnecessary operational overhead.
Amazon SageMaker Batch Transform is a fully managed distributed processing service engineered specifically for high-throughput, offline inference over massive datasets stored in Amazon S3. Batch Transform provisions an ephemeral cluster of compute instances, distributes input data partitions across workers, feeds mini-batches to model containers over local HTTP endpoints, persists inference results to Amazon S3, and immediately terminates all compute infrastructure upon completion.
1. Batch Transform Architectural Workflow
+--------------------------------------------------------------------------------------------------+
| BATCH TRANSFORM EXECUTION LIFECYCLE |
| |
| [Amazon S3 Input Bucket] |
| - s3://input/data_part_01.csv (10 million rows) |
| - s3://input/data_part_02.csv |
| | |
| v (Data Partitioning & SplitType='Line') |
| +------------------------------------------------------------------------------------------+ |
| | SAGEMAKER MANAGED TRANSFORM CLUSTER | |
| | | |
| | [Instance 1 (ml.c5.4xlarge)] [Instance 2 (ml.c5.4xlarge)] | |
| | +------------------------------------+ +------------------------------------+ | |
| | | Container HTTP Server (Port 8080) | | Container HTTP Server (Port 8080) | | |
| | | - InputFilter: $[1:] (Drops ID) | | - InputFilter: $[1:] (Drops ID) | | |
| | | - MultiRecord: 6 MB mini-batches | | - MultiRecord: 6 MB mini-batches | | |
| | | - Model scoring: /invocations | | - Model scoring: /invocations | | |
| | | - JoinSource: 'Input' (Joins ID) | | - JoinSource: 'Input' (Joins ID) | | |
| | +------------------------------------+ +------------------------------------+ | |
| +------------------------------------------------------------------------------------------+ |
| | |
| v (AssembleWith='Line' & S3 Upload) |
| [Amazon S3 Output Bucket] |
| - s3://output/data_part_01.csv.out (ID + Prediction columns joined, ordered matching input) |
| - s3://output/data_part_02.csv.out |
| | |
| v |
| [Cluster Automatically Terminates - Zero Idle Billing Incurred] |
+--------------------------------------------------------------------------------------------------+
2. Core Configuration Parameters
Fine-tuning Batch Transform parameters is essential to maximize container saturation, prevent out-of-memory (OOM) crashes, and ensure deterministic record ordering.
+--------------------------------------------------------------------------------------------------+
| BATCH TRANSFORM KEY PARAMETERS MATRIX |
| |
| Parameter Allowed Values Functional Purpose |
| --------- -------------- ------------------ |
| SplitType None, Line, RecordIO How input files are divided into |
| individual inference records. |
| |
| BatchStrategy MultiRecord, SingleRecord Whether to pack multiple records |
| into a single HTTP POST request. |
| |
| MaxPayloadInMB Integer (1 to 100 MB, def: 6) Maximum payload size sent to the |
| container in each HTTP request. |
| |
| MaxConcurrentTransforms Integer (>= 0, def: 0 auto) Maximum parallel HTTP requests sent |
| to a single worker container. |
| |
| AssembleWith None, Line How prediction outputs are stitched |
| together in the final S3 object. |
+--------------------------------------------------------------------------------------------------+
2.1 SplitType
SplitType defines how SageMaker splits input S3 files before feeding them to the container:
None(Default): The entire file is treated as a single undivided input payload. Required for file formats that cannot be split by line (e.g., raw JPEG/PNG image files, DICOM medical files, or whole uncompressed Parquet/ZIP archives).Line: The input file is split by newline (\n) delimiters. Mandatory for tabular CSV, text files, andJSONLines(application/jsonlines). Enables SageMaker to distribute subsets of rows across concurrent workers.RecordIO: Splits binary data formatted in MXNet RecordIO-protobuf format.
2.2 BatchStrategy
BatchStrategy determines how individual records are batched into HTTP requests sent to the model container's /invocations endpoint:
MultiRecord: SageMaker packs as many consecutive records as possible into a single HTTPPOSTpayload until reachingMaxPayloadInMB. This dramatically increases inference throughput by saturating CPU/GPU tensor vectorization and minimizing HTTP round-trip overhead. Highly recommended for tabular models (XGBoost, Linear Learner).SingleRecord: SageMaker sends exactly one record per HTTPPOSTinvocation. Essential when processing heavy deep learning inputs (such as high-resolution images or variable-length audio clips) or when the serving container does not support vectorized mini-batch parsing.
2.3 Sizing and Concurrency Tuning: MaxPayloadInMB & MaxConcurrentTransforms
MaxPayloadInMB: Sets the maximum size (in megabytes, up to 100 MB) of each mini-batch payload. IncreasingMaxPayloadInMBimproves throughput for tabular data. However, if set too high for memory-intensive models, the container may run out of memory (OOM) and return HTTP500or413 Request Entity Too Largeerrors.MaxConcurrentTransforms: The maximum number of concurrent HTTP requests that SageMaker can send simultaneously to each worker container instance. Setting this parameter to1or2is critical for memory-constrained GPU workloads to avoid VRAM exhaustion.
2.4 AssembleWith
AssembleWith defines how model outputs from individual batch requests are concatenated into the final S3 output file:
None(Default): Writes the raw binary output chunks into S3 without delimiter formatting.Line: Appends a newline (\n) character after each prediction record, ensuring the output file is a clean, row-by-row CSV or JSONLines file whose line numbers map one-to-one with the input file.
3. Zero-ETL Data Association with DataProcessing
In standard ML pipelines, input datasets contain non-predictive metadata columns (such as customer_id, account_number, or timestamp) alongside feature columns. However, trained model containers (such as SageMaker XGBoost or Scikit-learn) expect pure feature arrays and crash if unexpected identifier strings are passed to /invocations.
Historically, engineers had to write separate AWS Glue ETL jobs to strip IDs before inference and join them back after inference. SageMaker Batch Transform DataProcessing natively solves this with zero extra ETL code.
+--------------------------------------------------------------------------------------------------+
| DATAPROCESSING FILTERING PIPELINE |
| |
| Input CSV Row: [ "CUST_98214", 34, 1.2, 0.45, 0 ] (ID is column 0, features are columns 1-4) |
| | |
| v |
| [1. InputFilter: '$[1:]'] ---> Strips CUST_98214 ---> Passes [ 34, 1.2, 0.45, 0 ] to container |
| | |
| v |
| [Model Container /invocations] |
| Generates Prediction: [ 0.874 ] (Churn Prob) |
| | |
| +-----------------------------------+ |
| | |
| v |
| [2. OutputFilter: '$'] ---> Extracts prediction output [ 0.874 ] |
| | |
| v |
| [3. JoinSource: 'Input'] ---> Joins original input with prediction |
| | |
| v |
| Final S3 Output: [ "CUST_98214", 34, 1.2, 0.45, 0, 0.874 ] |
+--------------------------------------------------------------------------------------------------+
The Three DataProcessing Parameters:
InputFilter: Uses JSONPath syntax (or CSV column slicing) to filter data before sending it to the model container.- Example for CSV:
$[1:]strips the first column (e.g.,customer_id) and forwards columns 1 through N (the numerical features) to the model. - Example for JSONLines:
$.featuresextracts only the nested feature array.
- Example for CSV:
OutputFilter: Uses JSONPath to select specific fields from the container's inference output to be included in the output file.- Example:
$.predictionor$[0]filters out auxiliary metadata from container responses, retaining only raw scores.
- Example:
JoinSource: Specifies whether to join the original input data with the model prediction output.None(Default): Writes only the model predictions to S3.Input: Concatenates the original input record with the model prediction. When combined withAssembleWith='Line', the final S3 output file preserves complete end-to-end lineage (identifying record + predictions) without requiring post-processing database joins.
4. Python SDK Implementation Example
import sagemaker
from sagemaker.transformer import Transformer
session = sagemaker.Session()
role = sagemaker.get_execution_role()
# Configure the Batch Transformer
transformer = Transformer(
model_name="xgboost-churn-production-v2",
instance_count=4, # 4 distributed worker instances
instance_type="ml.m5.2xlarge",
strategy="MultiRecord", # Pack multiple records per HTTP POST
max_payload=6, # 6 MB per mini-batch
max_concurrent_transforms=4, # 4 parallel HTTP requests per worker
assemble_with="Line", # Delimit output predictions with newlines
output_path="s3://customer-scoring-prod/batch-output/2026-08-16/",
sagemaker_session=session
)
# Execute the Transform Job with DataProcessing (Zero-ETL Join)
transformer.transform(
data="s3://customer-scoring-prod/raw-features/",
data_type="S3Prefix",
content_type="text/csv",
split_type="Line", # Split input CSV files by newline
input_filter="$[1:]", # Strip Column 0 (CustomerID) before inference
join_source="Input", # Join original input row with prediction output
output_filter="$[0, -1]" # Retain only CustomerID ($[0]) and ChurnScore ($[-1])
)
# Wait for transform completion
transformer.wait()
5. Batch Transform vs. Real-Time vs. Asynchronous Inference Decision Guide
+--------------------------------------------------------------------------------------------------+
| OFFLINE VS ONLINE INFERENCE DECISION MATRIX |
| |
| Criterion Batch Transform Online Endpoints (RT / Async) |
| --------- --------------- ----------------------------- |
| Latency Expectation Hours / Offline batch Sub-second up to 1 hour |
| Input Source Large S3 dataset files Individual HTTP requests / S3 URIs |
| Compute Lifecycle Ephemeral (job auto-stops) Persistent or Scale-to-Zero |
| Data Pre/Post Processing Built-in DataProcessing Requires Serial Pipeline / Lambda |
| Pricing Billed only for job run Hourly instance or per-request |
| Best Workloads Daily Churn, Weekly Demand API Backends, Mobile Apps, Real-Time |
+--------------------------------------------------------------------------------------------------+
[!TIP] Exam Trap Alert: If an exam question asks how to run offline batch inference on CSV data where the first column contains a
user_idthat the model rejects, do NOT choose creating a separate AWS Glue ETL job to drop the column. The most architecturally efficient, serverless, and AWS-native solution is configuringDataProcessingwithInputFilter='$[1:]'andJoinSource='Input'directly on the SageMaker Batch Transform job.
A telecommunications company generates a 2 TB dataset of customer usage logs in Amazon S3 every Sunday evening. An ML engineer must generate churn risk scores for all 40 million subscribers by Monday morning. The dataset is stored as newline-delimited CSV files, where the first column is subscriber_id and the remaining 48 columns are numerical features. The trained XGBoost model expects exactly 48 feature inputs and fails if the subscriber_id string is included. The marketing team requires an output CSV file containing the subscriber_id joined with the predicted churn probability. What is the most operationally efficient solution?
An ML engineer is running a SageMaker Batch Transform job on a cluster of 8 ml.g5.2xlarge GPU instances to generate embeddings from 500,000 document text files stored in Amazon S3. The transform job fails repeatedly with HTTP 500 container crash errors and out-of-memory (OOM) logs in Amazon CloudWatch. The engineer observes that the default configuration attempts to send 20 MB multi-record batches with 8 concurrent requests per worker instance. Which adjustments will resolve the container memory exhaustion while maintaining distributed execution?
A healthcare imaging system processes 100,000 DICOM radiological images every night using a deep learning segmentation model. Each DICOM image is a standalone binary file of roughly 35 MB stored in Amazon S3. The model container expects to receive one complete image file per inference request and outputs a binary segmentation mask. Which combination of Batch Transform parameters must the ML engineer configure?
An insurance analytics platform runs weekly batch risk assessments across millions of policyholder records using SageMaker Batch Transform. The data science team notices that the output predictions in S3 are misaligned with downstream database schemas because the model outputs only raw probability floats, losing the corresponding policy identifier. The team wants to include the original policy_id and retain all input features alongside the prediction in the output files without writing post-processing Python scripts. Which parameter configuration achieves this?