2.2 High-Performance Storage for Training and Inference

High-Performance Storage for Training and Inference

Training deep neural networks and large-scale tabular models requires sustained high-throughput, low-latency access to training datasets. When GPU worker instances spend compute cycles waiting for data I/O from storage, training times increase and expensive accelerator hardware sits idle. Choosing the correct storage tier and data access mode in AWS directly impacts training cost, startup latency, and overall throughput.

---

1. Amazon SageMaker Training Input Data Modes

When launching a SageMaker training job via the Python SDK (sagemaker.estimator.Estimator), you configure how data is ingested from Amazon S3 into the training container via the input_mode parameter.

+-----------------------------------------------------------------------------+
|                   SAGEMAKER TRAINING INPUT DATA MODES                       |
|                                                                             |
|   1. FILE MODE:                                                             |
|   [Amazon S3] ===(Download All Data Ahead of Time)===> [Local EBS Disk]     |
|                                                              | (Local Read) |
|                                                              v              |
|                                                    [Training Script]        |
|                                                                             |
|   2. FAST FILE MODE:                                                        |
|   [Amazon S3] <---(On-Demand Chunk Streaming / POSIX Mount)--- [Training]   |
|                                                                             |
|   3. PIPE MODE:                                                             |
|   [Amazon S3] ===(Stream via Linux Named Pipe FIFO)===> [Memory / Stream]   |
+-----------------------------------------------------------------------------+

Comprehensive Comparison of SageMaker Input Modes

Practical Code Example: Configuring Input Modes

import sagemaker
from sagemaker.inputs import TrainingInput
from sagemaker.pytorch import PyTorch

# Configure TrainingInput channel with Fast File Mode
train_input = TrainingInput(
    s3_data="s3://ml-dataset-prod/imagenet-1k/",
    input_mode="FastFile",  # Options: 'File', 'Pipe', 'FastFile'
    content_type="application/x-image"
)

# Define PyTorch Estimator
estimator = PyTorch(
    entry_point="train.py",
    role=sagemaker.get_execution_role(),
    instance_count=4,
    instance_type="ml.p4d.24xlarge",
    framework_version="2.1.0",
    py_version="py310",
    # Notice: No need to allocate 2000 GB of expensive EBS storage
    volume_size=50
)

estimator.fit({"train": train_input})

[!NOTE] Fast File Mode vs. Pipe Mode on the Exam: Fast File Mode is the modern recommendation for arbitrary file formats (images, audio, video, custom PyTorch/TensorFlow datasets) because it requires zero code changes while avoiding upfront download delays. Pipe Mode provides maximum streaming throughput for sequential record streams (RecordIO/CSV) and built-in SageMaker algorithms (e.g., BlazingText, Image Classification, XGBoost).

---

2. Amazon S3 Storage Classes for ML Workloads

Amazon S3 provides multiple storage tiers with distinct performance, access latency, and cost structures.

+-----------------------------------------------------------------------------+
|                        S3 STORAGE TIER SPECTRUM                             |
|                                                                             |
|   [LOWEST LATENCY / HIGH IOPS]                         [LOWEST COST ARCHIVE]|
|                                                                             |
|   S3 Express   --->   S3 Standard   --->   Intelligent-   --->   Glacier    |
|   One Zone            (Multi-AZ)           Tiering               Deep Archive|
|   (Single-digit ms)   (11 9s Durability)   (Auto Cost Opt)       (Long-term) |
+-----------------------------------------------------------------------------+

1. Amazon S3 Express One Zone

• Architecture: Purpose-built, high-performance storage class that colocates data within a single Availability Zone in dedicated directory buckets (s3://bucket--az_id--x-s3/).
• Performance: Single-digit millisecond data access latency; up to 10x lower latency and 50% lower request costs compared to S3 Standard.
• ML Workload Fit:
  • High-frequency model checkpointing during distributed multi-node LLM training.
  • Storing hundreds of thousands of small image/audio files for immediate random access.
  • Eliminates S3 503 Slow Down throttling during intensive parallel training reads without provisioning external cache clusters.

2. Amazon S3 Standard & Intelligent-Tiering

• S3 Standard: Default multi-AZ storage providing 99.999999999% (11 9s) durability. Ideal for active training data lakes, validated feature stores, and frequently queried datasets.
• S3 Intelligent-Tiering: Automatically moves objects between three access tiers (Frequent, Infrequent [30 days], and Archive Instant Access [90 days]) based on changing access patterns with zero retrieval fees. Ideal for ML data lakes where research datasets have unpredictable access frequencies.

3. S3 Glacier Flexible & Deep Archive

• S3 Glacier Flexible Archive (formerly Glacier): Retrieval times from minutes to hours; suitable for historical dataset snapshots and model audit logs.
• S3 Glacier Deep Archive: Lowest-cost storage (retrieval within 12–48 hours); ideal for long-term regulatory compliance data retention (e.g., retaining clinical trial training inputs for 7–10 years).

---

3. Amazon FSx for Lustre for Distributed Deep Learning

When training massive deep learning models (computer vision, transformers, generative AI) across distributed GPU clusters (e.g., dozens of ml.p4d.24xlarge or ml.p5.48xlarge nodes), reading directly from S3 can encounter network bandwidth bottlenecks or API throttling.

+-----------------------------------------------------------------------------+
|           AMAZON FSx FOR LUSTRE WITH S3 DATA REPOSITORY                     |
|                                                                             |
|   [Amazon S3 Dataset Bucket] (Petabytes of raw training images/audio)       |
|                 |                                                           |
|                 | Data Repository Association (DRA)                         |
|                 v                                                           |
|   [Amazon FSx for Lustre Cluster] (NVMe SSD / Sub-ms Latency)               |
|   ├── Metadata Sync: All S3 files visible immediately                       |
|   ├── Lazy Loading: Byte-ranges loaded from S3 on first read (hsm_restore)  |
|   └── Subsequent Epochs: Read 100% locally from Lustre NVMe cache (HPC Speed)|
|                 |                                                           |
|                 | POSIX Mount (`/opt/ml/input/data/training`)               |
|                 v                                                           |
|   [SageMaker Distributed GPU Training Cluster] (PyTorch FSDP / DeepSpeed)   |
+-----------------------------------------------------------------------------+

Architectural Features of FSx for Lustre:

1. Data Repository Association (DRA): Links an S3 bucket or prefix directly to the Lustre filesystem.
2. Lazy Loading (On-Demand Hydration): When FSx for Lustre is created, it instantly populates directory metadata (file names, sizes, permissions) without copying file contents. When a training worker opens a file, FSx streams the specific byte ranges from S3 on demand (hsm_restore).
3. Multi-Epoch Acceleration: In epoch 1, data streams from S3 through FSx. In epochs 2 through 100+, all reads are served entirely from the local FSx NVMe/SSD parallel storage tier at hundreds of GB/s throughput and sub-millisecond latency, maximizing GPU utilization.
4. Data Repository Tasks (Export Sync): Training checkpoints written to /fsx/checkpoints/ can be automatically or asynchronously synced back to Amazon S3 as durable objects.
5. Deployment Options:
   • Scratch File System: Designed for temporary storage and shorter-term training jobs. Data is not replicated across servers; provides the highest compute-to-cost ratio.
   • Persistent File System: Designed for longer-term workloads. Highly available with automatic replication within an Availability Zone.

# Using FSx for Lustre with SageMaker Estimator via Python SDK
from sagemaker.inputs import FileSystemInput

fsx_input = FileSystemInput(
    file_system_id="fs-0123456789abcdef0",
    file_system_type="FSxLustre",
    directory_path="/fsx/imagenet/",
    file_system_access_mode="ro"  # Read-only or Read-write
)

estimator.fit({"training": fsx_input})

---

4. Amazon Elastic File System (EFS) for Shared ML Assets

Amazon EFS provides a serverless, fully elastic, multi-AZ POSIX-compliant Network File System (NFSv4).

+-----------------------------------------------------------------------------+
|                   AMAZON EFS IN SAGEMAKER STUDIO                            |
|                                                                             |
|   [SageMaker Studio Domain] (Multi-User Environment)                        |
|   ├── User A Studio Notebook  \                                             |
|   ├── User B Studio Notebook   ---> [Amazon EFS Volume (Multi-AZ)]          |
|   └── User C Studio Notebook  /     ├── `/home/sagemaker-user/`             |
|                                     ├── Custom Conda Virtualenvs            |
|                                     └── Shared Git Repositories / Scripts   |
+-----------------------------------------------------------------------------+

Use Cases for Amazon EFS in ML:

• SageMaker Studio User Workspaces: Serves as the persistent storage layer for SageMaker Studio domains. User home directories, custom Python packages, and Jupyter notebooks persist across kernel and instance lifecycle restarts.
• Shared Project Assets: Enables multiple data scientists and distributed instances to share utility scripts, tokenizer vocabularies, and pre-computed embedding caches concurrently.
• Throughput Modes: Configure Elastic Throughput for unpredictable notebook workloads, or Provisioned Throughput for dedicated batch preprocessing tasks.

---

5. Storage Architecture Comparison Matrix

[!IMPORTANT] Decision Framework for the MLA-C01 Exam:

1. If an exam question asks to train a model on TB-scale unstructured image data with minimal startup delay and no code modifications, select SageMaker Fast File Mode.
2. If the question involves multi-node GPU distributed deep learning across many epochs where S3 read latency is a bottleneck, select Amazon FSx for Lustre with S3 Data Repository Association.
3. If the question requires collaborative notebook storage shared across multiple SageMaker Studio users that persists across instance restarts, select Amazon EFS.