3.2 Amazon EBS Performance Metrics & Volume Tuning
Key Takeaways
- Amazon EBS gp3 volumes provide a baseline performance of 3,000 IOPS and 125 MB/s throughput independently of storage capacity, eliminating gp2's requirement to overprovision disk size to obtain I/O performance.
- Mission-critical workloads requiring sub-millisecond latency and extreme I/O performance use io2 Block Express, delivering up to 256,000 IOPS, 4,000 MB/s throughput, and 99.999% durability.
- Key CloudWatch metrics for identifying EBS bottlenecks include VolumeQueueLength, VolumeThroughputPercentage, and BurstBalance, with high queue length and elevated latency signaling I/O starvation.
- Micro-bursting occurs when rapid I/O bursts exhaust volume or instance bandwidth limits within sub-minute intervals, often masked by standard 1-minute or 5-minute CloudWatch metric averages.
- Elastic Volumes allow online modification of volume size, IOPS, and volume type with zero downtime, subject to a 6-hour cooldown period between modifications on a single volume.
3.2 Amazon EBS Performance Metrics & Volume Tuning
CloudOps Blueprint Focus: The AWS Certified CloudOps Engineer – Associate (SOA-C03) exam tests operational mastery of Amazon EBS storage systems: evaluating volume performance profiles, interpreting CloudWatch storage metrics, diagnosing micro-bursting and instance limits, and executing zero-downtime volume modifications with Elastic Volumes.
Amazon EBS Storage Fabric Architecture
Amazon Elastic Block Store (EBS) delivers persistent block storage for Amazon EC2 compute instances. EBS volumes are network appliances operating within an Availability Zone. Every I/O request traverses internal instance buses, passes through the hypervisor or AWS Nitro card, and crosses network links to replicated storage servers. Performance is bounded by both volume specifications and host EC2 instance limits.
EBS Volume Types & Performance Characteristics
Selecting the appropriate EBS volume type requires matching workload I/O profiles (random vs. sequential, small-block vs. large-block) against throughput, IOPS baselines, and cost:
+-------------------------------------------------------------------------+
| Amazon EBS Volume Taxonomy |
+------------------------------------+------------------------------------+
| Solid State Drives (SSD) | Hard Disk Drives (HDD) |
| (Transactional, Small-Block I/O) | (Throughput, Large Sequential) |
+------------------+-----------------+------------------+-----------------+
| General Purpose | Provisioned IOPS| Throughput Opt. | Cold HDD |
| (gp3 / gp2) | (io2 / io2 Block| (st1) | (sc1) |
| | Express) | | |
+------------------+-----------------+------------------+-----------------+
1. General Purpose SSD: gp3 vs. gp2
- gp3 (Recommended Default): Decouples storage capacity from performance. Every gp3 volume includes 3,000 IOPS and 125 MB/s throughput across all sizes (1 GB to 16 TB). Engineers can scale up to 16,000 IOPS and 1,000 MB/s throughput independently without adding storage, saving 20% per GB compared to gp2.
- gp2 (Legacy Burst Model): Couples performance to storage at 3 IOPS per GB. Volumes under 1,000 GB burst to 3,000 IOPS using I/O credits. When depleted, performance collapses to baseline (e.g., 100 GB drops to 300 IOPS). Guaranteeing 6,000 IOPS on gp2 required provisioning 2,000 GB, wasting capacity.
2. Provisioned IOPS SSD: io2 & io2 Block Express
- Engineered for mission-critical transactional databases (SAP HANA, Oracle, SQL Server) requiring sub-millisecond latency.
- io2 Block Express: Powered by the AWS Nitro System, delivers up to 256,000 IOPS, 4,000 MB/s throughput, and up to 64 TB volumes with 1,000 IOPS per GB. Durability is 99.999% (a 100x improvement over standard SSDs).
3. Magnetic Hard Disk Drives: st1 & sc1
- Throughput Optimized HDD (st1): Low-cost magnetic storage for frequent, large sequential I/O (MapReduce, Kafka logs, data warehouses), delivering up to 500 MB/s using burst credits.
- Cold HDD (sc1): Lowest-cost block storage for infrequently accessed sequential workloads.
- Operational Restriction: Neither st1 nor sc1 can serve as bootable root volumes.
| Volume Type | Max IOPS | Max Throughput | Latency Target | Common Workload |
|---|---|---|---|---|
| gp3 | 16,000 | 1,000 MB/s | Single-digit ms | Web servers, dev environments, medium DBs |
| gp2 | 16,000 | 250 MB/s | Single-digit ms | Legacy deployments (migrating to gp3) |
| io2 Block Express | 256,000 | 4,000 MB/s | Sub-millisecond | Core enterprise transactional databases |
| st1 | 500 | 500 MB/s | Tens of ms | Sequential streaming, big data, log processing |
| sc1 | 250 | 250 MB/s | Tens of ms | Infrequent sequential archives, cold data |
Critical CloudWatch EBS Performance Metrics
Monitoring EBS performance requires tracking volume-level CloudWatch metrics:
VolumeReadOps&VolumeWriteOps: Completed I/O operations per period. Average IOPS equals total operations divided by period duration in seconds.VolumeReadBytes&VolumeWriteBytes: Data transferred. Throughput (MB/s) equals total bytes divided by period duration.VolumeTotalReadTime&VolumeTotalWriteTime: Total time executing operations. Dividing total elapsed time by total operations yields average I/O latency.VolumeQueueLength: Number of outstanding read and write requests waiting in queue. A sustained high queue length combined with elevated latency signals storage saturation.VolumeThroughputPercentage: Percentage of provisioned throughput consumed.BurstBalance: Percentage of remaining burst credits (0% to 100%) for burstable volumes (gp2, st1, sc1). Depletion to 0% throttles performance to baseline.
Troubleshooting Bottlenecks: Micro-bursting vs. Instance Limits
When disk latency degrades performance, engineers must isolate the constraint:
Micro-bursting
Applications often execute sharp bursts of I/O lasting milliseconds. Standard CloudWatch 1-minute or 5-minute averages flatten these spikes, reporting an average utilization of only 30%. However, during those fractional seconds, operations exceed limits and queue up, stalling application threads. Engineers detect micro-bursting by tracking VolumeQueueLength spikes or inspecting operating system tools like iostat -xz 1 for elevated %util and await.
Instance-Level EBS Bandwidth Throttling
Every EC2 instance type possesses dedicated EBS-optimized bandwidth and IOPS limits. Attaching a gp3 volume provisioned for 10,000 IOPS and 500 MB/s to an m5.large instance results in hypervisor-level throttling, because that instance size sustains only a baseline of 3,600 IOPS and 650 Mbps (81.25 MB/s) of EBS bandwidth, bursting to 18,750 IOPS and 4,750 Mbps (593.75 MB/s) for a limited period each day. Excess I/O queues before reaching storage, resulting in elevated VolumeQueueLength while the volume appears underutilized in CloudWatch.
Elastic Volumes: Dynamic Tuning & Cooldown Constraints
Amazon EBS Elastic Volumes allows administrators to modify volume size, adjust provisioned IOPS and throughput, or alter volume types (e.g., gp2 to gp3) dynamically without unmounting filesystems, detaching volumes, or rebooting instances.
The Elastic Volume Lifecycle
- Modifying: The modification request is registered by the EBS control plane.
- Optimizing: Performance characteristics adjust. New capacity and baseline IOPS are immediately available, while background block restriping continues.
- Completed: Background restriping completes.
[Trigger Modification] ---> [Modifying State] ---> [Optimizing State] ---> [Completed State]
(Capacity Active) (Background restripe) (Cooldown active)
[!WARNING] Once a modification begins, AWS enforces a mandatory 6-hour cooldown period before another modification can be initiated on the same volume.
Operating System Filesystem Expansion
After Elastic Volumes expands the block allocation, the operating system filesystem must be extended online using partition utilities (growpart) and filesystem tools (xfs_growfs for XFS or resize2fs for ext4) without downtime.
A database administrator reports that a production PostgreSQL instance running on an m5.large instance with a 500 GB gp3 volume (provisioned for 10,000 IOPS and 500 MB/s throughput) experiences severe I/O latency during peak transactions. CloudWatch metrics show VolumeQueueLength is consistently elevated, yet VolumeThroughputPercentage and VolumeReadOps/VolumeWriteOps remain well below the volume's provisioned 10,000 IOPS and 500 MB/s limits. What is the root cause of this performance bottleneck?
A company maintains a legacy analytics server with an attached 3 TB gp2 EBS volume. The volume was originally provisioned at 3 TB solely to achieve 9,000 IOPS based on gp2's 3 IOPS/GB allocation formula, although the analytics application only stores 400 GB of active data. As a CloudOps engineer, how can you optimize this storage configuration to reduce ongoing operational costs while sustaining the required 9,000 IOPS performance?
An operations engineer uses the AWS CLI to expand a 100 GB gp3 volume to 300 GB on a production application server. Ten minutes later, the application team requests that the provisioned IOPS on the same volume be increased from 3,000 to 8,000 IOPS to handle an unexpected traffic surge. When attempting to execute the modification command, the AWS CLI returns an error. What is the cause of this failure?