3.4 Shared Storage (EFS/FSx) & Database Performance
Key Takeaways
- Amazon EFS provides serverless, POSIX-compliant shared file storage supporting Elastic, Provisioned, and Bursting throughput modes, alongside General Purpose (low latency) and Max I/O (high concurrent scale) performance modes.
- Amazon FSx provides purpose-built managed file storage, including FSx for Windows File Server (native SMB, Active Directory, DFS namespaces) and FSx for Lustre (ultra-high throughput and POSIX for HPC/ML integrated with S3).
- Critical Amazon RDS CloudWatch metrics include CPUUtilization, DatabaseConnections, FreeableMemory, DiskQueueDepth, and read/write latency; persistent disk queue depth combined with elevated latency indicates storage subsystem bottlenecks.
- RDS Performance Insights visualizes database load using Average Active Sessions (AAS) against the vCPU threshold line, categorizing bottlenecks into CPU, Lock, IO, and Network wait events.
- Amazon RDS Proxy pools and multiplexes database connections from serverless applications (AWS Lambda), preventing database memory exhaustion and reducing Multi-AZ failover times by up to 66%.
3.4 Shared Storage (EFS/FSx) & Database Performance
CloudOps Blueprint Focus: The AWS Certified CloudOps Engineer – Associate (SOA-C03) exam tests operational mastery of shared file systems and database infrastructure: EFS and FSx tuning, RDS CloudWatch metrics, Performance Insights diagnostics, and Amazon RDS Proxy connection pooling.
Shared File Storage: Cloud Architectures & Operational Tuning
Workloads spanning multiple compute instances often require shared file systems, enabling hundreds of instances, containers, and Lambda functions to mount a common POSIX or SMB filesystem concurrently.
Amazon EFS: Serverless Elastic File System
Amazon Elastic File System (EFS) provides a serverless NFSv4.0/NFSv4.1 file system where capacity scales automatically as files are added or deleted.
Throughput Modes
A file system's throughput determines its data transfer speed:
- Bursting (Default Legacy): Scales at 50 KiB/s per GiB (100 MiB/s per TiB), bursting to 100 MiB/s using credits. Small systems (e.g., 20 GB) provide a minuscule baseline (1 MiB/s) and quickly exhaust credits, causing severe throttling.
- Provisioned Throughput: Decouples throughput from storage size, allocating dedicated throughput (e.g., 100 MiB/s) for small datasets.
- Elastic Throughput (Recommended): Automatically scales based on real-time activity with zero provisioning, paying only for data read or written.
Performance Modes
EFS performance modes are selected at creation and cannot be changed afterward:
- General Purpose: Lowest latency per operation (sub-millisecond). Ideal for web serving, content management systems (WordPress), container storage, and home directories.
- Max I/O: Scales to thousands of concurrent instances and millions of aggregate IOPS with slightly higher per-operation latency. Designed for big data analytics, genomics, and media rendering.
Lifecycle Management
EFS Lifecycle Management transitions files unaccessed for a specified policy window (e.g., 30, 60, or 90 days) to EFS Infrequent Access (IA) or EFS Archive, cutting storage costs by up to 90% while preserving client access.
Amazon FSx: Purpose-Built Specialized File Systems
When applications require enterprise file protocols or extreme throughput, AWS provides Amazon FSx:
1. Amazon FSx for Windows File Server
- Protocol & Integration: Native Windows Server SMB file system (SMB 2.0 through 3.1.1).
- Enterprise Features: Integrates with Microsoft Active Directory for authentication and granular Windows Access Control Lists (ACLs).
- High Availability & Scale: Supports DFS namespaces and Multi-AZ deployments with automatic failover.
2. Amazon FSx for Lustre
- High Performance: A POSIX file system engineered for compute-heavy workloads (machine learning, HPC, rendering).
- Throughput & Speed: Delivers sub-millisecond latencies, millions of IOPS, and hundreds of gigabytes per second of throughput.
- S3 Repository Integration: Directly links to Amazon S3 as a fast scratch cache, loading objects lazily on demand and exporting results back.
| Feature | Amazon EFS | FSx for Windows | FSx for Lustre |
|---|---|---|---|
| Protocol | NFSv4.0 / NFSv4.1 | SMB 2.0 - 3.1.1 | POSIX |
| Client OS | Linux, macOS, Lambda | Windows, Linux, macOS | Linux |
| Latency | Low ms (sub-ms General Purpose) | Sub-millisecond | Sub-millisecond |
| Key Workload | Web serving, shared Linux dirs | Active Directory, Windows apps | HPC, Machine Learning, S3 Cache |
Amazon RDS Performance Monitoring & CloudWatch Metrics
Proactive management of Amazon Relational Database Service (RDS) instances requires tracking key CloudWatch operational metrics:
CPUUtilization: Host CPU percentage. Sustained utilization above 85% indicates query inefficiency or undersized compute.DatabaseConnections: Active client sessions. Spikes can exhaust memory and trigger connection errors.FreeableMemory: Available RAM for buffer pools. When freeable memory nears zero, the host swaps to disk, spiking latency.ReadLatency&WriteLatency: Elapsed time per storage operation. Values exceeding 20 ms indicate storage saturation.DiskQueueDepth: Outstanding I/O requests queued for storage. High queue depth with elevated latency confirms a storage bottleneck.
Deep Database Engine Diagnosis: RDS Performance Insights
CloudWatch metrics indicate that RDS is slow, but RDS Performance Insights reveals why by exposing engine wait states:
Average Active Sessions (AAS)
DB Load |
| /---\ [CPU: Active execution]
8 vCPUs + - - -/- - -\- - - Max vCPU Line (Threshold)
| / \ [IO: wait/io/table/space]
| / \ [Lock: transaction locks]
+------------------- Time --->
- DB Load (Average Active Sessions - AAS): Measures average sessions actively running queries or waiting on resources.
- The Max vCPU Line: A dashed line representing total host vCPU cores. If DB Load exceeds Max vCPU, the database is saturated and queries are queuing.
- Wait Event Breakdown: Categorizes bottlenecks into CPU (processing query logic), IO (
wait/io/table/space,db file sequential readwaiting on disk reads), Lock (transactions blocked on table/row locks), or Network (waiting on client transmissions). - Top SQL: Identifies specific queries generating the largest share of database load for targeted optimization.
Connection Optimization with Amazon RDS Proxy
Serverless architectures (such as AWS Lambda) scale horizontally into thousands of concurrent executions during traffic surges. Each Lambda instance opens an independent database connection.
The Serverless Connection Storm
Thousands of simultaneous direct TCP connections cause two major failures:
- Connection Exhaustion: The database hits its
max_connectionslimit and rejects new client connections. - Memory Starvation: Maintaining thousands of idle connection threads consumes gigabytes of server RAM, starving database buffer pools.
RDS Proxy Architecture & Failover Acceleration
Amazon RDS Proxy is a managed database proxy sitting between applications and RDS/Aurora:
- Connection Pooling & Multiplexing: Maintains a persistent pool of established database connections and multiplexes thousands of ephemeral client queries across them.
- Failover Acceleration: During Multi-AZ failovers, RDS Proxy retains open client connections, queues traffic, and redirects queries to the promoted primary instance, slashing failover recovery times by up to 66%.
- Security: Enforces TLS encryption and integrates with AWS Secrets Manager and IAM database authentication.
A serverless e-commerce application uses AWS Lambda functions to handle order placement. During flash sales, thousands of concurrent Lambda functions invoke simultaneously, each attempting to establish a direct connection to an Amazon RDS for MySQL database. The database experiences sudden memory exhaustion and rejects new orders with 'Too many connections' errors. How should the CloudOps team resolve this connection bottleneck without modifying database instance size?
A CloudOps engineer is investigating severe performance degradation on an Amazon RDS for PostgreSQL instance. In the RDS Performance Insights dashboard, the DB Load (Average Active Sessions) graph is significantly above the dashed Max vCPU line. The load graph is dominated by the color blue, and the wait event table lists wait/io/table/space and db file sequential read as the primary contributors to the load. What does this Performance Insights profile indicate?
An enterprise is migrating an on-premises legacy Windows accounting application to AWS. The application requires a shared file system accessible over the Server Message Block (SMB) protocol, requires integration with Microsoft Active Directory for user-level Access Control Lists (ACLs), and must support Distributed File System (DFS) namespaces for high availability across multiple availability zones. Which AWS shared storage solution meets these technical requirements?