4.3 Dynamic Scalability Through Caching

4.3 Dynamic Scalability Through Caching

Scaling compute nodes and relational databases horizontally or vertically introduces steep cost curves and architectural bottlenecks. Relational databases face finite connection pools and disk I/O ceilings when hit with intense read spikes. Distributed caching absorbs high-frequency queries, flattens database loads, and delivers sub-millisecond response times. For the CloudOps exam, engineers must master multi-tier caching architectures, Amazon ElastiCache engine internals (Valkey/Redis vs. Memcached), caching patterns, eviction strategies, and CloudWatch performance metrics.

Multi-Tier Caching Architecture

Enterprise cloud architectures deploy caching across three distinct infrastructure tiers:

1. Edge Tier (Amazon CloudFront): Content Delivery Network (CDN) with hundreds of Points of Presence (PoPs) globally. CloudFront caches static assets (images, stylesheets, media) and cacheable REST/GraphQL responses using HTTP Cache-Control headers, terminating requests closest to users.
2. Application Tier (Local In-Memory Cache): Ephemeral process memory (e.g., local heap maps). Delivers nanosecond lookups but remains isolated per instance or container, creating cache synchronization challenges across auto-scaled fleets.
3. Database Tier (Amazon ElastiCache): Centralized, distributed in-memory data store shared across all application instances. Delivers sub-millisecond read latency, shields primary relational databases from query spikes, and provides centralized session management.

Amazon ElastiCache Engine Comparison: Valkey / Redis OSS vs. Memcached

Amazon ElastiCache offers managed engines based on Valkey / Redis OSS and Memcached:

Redis Cluster Modes

• Cluster Mode Disabled: Consists of 1 primary node and up to 5 read replicas in a single shard. All writes route strictly to the primary node. While read capacity scales horizontally across replicas, write throughput is strictly bounded by the single primary, and dataset size cannot exceed a single node's memory.
• Cluster Mode Enabled: Partitions data across 1 to 500 shards using 16,384 internal hash slots. Each shard contains 1 primary and up to 5 read replicas, enabling horizontal scaling of both read and write capacity across multi-terabyte datasets.

Caching Patterns & Invalidation Strategies

Applications interact with distributed caches using three primary patterns:

1. Cache-Aside (Lazy Loading)

The application directly orchestrates cache lookups:

1. Application queries the cache. On a cache hit, it returns data immediately.
2. On a cache miss, the application queries the database, writes the result to the cache, and returns it to the client.

• Trade-offs: Resilient against cache outages (falls back to database); caches only requested keys. However, misses incur three network round-trips, and data can become stale if updated directly in the database.
• Mitigation: Configure an explicit Time to Live (TTL) on all keys to force periodic re-fetching.

2. Write-Through

The application writes data to the cache and the persistent database simultaneously:

• Trade-offs: Cache data is always fresh, eliminating read-time database misses. However, write latency increases because writes must succeed across two systems, and memory can be flooded with data that is rarely read. Usually combined with eviction policies or TTLs.

3. Write-Around

Data is written directly to the database without populating the cache. Data enters the cache only upon a subsequent cache miss via Cache-Aside. This prevents cache pollution from high-volume write operations that are seldom read (e.g., historical audit logs).

Eviction Policies & Thundering Herd Mitigation

When memory reaches maxmemory, ElastiCache invokes an eviction policy to remove existing keys:

• volatile-lru / allkeys-lru: Evicts Least Recently Used keys (either with explicit TTL or across all keys).
• volatile-lfu / allkeys-lfu: Evicts Least Frequently Used keys.
• noeviction: Returns an out-of-memory error on write operations when memory is full.

Mitigating Cache Stampede (Thundering Herd)

A cache stampede occurs when a heavily queried key expires, causing thousands of concurrent threads to hit the backend database simultaneously on cache miss. CloudOps engineers mitigate stampedes using:

• Distributed Mutex Locking: Threads attempt to acquire a distributed lock in Redis (using SET key value NX PX 10000). Only the single thread obtaining the lock queries the database and refreshes the cache; all other threads wait or consume stale cached values.
• Probabilistic Early Expiration (XFetch): The application calculates randomized early expiration windows, refreshing keys in the background before the hard TTL expires.

Monitoring ElastiCache with CloudWatch

CloudOps engineers must track critical ElastiCache CloudWatch metrics:

• EngineCPUUtilization: Critical for Redis / Valkey. Because Redis executes commands in a single-threaded loop, a 4-vCPU instance shows 25% host CPUUtilization when its Redis engine core is 100% saturated. Always monitor EngineCPUUtilization to detect Redis bottlenecks. For Memcached, host CPUUtilization accurately reflects multi-threaded CPU usage.
• SwapUsage: Indicates OS memory paging. In an in-memory database, swapping causes severe latency spikes (from microseconds to milliseconds). SwapUsage must not exceed 50 MB.
• Evictions: Spikes indicate memory exhaustion where valid keys are being dropped, requiring cluster scale-up or shard addition.
• CurrConnections: Spikes indicate connection pooling failures or application connection leaks.

ElastiCache supports Application Auto Scaling to automatically add or remove read replicas or scale shards based on EngineCPUUtilization and memory metrics.