4.5 Containers, Lambda & Distributed Processing Performance

4.5 Containers, Lambda & Distributed Processing Performance

The correct compute service follows workload shape. Lambda is an event-driven function runtime. Amazon ECS and Amazon EKS run containerized applications with longer lifetimes and more control. EMR and Glue provide distributed data frameworks. Selecting compute by language alone misses the operational constraints the exam tests.

Start with volume, velocity, and variety

Characterize the input before selecting compute. Volume is the amount retained or processed per run; it drives storage, scan bytes, partition count, and whether work must be distributed. Velocity is the arrival and change rate; it drives shard throughput, batching windows, backpressure, checkpoints, and recovery lag. Variety is the number and variability of formats, schemas, and source systems; it drives deserialization, cataloging, schema evolution, normalization, and quarantine design.

For example, a nightly 20 TB Parquet backfill has high volume but low arrival velocity and fits a distributed batch engine. A continuous 2 MB/s JSON feed has lower daily volume but stricter velocity and schema-variation concerns, so a stream with checkpoints and a raw replay copy is more important. Quantify all three instead of equating “big data” only with byte count.

Lambda fit and limits

Lambda fits record handlers, lightweight transformations, schedulers, and control-plane automation. It has a maximum execution duration, so a multi-hour backfill belongs in Glue, EMR Serverless, AWS Batch, or a container workflow. Invocation payload limits and source integration limits also matter, especially now that Kinesis can accept intermittently large records that some downstream integrations cannot consume at the same size.

Increasing Lambda memory also increases available CPU. Benchmark duration and total cost instead of assuming the smallest memory is cheapest. Important controls include:

• Reserved concurrency: Guarantees and caps concurrency for one function, protecting a database or vendor API from uncontrolled scale-out.
• Provisioned concurrency: Keeps initialized environments ready when predictable low startup latency justifies the cost.
• Event-source batch size and window: Amortize invocation overhead, but larger batches increase retry scope unless partial batch response is used.
• Parallelization factor: Increases concurrent batches per Kinesis shard while preserving ordering for the same partition key.
• Timeout and memory: Set from measurements and leave enough time for cleanup or a failure destination.

Initialize SDK clients and static resources outside the handler so a warm environment can reuse them. Reuse database connections carefully through RDS Proxy, and cache secrets for a bounded interval with the Parameters and Secrets extension.

Lambda storage choices

The /tmp directory is configurable ephemeral storage attached to one execution environment. It is useful for decompression, local sort spill, or model files reused by warm invocations, but it is not shared or durable.

Lambda can mount an Amazon EFS access point when functions need a shared POSIX file system larger than ephemeral storage. The function must have VPC network connectivity, mount-target reachability, security-group rules, and file permissions. EFS adds network latency and does not turn a 15-minute function into a batch engine. Store durable objects in S3 when file-system semantics are unnecessary.

ECS and EKS

Use containers when a task needs a custom operating-system package, a long-running consumer, more predictable CPU or memory, local daemon behavior, or a runtime model outside Lambda's fit.

Right-size both requests and limits in Kubernetes. An inflated request wastes nodes; a low memory limit causes termination. For a parallel file workload, partition the work into independent messages and use service autoscaling from queue depth or custom lag metrics.

Distributed-computing mechanics

Distributed systems divide data into partitions processed by workers. Speedup stops when work is not divisible or coordination dominates. Common bottlenecks are:

1. Shuffle: Grouping or joining by key moves records across the network.
2. Skew: One hot key gives one task much more data than peers.
3. Serialization: Large row objects or inefficient formats consume CPU and network.
4. Driver concentration: Collecting a dataset or huge metadata list on one driver defeats distribution.
5. Small files: Scheduling and object-request overhead dominates useful work.

Data structures and algorithms in pipelines

A hash map gives expected O(1) key lookup for in-memory deduplication or joins, but memory grows with unique keys; an unbounded stream therefore needs windows, state expiry, or an external store. A queue or deque models ordered work and bounded buffering. A heap maintains a top-k set or priority order in O(log k) per update without sorting every record. A tree keeps ordered keys for range traversal; database B-trees make point and range access efficient but add write and index-maintenance cost. A graph models relationships and pipeline dependencies; breadth-first or depth-first traversal grows with the vertices and edges explored, so constrain path depth and filters.

Algorithm choice affects the distributed plan. Hash partitioning co-locates equal keys but can create a hot partition. Sort-merge operations support ordered/range work but add O(n log n) local sorting and shuffle. A DAG topological order is appropriate for dependencies only when there is no cycle. State the access pattern, input scale, memory bound, and failure behavior before choosing the structure.

Performance workflow

Measure input bytes, output bytes, CPU, memory, shuffle bytes, spill, task duration, and skew before scaling. Push filters and projections to the source, select columnar formats, broadcast only genuinely small dimensions, and compact files. Then increase parallelism until the source, network, or destination becomes the bottleneck.

For resilient processing, every unit of work needs an identifier and bounded retry. A container or Lambda retry can execute twice, so commit outputs transactionally or to a unique staging path before publishing them. Compute elasticity without idempotency produces faster duplicates.