4.3 Serverless Orchestration with AWS Step Functions

4.3 Serverless Orchestration with AWS Step Functions

In modern cloud architecture, complex data engineering pipelines consist of multiple decoupled services—extracting raw files from S3, running AWS Glue crawlers, executing EMR Spark jobs, querying Athena, and updating Amazon Redshift tables. Managing these dependencies using custom code, cron jobs, or nested Lambda calls leads to fragile, unmaintainable architectures.

AWS Step Functions solves this by providing a serverless visual orchestration service that manages workflow state, handles retries, and executes complex control flow using declarative Amazon States Language (ASL) JSON definitions.

---

Standard Workflows vs. Express Workflows

AWS Step Functions supports two workflow types tailored for distinct data engineering use cases:

Comparative Architectural Matrix

---

Service Integration Patterns: Request-Response vs. .sync vs. Callback

Step Functions provides three distinct execution models when integrating with external AWS services:

1. Request-Response (Default): Step Functions waits for the integrated API request to return, but it does not poll an asynchronous job started by that response. Optimized synchronous Lambda invocation waits for the function to return; work that the function launches asynchronously still needs a job-run pattern or callback.
2. Run a Job (.sync / Optimized Integration): Step Functions calls an AWS service API (e.g., glue:startJobRun.sync or elasticmapreduce:addJobFlowSteps.sync) and pauses state machine execution. Step Functions automatically polls the downstream job status and resumes execution only when the job returns SUCCEEDED, or triggers error handling if it returns FAILED or TIMED_OUT.
3. Wait for Callback with Task Token (.waitForTaskToken): Pauses the state machine and generates a unique token. The workflow remains paused until an external process calls SendTaskSuccess or SendTaskFailure with the token (ideal for human approval steps or third-party webhooks).

---

Core ASL State Types for Data Engineering

Amazon States Language (ASL) defines workflow structures using specific state types:

• Task: Executes a unit of work (invoking a Lambda function, running a Glue job, starting an EMR step).
• Choice: Evaluates JSONPath or JSONata logic predicates to perform conditional branching.
• Parallel: Executes static, independent execution branches concurrently.
• Map: Iterates over an array of items, executing identical processing steps across array elements.
  • Inline Map: Runs within the parent workflow history, supports up to 40 concurrent iterations, and is constrained by the 256 KiB state input/output payload limit and the Standard Workflow event-history limit.
  • Distributed Map: High-scale processing mode that reads manifest files or S3 prefixes directly, spawning up to 10,000 concurrent child workflow executions to process millions of S3 objects simultaneously.
• Pass: Passes input to output or injects static JSON data for testing.
• Wait: Delays workflow execution for a relative time or until an absolute timestamp.
• Fail / Succeed: Terminates state machine execution with explicit error or success status.

---

Input and Output Payload Transformation Lifecycle

Data engineering pipelines frequently pass JSON state payloads between states. Without transformation, intermediate states risk cluttering or exceeding Step Functions' 256 KB execution payload limit. Step Functions manages data filtering across 5 distinct JSONPath fields:

1. InputPath: Selects a specific subset of the incoming JSON state input to send to the task execution.
2. Parameters: Constructs a custom JSON payload passed to the service integration, using JSONPath expressions (.$) to sample dynamic values from the input.
3. ResultSelector: Filters the raw response returned by the AWS service integration before merging it into the workflow state.
4. ResultPath: Specifies where in the original workflow state payload to insert the task result. Setting "ResultPath": "$.GlueResult" appends the result without overwriting existing state input data. Setting "ResultPath": null discards task output entirely, passing the original input unaltered.
5. OutputPath: Filters the final combined JSON state before passing it to the subsequent state.

---

High-Scale Parallel Processing with Distributed Map States

For large-scale data engineering workflows, processing millions of files stored in Amazon S3 requires dynamic scaling beyond standard Lambda concurrency. Step Functions Distributed Map handles high-throughput batch processing natively:

• Direct S3 Inventory & Manifest Readers: Distributed Map configures an ItemReader that directly scans an Amazon S3 bucket prefix or reads an S3 inventory CSV/JSON manifest file, bypassing the need to pre-fetch object lists with a Lambda function.
• Concurrent Child Execution Scaling: Spawns up to 10,000 parallel child workflow executions. Each child execution can run as a Standard Workflow (for long steps) or an Express Workflow (for rapid, low-cost processing).
• Batching Configurations (ItemBatcher): Groups multiple S3 objects into single worker invocations using MaxItemsPerBatch (e.g., 500 files per worker) or MaxInputBytesPerBatch to optimize worker throughput and reduce overhead.
• ResultWriter to Amazon S3: Exports execution results, child status logs, and failure details directly into a target S3 bucket, preventing payload threshold exhaustion when consolidating results across millions of child items.

---

Robust Error Handling: Retry & Catch Constructs

Step Functions provides native error handling at the individual state level, avoiding pipeline crashes caused by transient network glitches or temporary AWS service limits.

{
  "Retry": [
    {
      "ErrorEquals": ["Glue.ConcurrentRunsExceededException", "States.TaskFailed"],
      "IntervalSeconds": 15,
      "MaxAttempts": 3,
      "BackoffRate": 2.0
    }
  ],
  "Catch": [
    {
      "ErrorEquals": ["States.ALL"],
      "Next": "HandlePipelineFailure"
    }
  ]
}

Key Retry Parameters for Data Engineers

• ErrorEquals: Array of specific error codes (e.g., EMR.ThrottlingException, Glue.ConcurrentRunsExceededException) or wildcards (States.TaskFailed, States.ALL).
• IntervalSeconds: Initial delay in seconds before performing the first retry attempt.
• BackoffRate: Multiplier applied to the retry interval on each subsequent failure (e.g., 2.0 doubles wait time: 15s, 30s, 60s).
• MaxAttempts: Maximum number of retry attempts before giving up and transitioning to a Catch block or failing the state.

---

Production Orchestration Scenarios & Architecture Patterns

Data engineers frequently encounter specific architectural patterns on the AWS Data Engineer (DEA-C01) exam:

1. AWS Glue ETL + Amazon Redshift Data API Pipeline

When orchestrating data warehouse updates, Step Functions can invoke glue:startJobRun.sync. After Glue succeeds, use the Redshift Data API AWS SDK integration to call asynchronous ExecuteStatement, preserve a stable ClientToken across retries, and poll DescribeStatement (or route a completion event) before continuing. Redshift Data API does not expose an executeStatement.sync resource.

2. EMR Serverless Job Submission with Step Functions

For EMR Serverless workloads, Step Functions can submit a job with emr-serverless:startJobRun.sync; the .sync integration waits for the job to finish. Waiting is not automatic retry behavior. Configure ASL Retry and Catch policies for service errors and, when appropriate, an EMR Serverless job retry policy for failed job attempts.

3. Cross-Account Data Pipeline Orchestration

To orchestrate workflows across separate AWS accounts (e.g., Ingestion Account to Analytics Account), Step Functions tasks assume an IAM role in the target account (sts:AssumeRole) or publish execution events to a centralized Amazon EventBridge event bus that triggers a secondary state machine in the target account.

---

Declarative ASL State Machine Definition

The following ASL JSON document orchestrates an AWS Glue ETL job using .sync, catches errors, and conditionally triggers an Amazon SNS notification:

{
  "Comment": "Production Data Engineering Pipeline Orchestration",
  "StartAt": "TriggerGlueETLJob",
  "States": {
    "TriggerGlueETLJob": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName": "DailySalesAggregationJob",
        "Arguments": {
          "--ExecutionDate.$": "$.execution_date"
        }
      },
      "Retry": [
        {
          "ErrorEquals": ["Glue.ConcurrentRunsExceededException"],
          "IntervalSeconds": 30,
          "MaxAttempts": 3,
          "BackoffRate": 2.0
        }
      ],
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "Next": "HandlePipelineFailure"
        }
      ],
      "Next": "VerifyJobStatus"
    },
    "VerifyJobStatus": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.JobRun.JobRunState",
          "StringEquals": "SUCCEEDED",
          "Next": "PipelineSuccess"
        }
      ],
      "Default": "HandlePipelineFailure"
    },
    "HandlePipelineFailure": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": {
        "TopicArn": "arn:aws:sns:us-east-1:123456789012:PipelineAlerts",
        "Subject": "ETL Pipeline Failure Alert",
        "Message.$": "$.Error"
      },
      "Next": "PipelineFailed"
    },
    "PipelineFailed": {
      "Type": "Fail",
      "Error": "ETLJobFailed",
      "Cause": "Glue ETL job execution failed or timed out."
    },
    "PipelineSuccess": {
      "Type": "Succeed"
    }
  }
}