4.4 Airflow Pipeline Orchestration with Amazon MWAA

4.4 Airflow Pipeline Orchestration with Amazon MWAA

Apache Airflow is the industry-standard open-source platform for programmatically authoring, scheduling, and monitoring complex data workflows as Directed Acyclic Graphs (DAGs) written in Python. Amazon Managed Workflows for Apache Airflow (MWAA) is a managed service that automates the deployment, scaling, database management, and security infrastructure for Apache Airflow on AWS.

Understanding MWAA component architecture, S3 synchronization workflows, auto-scaling worker mechanics, and native AWS security integrations is essential for the DEA-C01 exam.

---

Amazon MWAA System Architecture & Network Models

MWAA provisions a highly available Apache Airflow environment distributed across two isolated VPC boundaries:

Dual VPC Architecture Overview

+-------------------------------------------------------------------------+
| Customer VPC (Execution VPC)                                            |
|  - Subnets across 2 AZs (Private Subnets)                               |
|  - CeleryExecutor Workers (AWS Fargate)                                 |
|  - VPC Endpoints (S3, CloudWatch, EMR, Glue, Secrets Manager)           |
+-------------------------------------------------------------------------+
                                     | Private VPC Connectivity
+-------------------------------------------------------------------------+
| AWS-Managed Control Plane VPC                                           |
|  - Airflow Web Server                                                   |
|  - Airflow Scheduler                                                    |
|  - Amazon RDS PostgreSQL Metadata Storage (Encrypted via KMS)           |
+-------------------------------------------------------------------------+

Key Architectural Components

• Airflow Web Server: Provides the visual UI dashboard for managing DAGs. Protected by AWS IAM authentication or IAM Identity Center SSO.
• Airflow Scheduler: Monitors DAG definitions, evaluates schedule intervals, and dispatches task instances to the Celery executor queue.
• CeleryExecutor Workers: Worker instances running on AWS Fargate that pull tasks from the queue and execute them. Workers scale out automatically based on task queue depth up to max_workers and scale in to min_workers when idle.
• Metadata Database: Fully managed Amazon RDS PostgreSQL database holding operational state, task history, and connection definitions.

Network Web Server Access Modes

1. Public Network Mode: The Airflow Web UI is accessible over the internet via a public endpoint secured by IAM user/role authorization.
2. Private Network Mode: The Airflow Web UI endpoint is accessible only within the customer's VPC or via connected networks (VPC Peering, AWS Site-to-Site VPN, or AWS Direct Connect).

---

S3 Bucket Structure & Environment Synchronization

MWAA environments monitor a dedicated Amazon S3 bucket to sync DAG definitions, libraries, and environment configurations:

s3://my-company-mwaa-bucket/
├── dags/
│   ├── sales_etl_dag.py
│   └── customer_cleanup_dag.py
├── requirements.txt
├── plugins.zip
└── startup_script.sh

File Roles & Update Dynamics

• dags/ Folder: Stores Python DAG scripts. MWAA automatically scans and syncs changes to the Airflow Scheduler every ~30 seconds.
• requirements.txt: Declares custom Python packages (pip dependencies) required by DAGs. After uploading a new version to the versioned S3 bucket, update the MWAA environment and select that S3 object version. Merely overwriting the key does not install the new dependencies. Test constraints against the environment's Airflow and Python versions before deployment.
• plugins.zip: Contains custom Airflow operators, hooks, sensors, or UI extensions.
• startup_script.sh: Shell script executed on environment startup to install system binary packages or configure environment variables prior to Airflow startup.

---

Native AWS Integrations & Provider Operators

MWAA leverages the open-source apache-airflow-providers-amazon package, allowing data engineers to trigger AWS services natively using specialized Airflow operators:

Key AWS Airflow Operators

• GlueJobOperator: Triggers an AWS Glue ETL job run and polls for completion.
• EmrServerlessStartJobOperator: Submits a Spark job to an EMR Serverless application.
• SageMakerTransformOperator: Executes a SageMaker batch transform inference job.
• AthenaOperator: Runs SQL queries against Amazon Athena and outputs results to S3.
• StepFunctionStartExecutionOperator: Triggers execution of an AWS Step Functions state machine.

---

Secrets Management: AWS Secrets Manager Integration

Hardcoding database credentials, API keys, or connection URIs inside Python DAG code or standard Airflow environment variables is a major security vulnerability.

MWAA integrates natively with AWS Secrets Manager and AWS Systems Manager Parameter Store as secret backends. When Airflow looks up a connection or variable, it queries Secrets Manager automatically using standard naming prefixes:

• Airflow Connection Lookup: airflow/connections/<connection_id>
• Airflow Variable Lookup: airflow/variables/<variable_name>

Secrets Manager JSON Secret Structure

For an Airflow connection named redshift_dw, create a secret in Secrets Manager under the path airflow/connections/redshift_dw:

{
  "conn_type": "redshift",
  "host": "redshift-cluster.123456789012.us-east-1.redshift.amazonaws.com",
  "login": "aws_admin",
  "password": "SecurePassword123!",
  "port": 5439,
  "schema": "dev"
}

---

Python Airflow DAG Example

The following complete Airflow DAG demonstrates orchestrating an EMR Serverless Spark job followed by an AWS Glue Crawler, utilizing native Amazon provider operators:

from datetime import datetime, timedelta
from airflow import DAG
from airflow.providers.amazon.aws.operators.emr import EmrServerlessStartJobOperator
from airflow.providers.amazon.aws.operators.glue_crawler import GlueCrawlerOperator

default_args = {
    'owner': 'data_engineering',
    'depends_on_past': False,
    'start_date': datetime(2026, 8, 1),
    'email_on_failure': False,
    'retries': 2,
    'retry_delay': timedelta(minutes=5),
}

with DAG(
    dag_id='emr_serverless_to_glue_catalog',
    default_args=default_args,
    schedule_interval='0 2 * * *', # Daily at 02:00 UTC
    catchup=False,
    tags=['production', 'emr_serverless', 'glue'],
) as dag:

    # 1. Execute Spark Job on EMR Serverless
    run_spark_job = EmrServerlessStartJobOperator(
        task_id='run_spark_etl',
        application_id='00f1a2b3c4d5e6f7',
        execution_role_arn='arn:aws:iam::123456789012:role/EMRServerlessJobRole',
        job_driver={
            'sparkSubmit': {
                'entryPoint': 's3://my-company-mwaa-bucket/scripts/etl_spark.py',
                'entryPointArguments': ['--output-path', 's3://my-company-data-lake/curated/'],
                'sparkSubmitParameters': '--conf spark.executor.cores=4 --conf spark.executor.memory=16g',
            }
        },
        configuration_overrides={
            'monitoringConfiguration': {
                's3MonitoringConfiguration': {
                    'logUri': 's3://my-company-mwaa-bucket/emr_serverless_logs/'
                }
            }
        },
        wait_for_completion=True
    )

    # 2. Trigger AWS Glue Crawler to sync S3 partitions
    trigger_crawler = GlueCrawlerOperator(
        task_id='sync_glue_catalog',
        config={'Name': 'curated-sales-data-crawler'},
        wait_for_completion=True
    )

    # Define Task Dependency
    run_spark_job >> trigger_crawler

Operational failure handling

Airflow retries belong on idempotent tasks. A retrying task that blindly appends output can duplicate data, so use a run identifier, overwrite a deterministic partition, or perform a transactional merge. Configure task timeouts and pools so one downstream system cannot consume every worker slot. Use deferrable operators or service-aware sensors where supported so a task waiting for an EMR or Glue job does not hold a worker unnecessarily.

An MWAA environment update can replace workers. Pin dependencies, validate the requirements file locally with the AWS-provided MWAA container tooling, and use the graceful replacement strategy when active work must finish. CloudWatch receives scheduler, worker, web server, DAG processing, and task logs when enabled; inspect the relevant log group before treating every failed DAG as a service-capacity problem.