16.1 Orchestrating ML Workflows with Managed Service for Apache Airflow

The exam guide names Managed Service for Apache Airflow as a pipeline orchestration option. On April 15, 2026, Google announced that Cloud Composer is evolving to become Managed Service for Apache Airflow (Managed Airflow) to reflect its open-source foundation. Older materials and code still say "Composer," and the Airflow Google provider keeps compatibility names alongside new ManagedAirflow... operator aliases.

What Managed Airflow Provides

Apache Airflow defines workflows as DAGs (directed acyclic graphs) of tasks in Python files. Managed Airflow runs Airflow for you:

Environment generations are named Managed Airflow (Gen 3), (Gen 2), and (Legacy Gen 1). Airflow 3 is generally available in Gen 3 environments, alongside Airflow 2 builds. You manage environments through the console, gcloud, the API (still called the Cloud Composer API), or Terraform. Logs go to Cloud Logging and metrics to Cloud Monitoring.

DAGs for ML Workflows

The Apache Airflow Google provider includes operators for Agent Platform, BigQuery, Dataflow, Managed Spark, and more. Agent Platform-related operators include:

with DAG("nightly_churn", schedule="0 2 * * *", start_date=datetime(2026, 9, 1), catchup=False):
    load = BigQueryInsertJobOperator(task_id="build_features", configuration={...})
    wait = GCSObjectExistenceSensor(task_id="wait_for_crm_export", bucket="crm", object="{{ ds }}/export.csv")
    train = RunPipelineJobOperator(task_id="train_pipeline", display_name="churn-{{ ds }}",
                                   template_path="gs://ml/pipelines/churn.yaml",
                                   parameter_values={"run_date": "{{ ds }}"}, region="us-central1", project_id="p")
    score = CreateBatchPredictionJobOperator(task_id="score", ...)
    publish = BigQueryInsertJobOperator(task_id="publish_scores", configuration={...})
    [load, wait] >> train >> score >> publish

Airflow features that matter for ML operations:

• Schedules (cron or presets) with catchup and backfill for past periods.
• Sensors that wait for files, tables, or external jobs.
• Retries, SLAs, and alerting callbacks.
• Templating (for example, {{ ds }}) to pass the logical run date into queries and pipeline parameters.
• Event-based triggering, such as DAGs triggered by Cloud Storage changes.
• Custom plugins and Python dependencies installed in the environment.

Managed Airflow vs. Agent Platform Pipelines

The common hybrid pattern

Let Managed Airflow orchestrate the enterprise data workflow (ingest from SaaS, wait for upstream systems, build warehouse tables), then trigger an Agent Platform Pipeline for the ML portion with RunPipelineJobOperator. The ML pipeline keeps lineage, caching, and evaluation gates, and Airflow keeps cross-system dependencies and backfills.

Operational Considerations

• Environment sizing: scale workers and schedulers to DAG volume. Heavy compute belongs in BigQuery, Dataflow, Managed Spark, or Agent Platform jobs, not on Airflow workers.
• Identity: give the environment's service account least-privilege roles for the services its DAGs call.
• Networking: use private IP environments and VPC Service Controls where required.
• Version upgrades: plan Airflow 2 to Airflow 3 migrations, because DAG APIs changed.
• Cost: the environment runs continuously, which suits organizations already running many DAGs but is expensive for a single weekly ML job.

Designing Reliable ML DAGs

• Idempotent tasks: rerunning a task for the same logical date should overwrite, not duplicate, outputs (for example, write to a date-partitioned table).
• Pass the logical date into every query and pipeline parameter, so backfills reproduce historical runs correctly.
• Keep tasks thin: Airflow tasks should submit and monitor work in managed services, not do the heavy processing.
• Separate data readiness from ML logic: sensors and data checks in Airflow, and ML validation gates inside the Agent Platform Pipeline.
• Alert on failures and SLA misses so late data doesn't silently produce stale models.

Worked Scenario

A retailer's data platform team runs 400 Airflow DAGs that load POS, e-commerce, and supplier data. The ML team needs weekly demand model retraining that must start only after all supplier feeds for the week land, and must backfill four weeks after an outage.

• Add a DAG in the existing Managed Airflow environment with sensors for the supplier feeds and catchup/backfill for missed weeks.
• Trigger the existing Agent Platform Pipeline (validation, training, evaluation, and conditional deployment) with RunPipelineJobOperator, passing the week as a parameter.
• The ML team keeps lineage and evaluation gates in Pipelines. The platform team keeps one orchestration layer for data dependencies.