1.4 Lakeflow & Delta Live Tables (DLT) Declarative Pipelines

1.4 Lakeflow & Delta Live Tables (DLT) Declarative Pipelines

Delta Live Tables (DLT) is a declarative framework within the Databricks Lakeflow ecosystem that simplifies the creation, management, and monitoring of reliable data pipelines. For the exam, you must master DLT syntax in both Python and SQL, understand materialization strategies, Change Data Capture (CDC), and data quality expectations. This section expands extensively on how DLT fundamentally shifts data engineering from imperative steps to declarative pipeline graphs.

Declarative Pipeline Syntax

Unlike traditional Spark Structured Streaming where you manually manage checkpoints, retries, schemas, and state, DLT abstracts infrastructure management. You declare the desired state of your data, and DLT orchestrates the execution graph, provisioning clusters dynamically and handling failures gracefully.

DLT Python Syntax

In Python, you define pipelines using decorators provided by the dlt module.

• @dlt.table: Defines a materialized table. It automatically creates and maintains a Delta table in the target catalog. DLT manages the underlying schema evolution and checkpointing.
• @dlt.view: Defines a temporary view that is available only within the pipeline execution. Views are not persisted to storage, making them ideal for intermediate transformations, data masking, or staging before final materialization.

To process streaming data, you combine the @dlt.table decorator with spark.readStream within the function body. DLT automatically manages the streaming checkpoints.

import dlt
from pyspark.sql.functions import col

@dlt.table(
  name="raw_events_stream",
  comment="Raw ingested events"
)
def get_raw_events():
  return spark.readStream.format("cloudFiles") \
    .option("cloudFiles.format", "json") \
    .load("/path/to/raw/data")

DLT SQL Syntax

In SQL, you define tables using the CREATE OR REFRESH syntax, which is specific to DLT.

• CREATE OR REFRESH STREAMING TABLE: Defines a table that incrementally processes append-only data exactly once. It is equivalent to a streaming read in Python.
• CREATE OR REFRESH MATERIALIZED VIEW: Defines a table that computes results over the entire dataset, though DLT will attempt to incrementally compute the results where possible.

CREATE OR REFRESH STREAMING TABLE raw_events_stream
AS SELECT * FROM cloud_files("/path/to/raw/data", "json");

Streaming Tables vs Materialized Views

Understanding when to use Streaming Tables versus Materialized Views is a core exam competency and critical for pipeline design.

• Streaming Tables: Designed for ingest and append-only workloads. They process each record exactly once. If a historical record in the source changes, the Streaming Table will not automatically detect or recalculate the change without a full refresh. They are optimized for high-throughput appending.
• Materialized Views (MVs): Designed for complex aggregations, joins, and scenarios where source data changes. MVs automatically track changes in upstream dependencies and efficiently recompute the results to guarantee the table reflects the exact state of the source queries. MVs abstract the complexity of figuring out what changed and recalculating only the necessary parts.

Change Data Capture (CDC) with APPLY CHANGES INTO

Handling CDC—applying inserts, updates, and deletes from a source system to a target table—is notoriously difficult in traditional Spark. DLT simplifies this with the APPLY CHANGES INTO API (SQL) or dlt.apply_changes() (Python).

To implement CDC, you need:

1. A source Streaming Table containing the change events (inserts, updates, deletes).
2. A target Streaming Table acting as the final sink.
3. The APPLY CHANGES INTO logic linking them.

A critical parameter is the sequence_by column. Because streaming events can arrive out of order, the sequence_by column (often a timestamp or incrementing ID) tells DLT how to resolve conflicts. If an update arrives before the initial insert, DLT uses the sequence_by value to ensure the final state is correct. Furthermore, you must define the keys (primary keys) to match records, and optionally, an apply_as_deletes condition.

dlt.create_streaming_table("target_users")

dlt.apply_changes(
  target = "target_users",
  source = "source_user_changes",
  keys = ["user_id"],
  sequence_by = col("updated_at"),
  apply_as_deletes = expr("operation = 'DELETE'"),
  except_column_list = ["operation"]
)

Enforcing Data Quality with Expectations

DLT natively integrates data quality checks called Expectations. These act as constraints on the data flowing through the pipeline. When an expectation is violated, DLT handles the record based on the defined severity.

• expect(name, condition): Logs the violation in the event log but allows the invalid record to pass into the target table. Useful for monitoring data quality without disrupting flow.
• expect_or_drop(name, condition): Discards the invalid record and logs the event, while allowing valid records to process normally.
• expect_or_fail(name, condition): Halts the entire pipeline immediately upon detecting a violation. Used for critical data where downstream contamination is unacceptable.

@dlt.table
@dlt.expect("valid_age", "age IS NOT NULL AND age > 0")
@dlt.expect_or_drop("valid_email", "email LIKE '%@%'")
def cleansed_users():
  return dlt.read("raw_users")

Implementing Quarantine Tables

A common architectural pattern tested on the exam is the Quarantine Table. Sometimes you cannot drop invalid data; it must be stored for auditing or manual correction.

To implement a quarantine pattern in DLT, you define two separate tables reading from the same source view.

1. The primary table uses a positive expectation (e.g., @dlt.expect('valid_id', 'id IS NOT NULL')) to capture good records.
2. The quarantine table uses the inverse logic (e.g., @dlt.expect('invalid_id', 'id IS NULL')) to capture the bad records.

By utilizing DLT's dependency graph, both tables process the stream simultaneously, effectively routing records based on quality criteria. This ensures no data is lost while maintaining pristine quality in your primary tables.

Control Flow Operators in Lakeflow Pipelines and Jobs

Real production pipelines rarely execute as a single linear chain of tasks. Branching, conditional execution, and looping are required to build adaptive, environment-aware workflows. The Databricks Data Engineer Professional exam explicitly tests your ability to create a pipeline component that uses control flow operators such as if/else and for/each. Databricks exposes two layers of control flow: orchestration-level control flow in Lakeflow Jobs, and pipeline-level control flow in Lakeflow Declarative Pipelines.

Lakeflow Jobs: If/Else and For Each Tasks

Lakeflow Jobs support dedicated task types that implement branching and looping directly in the job graph, without custom Python orchestration code.

If/else conditional tasks evaluate a condition against task values, job parameters, or dynamic value references, and route execution to the corresponding branch. The supported comparison operands are ==, !=, >, >=, <, and <=. A task publishes a value using taskValues, and a downstream If/else task can branch on that value — for example, running a model retraining task only when an upstream validation task reports a data-quality score above a threshold.

Run if conditional tasks are related: they gate a downstream task on the status of its upstream dependencies (All succeeded, At least one succeeded, None failed, All done, At least one failed, All failed). This is the idiomatic way to skip cleanup or alerting tasks when nothing went wrong.

For each tasks run another (nested) task in a loop, passing a different set of parameters to each iteration. The nested task is one of the standard Databricks task types (notebook, Python script, SQL, pipeline, and so on). This is the recommended pattern when you must process many partitions, tables, or tenants with identical logic but distinct parameters, replacing hand-rolled shell loops with a managed, observable construct.

Lakeflow Declarative Pipelines: The control_flow Block

Within Lakeflow Declarative Pipelines, the control_flow block brings conditional logic directly into declarative pipeline definitions. The two foundational constructs are if (decision-making) and foreach (iteration over a collection). Because the logic lives in the pipeline definition, it is version-controlled alongside the rest of the pipeline and auditable.

A typical use of if is environment-aware pipeline authoring: a single declarative definition can run one set of flows in production and a reduced set in a dev workspace by branching on an environment parameter. foreach iterates across a list — for example, ingesting a configurable list of source tables through the same flow definition, so one pipeline scales to many datasets without duplicated code.

Exam Scenarios and Common Traps

• Choose the right layer. If the decision depends on the status of an upstream task or a computed task value, use Lakeflow Jobs If/else or Run if tasks. If the decision is about which declarative flows run in a pipeline (for example, environment-specific flow sets), use the control_flow block.
• For each vs. partitionBy. Do not confuse For each tasks (which launch separate task runs per parameter set) with Spark partitionBy (which parallelizes within a single task). For each is orchestration-level looping; partitioning is data-level parallelism.
• Condition operands. If/else tasks support the six relational operands listed above; they do not support arbitrary SQL predicates. Move complex logic into an upstream task that sets a boolean task value, then branch on that value.
• Disabled tasks keep configuration and run history but skip execution; downstream Run if conditions are still evaluated, so disabling is not the same as deleting a task.