7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory

7.6 Choosing an Ingestion Tool: Lakeflow Connect, Notebooks, & Azure Data Factory

DP-750 Exam Focus: Two blueprint bullets - "Choose an appropriate data ingestion tool, including Lakeflow Connect, notebooks, and Azure Data Factory" and "Ingest data by using Lakeflow Connect, including batch and streaming." Sections 7.1 through 7.5 taught the notebook-based paths. This section adds the managed path and gives the decision rule.

---

1. What Lakeflow Connect Is

Lakeflow Connect is Databricks' managed ingestion service. You configure a source and a destination; Databricks operates the pipeline. Every managed connector produces an ingestion pipeline that is:

• Governed by Unity Catalog - the destination is a Unity Catalog table, with the usual privilege model
• Powered by serverless compute and Lakeflow pipelines
• Incremental - efficient incremental reads and writes rather than full reloads
• Writing into streaming tables as its destination objects

The Six Connector Families

If none of these fit, you can build a custom connector and run it in your workspace.

---

2. Component Architecture Differs by Connector Type

This is the detail most often tested, because it explains why a database connector costs more and has more moving parts than a SaaS connector.

A query-based connector is the lighter-weight database option: it queries the source directly on a schedule with no gateway and no staging storage, at the cost of not capturing every intermediate change the way CDC does.

---

3. Incremental Ingestion and Orchestration

Incremental behavior. On the first run, the pipeline ingests all selected data from the source while simultaneously beginning to track changes. Every subsequent run uses that change tracking to load only what changed since the prior run, where the source supports it. The exact mechanism depends on the source - SQL Server can use both change tracking and CDC, while the Salesforce connector selects a cursor column from a fixed list of options. Some sources and some individual tables do not support incremental ingestion at all.

Orchestration. You can run an ingestion pipeline on one or more custom schedules. For each schedule you add, Lakeflow Connect automatically creates a Job, with the ingestion pipeline as a task inside it. You can then add further tasks to that job - a silver transformation, a data quality check, a dashboard refresh. For database connectors, the ingestion gateway runs in its own job as a continuous task.

Networking. SaaS connectors call the source's APIs and are automatically compatible with serverless egress controls. Cloud database connectors can connect over Private Link, or the gateway can be deployed inside a VNet peered with the one hosting the database. On-premises databases connect through services such as Azure ExpressRoute.

Failure recovery. As a managed service, a connector automatically retries with exponential backoff. When an error requires intervention - expired credentials, for example - the connector stores the last cursor position so it can resume from that point rather than losing data.

Monitoring. Event logs, cluster logs, pipeline health metrics, and data quality metrics are all available, and system.billing.usage tracks connector cost.

---

4. Creating Connections and Deploying Pipelines as Code

Ingestion pipelines can be deployed with Declarative Automation Bundles (the bundle format also referred to on the exam as Databricks Asset Bundles), which brings source control, code review, testing, and CI/CD to ingestion - and lets the same definition target dev, staging, and prod workspaces.

For connectors that use API-only authentication (all database connectors and most SaaS connectors), connections can be created programmatically instead of through Catalog Explorer:

# Databricks CLI - the JSON body matches the Connections REST API schema
databricks connections create --json '{
  "name": "sqlserver_orders_prod",
  "connection_type": "SQLSERVER",
  "options": { "host": "orders-prod.database.windows.net", "port": "1433" }
}'

The same call can be made from a notebook through the Connections API, or from a bundle pre-deploy script or job task before the pipeline is deployed.

The OAuth exception. Connectors whose only authentication option is browser-based OAuth (OAuth U2M) cannot be created programmatically, because the initial token requires interactive sign-in. Confluence, Google Ads, HubSpot, Jira, Meta Ads, Slack, Slack Audit Logs, TikTok Ads, and Zendesk Support fall into this group - a real constraint on fully automated CI/CD.

---

5. The Decision Rule

Lakeflow Connect Versus Lakehouse Federation

When a source supports both, Databricks recommends Lakeflow Connect if performance on higher data volumes and lower latency are priorities. Federation is positioned for ad hoc reporting and proof-of-concept access.

Where Azure Data Factory Still Belongs

ADF is named in the DP-750 audience profile, and the exam expects you to know its role rather than dismiss it:

• Orchestration across services - copy from an on-premises system, then trigger a Databricks job, then refresh a Power BI dataset.
• Sources Databricks does not connect to - ADF's connector catalog is broader than the managed connector list.
• Existing investment - many enterprises already run ADF as the enterprise scheduler, with Databricks as the compute.

What ADF should not be used for is re-implementing transformation logic that belongs in Lakeflow Spark Declarative Pipelines, or polling a folder that Auto Loader would handle incrementally and more cheaply.

---

6. Exam Traps

• "Lakeflow Connect writes to a Delta table you create first" is wrong. The destinations are streaming tables that the ingestion pipeline creates and owns.
• A SaaS connector has no gateway. Only database CDC connectors add a gateway and staging storage.
• Adding a schedule creates a Job, not just a cron entry. That job is where you attach downstream tasks.
• Not every source is incremental. Some sources and tables do not yet support incremental ingestion, so a scenario stating a full reload each run is not automatically a misconfiguration.
• Auto Loader is not part of Lakeflow Connect. Auto Loader is a cloud-file source you use in notebooks and declarative pipelines; Lakeflow Connect is the managed connector service.