1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture

1.1 DP-750 Exam Overview & Azure Databricks Lakehouse Architecture

Welcome to the study guide for the Microsoft Certified: Azure Databricks Data Engineer Associate (DP-750) certification. This certification validates your end-to-end technical expertise in designing, implementing, securing, and optimizing enterprise data engineering solutions using Azure Databricks, Apache Spark, Delta Lake, and Unity Catalog.

Understanding the underlying architectural boundaries—specifically how Azure Databricks coordinates services between Microsoft-managed infrastructure and your Azure subscription—is a critical prerequisite for passing the DP-750 exam and building resilient enterprise data pipelines.

---

1. DP-750 Certification Blueprint & Exam Mechanics

The DP-750 exam targets professional data engineers responsible for ingesting, transforming, orchestrating, and governing batch and streaming datasets on Azure. Microsoft publishes the outline as four skill areas (skills measured as of March 11, 2026), each broken into functional groups and illustrative bullets:

Watch the weights. Skill areas 3 and 4 are each roughly double skill areas 1 and 2. Together, Prepare and process data and Deploy and maintain data pipelines and workloads account for 60-70% of the scored exam, so Delta Lake, Auto Loader, Lakeflow, and production operations deserve the majority of your preparation time.

Verified Exam Logistics

Feature currency. Microsoft states that most DP-750 questions cover features that are generally available (GA), and that Preview features may appear only when they are commonly used. When a Databricks capability has both a legacy and a current name -- Delta Live Tables and Lakeflow Spark Declarative Pipelines, or Workflows and Lakeflow Jobs -- learn the current name and recognize the legacy one.

Exam Tip: The DP-750 exam emphasizes hands-on operational scenarios over rote memorization. You will be asked how to configure secure network topologies, troubleshoot failing Spark jobs, optimize slow Delta Lake queries, and enforce fine-grained Unity Catalog data governance.

---

2. Azure Databricks Lakehouse Architecture: Control Plane vs. Customer Data Plane

Azure Databricks operates on a split-plane architecture that separates control plane management operations from data processing and persistence. Understanding where specific components reside and where your data is processed is foundational to enterprise security and cloud governance.

+-------------------------------------------------------------------------+
|                        MICROSOFT / DATABRICKS MANAGED                   |
|                                CONTROL PLANE                            |
|  +-------------------------------------------------------------------+  |
|  |  Web Application UI  |  Notebooks & Workspaces  |  Job Scheduler  |  |
|  |  Cluster Manager     |  Unity Catalog Metastore |  SCC Relay Host |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
                                     |  ^ (Outbound Port 443 / TLS Only)
                                     v  |
+-------------------------------------------------------------------------+
|                        CUSTOMER AZURE SUBSCRIPTION                      |
|                            CUSTOMER DATA PLANE                          |
|  +-------------------------------------------------------------------+  |
|  | Azure Virtual Network (VNet)                                      |  |
|  |   +--------------------------+     +--------------------------+   |  |
|  |   | Host / Public Subnet     |     | Container / Priv Subnet  |   |  |
|  |   | (Driver & Worker VMs)    | <-> | (Container Runtimes)     |   |  |
|  |   +--------------------------+     +--------------------------+   |  |
|  |                                                                   |  |
|  |   +-------------------------------------------------------------+ |  |
|  |   | Storage Access: ABFSS / TLS to ADLS Gen2 Storage Account    | |  |
|  |   +-------------------------------------------------------------+ |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

The Control Plane

The Control Plane resides in a Microsoft/Databricks-managed Azure subscription. It hosts the backend management software, orchestration engines, and workspace frontend services, including:

• Web Application & UI: The interactive portal used to navigate catalogs, write notebooks, run SQL queries, and manage workspace settings.
• Notebook Storage & Management: Storage for workspace notebook metadata, revision history, and user workspace folders.
• Job Scheduler & Workflow Coordinator: The orchestration service that triggers, monitors, and tracks automated jobs and pipeline DAGs.
• Cluster Manager Coordinator: The control system responsible for requesting VM allocation, issuing configuration commands, and receiving node health heartbeats.
• Unity Catalog Metastore Services: The centralized governance service that validates authorization tokens, evaluates access control lists (ACLs), and audits securable access requests.

The Customer Data Plane

The Customer Data Plane resides entirely within your organization's Azure subscription. All compute clusters execute here, and all business data remains strictly within your security perimeter:

• Virtual Machines (Compute Nodes): Azure Virtual Machine instances provisioned for Spark Driver and Worker nodes (e.g., Standard_D8s_v5, Standard_E16ds_v5).
• Virtual Network (VNet): The Azure VNet containing the subnets where compute instances communicate with each other over internal network interfaces.
• Root DBFS Storage: An Azure Data Lake Storage Gen2 (or General Purpose Blob) account created inside a managed resource group, used for system logs, temporary scratch data, and workspace library files.
• Enterprise Persistent Storage: Your external ADLS Gen2 accounts housing Bronze, Silver, and Gold Delta tables. Cluster worker nodes read and write directly to ADLS Gen2 without passing data through the Control Plane.

Shared Responsibility Model Matrix

---

3. Secure Cluster Connectivity (SCC / No Public IPs / NPIP)

In early cloud architectures, Spark worker VMs required public IP addresses so the Control Plane cluster manager could establish inbound SSH or HTTPS connections. This introduced significant attack surface and required complex perimeter defenses.

Secure Cluster Connectivity (SCC)—also referred to as No Public IPs (NPIP)—is the standard, hardened security architecture for all modern Azure Databricks workspaces.

Architectural Mechanics of SCC

1. Zero Inbound Open Ports: Worker and driver virtual machines in the customer data plane are provisioned without public IP addresses. No inbound network security group (NSG) ports need to be opened to the internet or control plane.
2. Outbound Port 443 Relay: When a cluster starts, the cluster manager daemon on the Driver VM initiates an outbound HTTPS/TLS connection on port 443 to the Databricks Secure Cluster Connectivity Relay hosted in the Control Plane.
3. Bidirectional Communication Tunnel: The SCC relay establishes a secure, persistent reverse-tunnel over this outbound connection. The Control Plane uses this established tunnel to send job commands, stream driver logs, and collect metrics without ever needing inbound routing to the customer VNet.

Customer VNet (Data Plane)                      Databricks Control Plane
+-----------------------+                      +-----------------------+
| Driver / Worker Node  | -- Outbound Port 443 --> |  Secure Cluster       |
| (Private IP Only)     | <-- Reverse Tunnel --- |  Connectivity Relay   |
+-----------------------+                      +-----------------------+

Key Exam Point: Under SCC / No Public IPs, all communication between the customer data plane and the control plane is strictly outbound-initiated over port 443. Workspaces created with SCC enforce zero inbound attack vectors from the public internet.

---

4. Virtual Network (VNet) Injection

By default, creating an Azure Databricks workspace provisions a Managed Resource Group containing a default Azure VNet managed by Microsoft. However, enterprise organizations almost universally require VNet Injection to integrate Databricks compute into their existing corporate network topologies, firewalls, and on-premises ExpressRoute connections.

Subnet Requirements for VNet Injection

When deploying Azure Databricks via VNet Injection, you must supply an existing Azure Virtual Network containing two dedicated subnets:

1. Host Subnet (Public Subnet): Used by the host infrastructure VMs to communicate with the Azure Databricks control plane services.
2. Container Subnet (Private Subnet): Used by the Spark container runtime environment for intra-cluster communication between the driver and worker executors.

Subnet Sizing Rules & CIDR Constraints

• Subnet Delegation: Both subnets must be delegated exclusively to the Microsoft.Databricks/workspaces resource provider. No other non-Databricks Azure resources can share these subnets.
• IP Allocation Formula: Databricks allocates 2 IP addresses per cluster node (one IP from the host subnet and one IP from the container subnet), plus Azure reserves 5 IP addresses per subnet.
• Subnet Size Thresholds:
  • Minimum supported CIDR block: /26 (64 total IPs, allowing a maximum of ~27 cluster nodes).
  • Recommended enterprise CIDR block: /24 (256 IPs, ~120 nodes) to /18 (16,384 IPs, ~8,000 nodes) to prevent cluster provisioning failures during heavy autoscaling or multi-job concurrency.

Network Security Groups (NSGs) & User Defined Routes (UDRs)

Azure Databricks automatically manages the required Network Security Group rules associated with the injected subnets to enable:

• Intra-cluster traffic between the host and container subnets on internal ports.
• Outbound communication to Azure Storage, Azure Key Vault, Entra ID (Azure AD), and the Databricks Control Plane SCC relay.
• If egress traffic is routed through a central Azure Firewall or Network Virtual Appliance (NVA) using User Defined Routes (UDRs), you must configure firewall application/network rules allowing outbound access to required Databricks Control Plane FQDNs, Azure Storage endpoints, and the Metastore service.

---

5. Azure Private Link Integration Patterns

For highly regulated industries (finance, healthcare, government), Azure Databricks supports Azure Private Link to completely eliminate public internet routability across all architectural boundaries.

+-----------------------------------------------------------------------------------+
|                             ENTERPRISE PRIVATE LINK TOPOLOGY                      |
|                                                                                   |
|  [ Corporate User ] ---> ( Front-End Private Endpoint ) ---> [ Control Plane UI ] |
|                                                                        ^          |
|                                                                        |          |
|  [ Customer VNet ]  ---> ( Back-End Private Endpoint )  ---------------+          |
|   (Cluster Nodes)                                                                 |
|          |                                                                        |
|          +-------------> ( Storage Private Endpoint )  ---> [ ADLS Gen2 Storage ] |
+-----------------------------------------------------------------------------------+

1. Front-End Private Link (User-to-Workspace)

• Provisions an Azure Private Endpoint in your corporate hub VNet or on-premises network (via ExpressRoute/VPN).
• Allows data engineers and analysts to access the Databricks Web Application UI, REST APIs, and SQL endpoints using private corporate IP addresses without traversing the public internet.

2. Back-End Private Link (Data Plane-to-Control Plane)

• Provisions Azure Private Endpoints inside your customer VNet connected to the Databricks Control Plane private link service.
• Routes all cluster-to-control plane communication (SCC relay heartbeats, REST API calls, and metastore queries) across the private Microsoft backbone network, bypassing all public internet routing.

3. Azure Storage Private Endpoints (Data Plane-to-Storage)

• Connects your injected VNet directly to the dfs sub-resource of your ADLS Gen2 storage accounts (privatelink.dfs.core.windows.net).
• Enforces that all big data reads and writes from Spark executors resolve to private IP addresses within the VNet, enabling strict firewall lockdown (Deny All Public Access) on ADLS Gen2 accounts.

---

6. Lakehouse Storage Architecture on ADLS Gen2

The Azure Databricks Lakehouse architecture unifies the reliability, governance, and ACID transactions of enterprise data warehouses with the scalability, openness, and low cost of cloud data lakes.

+-------------------------------------------------------------------+
|                   AZURE DATABRICKS LAKEHOUSE STACK                |
+-------------------------------------------------------------------+
|  GOVERNANCE LAYER   | Unity Catalog (Catalogs, Schemas, Volumes)  |
+---------------------+---------------------------------------------+
|  TRANSACTION LAYER  | Delta Lake (_delta_log ACID metadata)       |
+---------------------+---------------------------------------------+
|  STORAGE DRIVER     | Azure Blob File System (ABFSS) Driver       |
+---------------------+---------------------------------------------+
|  PHYSICAL STORAGE   | ADLS Gen2 Object Storage (HNS Enabled)      |
+---------------------+---------------------------------------------+

Azure Data Lake Storage Gen2 (ADLS Gen2) & Hierarchical Namespace (HNS)

ADLS Gen2 combines the massive scale and cost economics of Azure Blob Storage with specialized big data capabilities:

• Hierarchical Namespace (HNS): Organizes objects into a true hierarchical directory structure (directories and subdirectories) rather than the virtual directory paths (flat object namespace) used in standard blob storage.
• Atomic Directory Operations: With HNS enabled, renaming or deleting a directory is a single atomic metadata operation (O(1) complexity). In flat blob storage, renaming a directory requires copying every single object file sequentially (O(N) complexity), which severely degrades Spark write and partition-commit performance.
• POSIX-Compliant Access Control Lists (ACLs): Provides low-level security permissions on files and directories, seamlessly integrated with Microsoft Entra ID.

The ABFSS Driver Protocol

Azure Databricks accesses ADLS Gen2 through the Azure Blob File System Secure (ABFSS) Hadoop-compatible filesystem driver. The standard URI syntax is:

abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<path-to-data>

• abfss://: Specifies secure, TLS-encrypted communication between Spark cluster worker nodes and the ADLS Gen2 DFS endpoint.
• .dfs.core.windows.net: Directs traffic to the high-performance Data Lake File System endpoint (rather than the legacy .blob.core.windows.net endpoint).

Delta Lake: The Transactional Engine on Object Storage

Cloud object storage is inherently eventually consistent and lacks native support for multi-file ACID transactions. Delta Lake solves this by placing an open-source, Parquet-based transactional storage layer on top of ADLS Gen2:

• _delta_log/ Directory: Maintains an immutable, serialized transaction log recording JSON commit files (000000.json, 000001.json) and consolidated Parquet checkpoint files.
• ACID Guarantees: Guarantees serializable or write-serializable isolation across concurrent readers and writers.
• Metadata Caching & Indexing: Min/max statistics, file sizes, and Liquid Clustering indexes are stored in the transaction log, allowing Spark to perform file skipping without costly directory listings.