1.2 Compute Types & Workload Matching

1.2 Compute Types & Workload Matching

In Azure Databricks, compute is completely decoupled from storage. Selecting the correct compute type for a given workload directly determines your system's performance, query latency, fault tolerance, and cloud operating costs.

The DP-750 exam thoroughly tests your ability to match specific business workloads—such as interactive exploratory data analysis (EDA), automated batch ETL pipelines, 24/7 streaming ingestion, and high-concurrency BI dashboarding—to the most cost-effective and performant compute architecture.

---

1. Overview of Azure Databricks Compute Categories

Azure Databricks provides three primary compute categories, each tailored to distinct execution lifecycles, concurrency patterns, and cost models:

+--------------------------------------------------------------------------------+
|                      AZURE DATABRICKS COMPUTE CATEGORIES                       |
+--------------------------------------------------------------------------------+
|                                                                                |
|  1. ALL-PURPOSE COMPUTE                                                        |
|     - Interactive notebook development, exploratory data analysis, debugging   |
|     - Long-running or auto-terminating lifecycle                               |
|     - Premium DBU pricing tier                                                 |
|                                                                                |
|  2. JOB COMPUTE (AUTOMATED WORKFLOWS)                                          |
|     - Scheduled ETL/ELT pipelines, Lakeflow Jobs, automated tasks              |
|     - Ephemeral lifecycle: spins up on trigger, terminates immediately on finish|
|     - Lowest DBU pricing tier (significant cost savings)                       |
|                                                                                |
|  3. SQL WAREHOUSES                                                             |
|     - High-concurrency SQL queries, Power BI / Tableau dashboards, ANSI SQL    |
|     - Photon-vectorized C++ engine with Predictive I/O                         |
|     - Classic, Pro, and Serverless tiers with multi-cluster horizontal scaling |
+--------------------------------------------------------------------------------+

---

2. All-Purpose Compute (Interactive Development)

All-Purpose Compute clusters are designed for interactive, ad-hoc data analysis, notebook collaboration, and code debugging.

Operational Characteristics

• Lifecycle: Manually provisioned by data engineers via the UI, REST API, or Terraform. Stays running continuously until manually terminated or stopped automatically by the Auto-Termination idle timeout.
• Multi-User Collaboration: Multiple users can attach their individual interactive notebooks to the same running cluster simultaneously (when configured in Shared access mode).
• Language Flexibility: Fully supports polyglot development, allowing engineers to mix Python (%python), SQL (%sql), Scala (%scala), and R (%r) within the same workspace environment.
• State Persistence: In-memory DataFrames, Spark broadcast variables, and temporary views persist across notebook cell executions until the cluster restarts or the session detached.

Cost & Governance Considerations

• Premium Billing Tier: All-Purpose compute incurs the highest DBU consumption rate per node-hour because it provides interactive shared development services.
• Waste Mitigation: To prevent runaway cloud costs, workspace administrators should enforce Cluster Policies that mandate auto-termination (e.g., stopping after 15–20 minutes of inactivity) and cap maximum worker node counts.

---

3. Job Compute (Automated Batch & Production Pipelines)

Job Compute clusters (often called Automated or Ephemeral Job clusters) are purpose-built for executing scheduled, production-grade data pipelines orchestrated by Lakeflow Jobs.

Operational Mechanics

1. Trigger & Dynamic Provisioning: When a scheduled or event-driven Lakeflow Job triggers, the Databricks cluster manager automatically requests the specified VM instances from Azure and boots the cluster runtime.
2. Isolated Task Execution: The job executes its transformation tasks (notebooks, Python scripts, JARs, SQL files, or DAB tasks) in a pristine, dedicated runtime environment.
3. Automatic Immediate Teardown: As soon as the final task succeeds or fails, the cluster manager immediately terminates the VMs and deallocates Azure resources. No idle compute hours are ever billed.

Enterprise Benefits

• Lowest DBU Pricing: Job compute is billed at the lowest DBU rate—frequently 40% to 50% cheaper than running the identical workload on All-Purpose compute.
• Total Fault & Dependency Isolation: Because each job runs on its own isolated cluster, memory leaks, rogue threads, or conflicting Python library installations from one job cannot crash or destabilize other production pipelines.
• Reproducibility: Eliminates state contamination from residual in-memory variables or lingering cached tables.

Exam Trap: Never use an All-Purpose cluster to run recurring production batch jobs. The DP-750 exam will penalize answers that attach scheduled workflows to running All-Purpose clusters instead of using ephemeral Job compute.

---

4. SQL Warehouses (BI & Analytical Dashboards)

SQL Warehouses are compute resources specialized exclusively for ANSI SQL execution, BI dashboard acceleration (e.g., Power BI DirectQuery, Tableau), and ad-hoc analytical queries.

                          SQL WAREHOUSE ARCHITECTURE
                       
  [ BI Tools / Analysts ] ---> [ Load Balancer / Query Router ]
                                      |
         +----------------------------+----------------------------+
         | (Low Concurrency)                                       | (Query Spike Surge)
         v                                                         v
  +---------------+                                         +---------------+---------------+ 
  |   Cluster 1   |                                         |   Cluster 1   |   Cluster 2   | 
  | (Photon Eng.) |                                         | (Photon Eng.) | (Photon Eng.) | 
  +---------------+                                         +---------------+---------------+ 

Architectural Features of SQL Warehouses

• Photon Vectorized Engine: Written natively in C++ to leverage hardware-level SIMD (Single Instruction, Multiple Data) parallelism, vectorized query processing, and predictive caching.
• Multi-Cluster Auto-Scaling & Load Balancing: When multiple analysts refresh dashboards simultaneously, the SQL Warehouse automatically spins up secondary compute clusters to distribute concurrent query queues without manual intervention, scaling back down when load subsides.
• ANSI SQL Compliance: Enforces standard ANSI SQL syntax, making it ideal for business analysts and SQL data modelers.

SQL Warehouse Tiers: Classic vs. Pro vs. Serverless

---

5. DBU Economics & Total Cost of Ownership (TCO)

Every Azure Databricks deployment incurs two distinct billing components:

$\text{Total Cost} = \text{Azure Infrastructure Cost (VMs, Disks, Egress)} + \text{Databricks DBU Cost}$

1. Azure Infrastructure Cost: The underlying compute hardware billed directly by Microsoft per VM core/hour (e.g., 8 vCPUs × $0.40/hr).
2. Databricks Unit (DBU) Consumption: A standardized unit of processing capability billed per second of active compute runtime based on the workload tier and VM SKU.

DBU Pricing Hierarchy (Relative Cost per Compute-Hour)

$\text{Automated Job Compute} < \text{Classic SQL} < \text{Pro SQL} < \text{Serverless SQL} \approx \text{All-Purpose Compute}$

The Serverless Billing Model

In Serverless Compute (Serverless SQL Warehouses, Serverless Notebooks, Serverless Jobs), Microsoft infrastructure VM costs and Databricks software licenses are consolidated into a single unified DBU meter. Key economic advantages include:

• Zero Idle Tax: Compute shuts down within 1–5 minutes of inactivity instead of staying warm for 15–30 minutes, drastically cutting idle spend.
• Per-Second Billing: You are billed strictly for active query and task processing seconds without paying for the 3–7 minute VM provisioning boot lag.

---

6. Latency Profiles & Startup Characteristics

Understanding cold-start versus warm-start latency is essential for architectural sizing and passing scenario-based exam questions.

---

7. Workload Matching Decision Matrix

Use the following authoritative decision matrix to match customer workload requirements to the correct compute configuration: