1.1 Databricks Architecture & Compute Concepts

Databricks Architecture & Compute Concepts

The Databricks Data Intelligence Platform is built upon a lakehouse architecture that combines the performance, structure, and governance of traditional data warehouses with the flexibility, open formats, and cost efficiency of cloud object storage. To effectively analyze data and build high-performance reporting solutions, a Databricks Data Analyst must understand how compute resources operate, how workloads are processed across different architectural tiers, and how to select the right compute engine for specific analytical tasks.

---

The Two-Tiered Architecture: Control Plane vs. Data Plane

Databricks operates using a fundamental separation of concerns divided into two primary planes: the Control Plane and the Data Plane. This decoupled structure ensures high operational reliability, centralized administration, and robust data security.

1. Control Plane

The Control Plane consists of backend services managed entirely by Databricks within its own cloud account infrastructure. It houses the services responsible for platform management and orchestration, including:

• Web UI & Workspace Application: The web interface for notebooks, SQL editors, dashboards, and Catalog Explorer.
• Notebook & Job Management: Storage and versioning of notebook code, job schedules, and workflow definitions.
• Query Orchestration & Parsing: Initial query receiving, user authentication, authorization checks, and command routing.
• Cluster Management Service: Launching, monitoring, and terminating compute clusters in the Data Plane.
• Control Plane Security: Management of user tokens, single sign-on (SSO) integrations, and audit logging.

Key Concept: Customer data at rest never resides in the Control Plane. The Control Plane only stores control metadata (such as notebook source code, workspace configurations, and user identity credentials).

2. Data Plane

The Data Plane is where your actual data resides and where compute engines execute code. Depending on your deployment architecture, the Data Plane operates in one of two environments:

• Classic Data Plane: Compute resources (virtual machines) are launched inside the customer's cloud account (AWS VPC, Azure VNet, or GCP VPC). Storage services such as AWS S3, Azure ADLS Gen2, or Google Cloud Storage remain in the customer account.
• Serverless Data Plane: Compute resources run in secure, isolated cloud accounts managed directly by Databricks. Databricks handles instance provisioning, scaling, networking, and software updates while connecting securely to the customer's data storage.

---

Classic Compute vs. Serverless Compute Models

Databricks provides two distinct compute provisioning models: Classic Compute and Serverless Compute.

Classic Compute

In the classic model, when a user provisions a cluster or SQL warehouse, Databricks requests virtual machines from the cloud provider within the customer's cloud subscription. The user selects specific VM node types (e.g., memory-optimized or compute-optimized), configures minimum and maximum node limits for auto-scaling, and sets auto-stop thresholds.

• Pros: Full network isolation within customer VPCs; custom cloud security configurations.
• Cons: Cold-start latency (typically 5 to 7 minutes while cloud instances initialize); potential cloud quota limits; manual cluster management overhead.

Serverless Compute

In the serverless model, compute resources run in a pool of pre-warmed infrastructure managed by Databricks. When a data analyst executes a query in Databricks SQL or runs a notebook attached to a serverless compute engine, compute capacity is assigned instantly.

• Pros: Ultra-fast start time (typically 2–6 seconds); zero infrastructure management; dynamic auto-scaling tailored precisely to query complexity; automatic patching and optimization.
• Cons: Requires cloud network configuration setup for private storage connectivity (e.g., Network Connectivity Configurations / NCC).

---

Compute Types & Workload Suitability

Databricks organizes compute resources into three primary types based on user personas and workload profiles:

1. All-Purpose Clusters

All-Purpose Clusters are interactive compute resources designed for collaborative development. Multiple data analysts and data scientists can attach interactive notebooks to the same shared cluster.

• Supported Languages: Python, SQL, R, Scala.
• Typical Use Cases: Exploratory data analysis, ad-hoc data transformations, machine learning prototyping, feature engineering.
• Lifecycle: Manually started by users or admins, auto-terminated after a configurable period of inactivity (e.g., 30 minutes).

2. Job Clusters

Job Clusters (also known as Automated Clusters) are created specifically to execute scheduled automated workflows and data pipelines.

• Lifecycle: The Databricks Workflow service provisions the job cluster automatically when a scheduled job starts, executes the assigned tasks, and immediately terminates the cluster upon job completion.
• Cost Efficiency: Job clusters are billed at a significantly lower Databricks Unit (DBU) rate compared to interactive All-Purpose Clusters.

3. SQL Warehouses

SQL Warehouses (formerly SQL Endpoints) are compute resources specifically optimized for SQL queries, BI dashboards, ad-hoc analytical queries, and external BI tool connections (such as Power BI, Tableau, and ThoughtSpot).

---

SQL Warehouse Architecture & Sizing

Databricks SQL Warehouses are available in three channels/types: Classic, Pro, and Serverless.

+-----------------------------------------------------------------------+
|                         DATABRICKS SQL WAREHOUSE                      |
|                                                                       |
|   +---------------------------------------------------------------+   |
|   |                     Photon Engine Acceleration                |   |
|   +---------------------------------------------------------------+   |
|   |  Cluster Instance Sizing (2X-Small -> 4X-Large T-Shirt Sizes) |   |
|   +---------------------------------------------------------------+   |
|   |  Multi-Cluster Auto-Scaling (Scaling out for Concurrency)    |   |
|   +---------------------------------------------------------------+   |
+-----------------------------------------------------------------------+

T-Shirt Sizing

SQL Warehouses use T-shirt sizing to simplify resource allocation:

Concurrency Scaling & Multi-Cluster Warehouses

When multiple analysts or BI dashboards query a SQL Warehouse simultaneously, query queueing can occur. To handle concurrent execution without performance degradation, SQL Warehouses support multi-cluster auto-scaling:

• Min Clusters: The minimum number of cluster instances running continuously (e.g., 1).
• Max Clusters: The maximum number of cluster instances Databricks can launch to handle peak concurrency (e.g., 10).
• Scaling Behavior: As query volume increases, Databricks automatically launches additional cluster instances of the selected T-shirt size. When concurrency drops, idle clusters are automatically spun down after 15 minutes.

---

Real-World Scenario: Designing Compute for an Analytics Team

Consider a retail organization with two distinct workload demands:

1. Morning Executive Dashboard Peak: 150 business analysts open interactive sales dashboards between 8:00 AM and 9:00 AM daily.
2. Overnight Data Mart Refresh: A scheduled workflow executes at 2:00 AM to process millions of transactions and update analytical summary tables.

Optimal Solution:

• For the Executive Dashboard Peak, configure a Serverless SQL Warehouse with T-shirt size Medium and multi-cluster auto-scaling set to Min: 1 and Max: 6. Serverless compute ensures instantaneous startup at 8:00 AM without keeping idle servers running overnight.
• For the Overnight Data Mart Refresh, configure a scheduled Databricks Workflow running on a Job Cluster (or Serverless Job Compute). This ensures lowest DBU billing costs and automatic cluster destruction immediately upon completion.