13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics

13.4 Workspace Monitoring with Azure Monitor, Diagnostic Settings, & Log Analytics

Enterprise lakehouse governance demands end-to-end operational visibility. Data engineers and platform administrators must monitor workspace health, enforce security compliance, audit data access, track Databricks Unit (DBU) consumption, and establish automated alerting for pipeline failures.

Azure Databricks integrates natively with Azure Monitor via Diagnostic Settings, allowing organizations to stream rich audit logs and telemetry to Azure Log Analytics, Azure Storage, and SIEM platforms.

---

1. Diagnostic Settings Architecture & Export Destinations

Diagnostic Settings capture control-plane audit events and system operations, routing them from the Azure Databricks resource to one or more enterprise destinations:

+-------------------------------------------------------------------------+
|               AZURE DATABRICKS DIAGNOSTIC TELEMETRY PIPELINE            |
+-------------------------------------------------------------------------+
|                                                                         |
|  AZURE DATABRICKS CONTROL PLANE                                         |
|  ├── Cluster Events  ├── Notebook Executions  ├── Unity Catalog Grants  |
|  ├── Job Triggers    ├── Account Logins       ├── DBFS Operations       |
|  └───────────────────────────────┬──────────────────────────────────────┘|
|                                  │ Azure Diagnostic Settings             |
|                                  ▼                                       |
|  ┌───────────────────────────────┼──────────────────────────────────┐   |
|  │                               │                                  │   |
|  ▼                               ▼                                  ▼   |
|  LOG ANALYTICS WORKSPACE         ADLS GEN2 STORAGE ACCOUNT          EVENT HUBS |
|  - Real-time KQL Queries         - Long-term Compliance (7+ yrs)    - SIEM Tool|
|  - Azure Monitor Alerting        - Cold Data Lake Archival          - Sentinel |
|  - Azure Workbooks               - Cost-effective Storage           - Splunk   |
+-------------------------------------------------------------------------+

Comparison of Export Destinations

---

2. Core Diagnostic Log Categories

When configuring Diagnostic Settings on an Azure Databricks workspace, administrators select which log categories to stream:

---

3. Auditing & Troubleshooting with Kusto Query Language (KQL)

Once logs are ingested into an Azure Log Analytics workspace, data engineers use Kusto Query Language (KQL) to query the DatabricksAccounts, DatabricksClusters, DatabricksJobs, DatabricksNotebook, and DatabricksUnityCatalog tables.

KQL Query 1: Detecting Unauthorized Unity Catalog Privilege Grants

Find all security GRANT events granting ALL PRIVILEGES or MANAGE permissions to non-admin users in the last 7 days:

DatabricksUnityCatalog
| where TimeGenerated >= ago(7d)
| where ActionName in ("grantPermissions", "updatePermissions")
| extend Request = parse_json(RequestParams)
| extend TargetObject = tostring(Request.securable_type),
         TargetName = tostring(Request.securable_name),
         Changes = tostring(Request.changes)
| where Changes has "ALL_PRIVILEGES" or Changes has "MANAGE"
| project TimeGenerated, Identity, Source_IP_Address, ActionName, TargetObject, TargetName, Changes
| order by TimeGenerated desc

KQL Query 2: Tracking Production Job Failures & Error Messages

Identify failed Lakeflow Job runs over the past 24 hours to monitor pipeline SLA compliance:

DatabricksJobs
| where TimeGenerated >= ago(24h)
| where ActionName == "runFailed"
| extend Response = parse_json(Response)
| extend JobId = tostring(Response.job_id),
         RunId = tostring(Response.run_id),
         ErrorMessage = tostring(Response.error)
| project TimeGenerated, Identity, JobId, RunId, ErrorMessage
| summarize FailureCount = count() by JobId, ErrorMessage
| order by FailureCount desc

KQL Query 3: Auditing High-Risk Notebook Data Exports

Detect instances where users exported or downloaded large notebook results to local workstations:

DatabricksNotebook
| where TimeGenerated >= ago(30d)
| where ActionName in ("downloadNotebookExport", "exportNotebook")
| project TimeGenerated, Identity, Source_IP_Address, ActionName, RequestParams
| order by TimeGenerated desc

---

4. Azure Monitor Metric Alerts & Action Groups

Azure Monitor collects platform-level performance metrics from the Azure Databricks resource and underlying infrastructure. Data engineers configure Metric Alert Rules to automatically trigger when operational thresholds are breached.

+-------------------------------------------------------------------------+
|                    AZURE MONITOR ALERTING PIPELINE                      |
+-------------------------------------------------------------------------+
|                                                                         |
|  1. TELEMETRY SOURCE       2. ALERT CONDITION       3. ACTION GROUP     |
|  +--------------------+    +--------------------+   +-----------------+ |
|  | Databricks Metrics |    | Condition:         |   | Action Group:   | |
|  | - CPU Utilization  | -> | Cluster CPU > 90%  |-> | - Email DataOps | |
|  | - Memory Pressure  |    | for 15 minutes     |   | - PagerDuty     | |
|  | - Failed Job Runs  |    | OR Job Failure > 0 |   | - Teams Webhook | |
|  +--------------------+    +--------------------+   | - Logic App     | |
|                                                     +-----------------+ |
+-------------------------------------------------------------------------+

Configuring an Alert Rule:

1. Scope: Target the Azure Databricks workspace resource.
2. Condition: Define signal logic (e.g., Signal: Cluster CPU Utilization, Aggregation: Average, Operator: Greater than, Threshold: 90%, Evaluation Period: 15 minutes).
3. Actions: Attach an Azure Action Group configured with:
   • Email / SMS / Push / Voice notifications to on-call data engineers.
   • Webhook: Dispatches JSON payloads to Microsoft Teams channels or Slack.
   • Azure Logic App / Azure Function: Automatically triggers self-healing workflows (such as restarting a stalled cluster or scaling compute).

---

5. Azure Monitor vs. Unity Catalog System Tables

Modern Azure Databricks environments provide two complementary observability pillars: Azure Monitor (Log Analytics) and Unity Catalog System Tables.

+-------------------------------------------------------------------------+
|            AZURE MONITOR VS. UNITY CATALOG SYSTEM TABLES                |
+-------------------------------------------------------------------------+
|  FEATURE            | AZURE MONITOR / LOG ANALYTICS | SYSTEM TABLES     |
|---------------------|-------------------------------|-------------------|
|  Query Engine       | Kusto Query Language (KQL)    | ANSI SQL (Spark)  |
|  Query Interface    | Azure Portal Log Analytics    | Databricks SQL    |
|  Data Location      | Log Analytics Workspace (Azure)| Unity Catalog     |
|  Primary Persona    | Azure Admins, SecOps, DevOps  | Data Engineers, BI|
|  Key Schemas/Tables | DatabricksClusters,           | system.access.audit
|                     | DatabricksJobs,               | system.billing.usage
|                     | DatabricksUnityCatalog        | system.compute.*  |
|  Best For           | Real-time infrastructure      | Lakehouse lineage |
|                     | alerts, SIEM integration      | joins, DBU billing|
+-------------------------------------------------------------------------+

Exam Tip: Use Azure Monitor / Log Analytics when integrating with enterprise-wide Azure monitoring, SIEMs (Microsoft Sentinel), and infrastructure alerting. Use Unity Catalog System Tables (system.access.audit, system.billing.usage) when data engineers and analysts need to join audit and cost telemetry directly with Delta tables in SQL dashboards.