10.1 AI/BI Genie Architecture & Space Setup

10.1 AI/BI Genie Architecture & Space Setup

Databricks AI/BI Genie represents a paradigm shift in enterprise business intelligence, replacing static dashboards with an intelligent, conversational interface. Built upon the principles of Compound AI Systems, Genie combines state-of-the-art Large Language Models (LLMs), deep integration with Unity Catalog governance, and the high-performance Databricks SQL (DBSQL) engine. Rather than relying on a single monolithic model to answer questions, Genie orchestrates specialized components to parse business intent, inspect governed data schemas, generate optimized SQL, and execute queries securely on serverless compute.

Architectural Overview & Query Execution Pipeline

Understanding how AI/BI Genie processes natural language questions is essential for data analysts designing enterprise spaces. When an end user submits a prompt like "What were our top 5 revenue-generating products in Q1 2026?", Genie routes the request through a multi-stage execution pipeline designed for security, accuracy, and performance.

The pipeline ensures that the LLM never accesses raw data directly during the translation phase; it operates exclusively on metadata, column names, comments, and predefined space context. Data retrieval occurs strictly within the governed Databricks SQL engine during Stage 4.

Designing and Provisioning a Genie Space

A Genie Space is a curated workspace object that exposes specific tables, views, and semantic context to business users. Creating an effective space requires deliberate planning regarding dataset boundaries and target user personas.

To provision a new Genie Space in the Databricks Workspace interface:

1. Navigate to the AI/BI Genie section in the left navigation sidebar and click New Space.
2. Provide a clear Space Name and Description that articulate the business domain (e.g., Executive Sales & Revenue Analytics).
3. Select an existing SQL Warehouse (Serverless SQL Warehouses are recommended due to instant startup times and automatic scaling).
4. Add governed Unity Catalog tables and views that contain the necessary analytical data.
5. Save the space draft and initiate initial curation by adding space instructions.

-- Example: Creating a gold-layer view optimized for Genie Space inclusion
CREATE OR REPLACE VIEW main.finance_gold.v_monthly_revenue_summary AS
SELECT 
    c.customer_id,
    c.customer_segment,
    DATE_TRUNC('month', o.order_date) AS order_month,
    SUM(o.net_amount) AS total_revenue,
    COUNT(DISTINCT o.order_id) AS total_orders
FROM main.sales_silver.orders o
JOIN main.sales_silver.customers c ON o.customer_id = c.customer_id
WHERE o.order_status = 'COMPLETED'
GROUP BY 1, 2, 3;

COMMENT ON VIEW main.finance_gold.v_monthly_revenue_summary 
IS 'Curated gold-layer view containing completed monthly revenue and order counts segmented by customer.';

Security, Privileges, and Warehouse Access Model

AI/BI Genie operates under a strict dual-layer security model that combines workspace object permissions with Unity Catalog data governance policies. End users cannot bypass data access controls through conversational prompts.

Workspace Access Control List (ACL) Permissions

Space creators manage user capabilities using three workspace privilege levels:

• CAN VIEW: Allows users to interact with the space, ask natural language questions, view generated charts, and export results. Users cannot edit space instructions or table configurations.
• CAN EDIT: Enables data analysts to add/remove tables, update space instructions, curate sample questions, attach trusted assets, and evaluate benchmarks.
• CAN MANAGE: Grants full administrative control, including deleting the space, modifying workspace permissions, and altering warehouse bindings.

Unity Catalog Privilege Enforcement

When a user asks a question in a Genie space, query execution inherits that specific user's identity (or a defined service principal context). To successfully execute generated queries, the user's identity must possess:

• USE CATALOG on the parent catalog.
• USE SCHEMA on the parent schema.
• SELECT on all queried tables and views.
• CAN USE permission on the assigned Databricks SQL Warehouse.

If a user lacks SELECT privileges on a specific column or table, Unity Catalog blocks query execution at runtime, returning an authorization error without exposing unauthorized data.

-- Granting required Unity Catalog privileges for Genie Space users
GRANT USE CATALOG ON CATALOG main TO ROLE sales_analysts;
GRANT USE SCHEMA ON SCHEMA main.finance_gold TO ROLE sales_analysts;
GRANT SELECT ON VIEW main.finance_gold.v_monthly_revenue_summary TO ROLE sales_analysts;
GRANT CAN USE ON SQL WAREHOUSE `prod-serverless-warehouse` TO ROLE sales_analysts;

Distributing and Embedding Genie Spaces

Creating a Genie space is only half of the exam objective — you must also share and distribute it.

Analyst checklist before distributing:

1. Curate only the Unity Catalog datasets the audience should see; Genie answers are bounded by both space curation and the runner’s data permissions.
2. Attach a right-sized SQL Warehouse so embedded/shared usage remains responsive.
3. Verify trusted assets and instructions so external consumers do not receive brittle SQL.
4. Prefer least-privilege ACLs: start with CAN VIEW/CAN RUN for consumers; reserve CAN MANAGE for owners.

Table Selection & Schema Curation Best Practices

Including raw, staging, or uncleaned tables in a Genie space degrades SQL generation quality. Analysts should follow structured curation rules during space setup:

1. Expose Gold-Layer Tables Only: Limit space objects to clean, aggregated, gold-layer star schema tables or consolidated analytical views. Exclude bronze (raw ingestion) and silver (cleaning/staging) tables.
2. Minimize Table Redundancy: Avoid adding multiple tables containing duplicate or overlapping metrics. If both orders and order_line_items are included, explicitly document their granular relationship.
3. Declare Primary and Foreign Keys: Unity Catalog primary and foreign key constraints serve as critical metadata signals. Genie utilizes declared relationships to construct accurate JOIN conditions automatically.
4. Prune Unnecessary Columns: Exclude technical surrogate keys, internal ETL timestamps, and raw payload strings to reduce context overhead and prevent model confusion.