10.4 Monitoring, Benchmarks, Feedback & Maintenance

10.4 Monitoring, Benchmarks, Feedback & Maintenance

Deploying an AI/BI Genie space is not a one-time event; it is an ongoing lifecycle that requires continuous monitoring, evaluation, and refinement. As business definitions evolve, new datasets are ingested, and user query patterns shift, data analysts must actively maintain Genie spaces to preserve answer quality, detect regression, and optimize warehouse compute costs. This section details the tools and procedures required for operational excellence, including Genie Benchmarks, User Feedback Loops, and System Table Telemetry.

The AI/BI Genie Maintenance Lifecycle

A healthy Genie space relies on a structured, four-stage feedback and maintenance cycle:

               +-------------------------------------------------+
               |  1. Deploy & Monitor                            |
               |  - Track query logs via System Tables           |
               |  - Monitor SQL Warehouse usage and costs        |
               +-------------------------------------------------+
                                       |
                                       v
               +-------------------------------------------------+
               |  2. Collect Feedback                            |
               |  - Review Thumbs Up / Thumbs Down ratings       |
               |  - Analyze user-edited SQL queries & comments   |
               +-------------------------------------------------+
                                       |
                                       v
               +-------------------------------------------------+
               |  3. Refine & Benchmark                          |
               |  - Update Instructions, Descriptions & Assets  |
               |  - Execute automated Benchmark Evaluation suites|
               +-------------------------------------------------+
                                       |
                                       v
               +-------------------------------------------------+
               |  4. Publish & Iterate                           |
               |  - Release space updates to end users           |
               |  - Verify resolution of unhandled questions     |
               +-------------------------------------------------+

User Feedback Collection & Query Audit Logs

Genie incorporates native feedback tools directly into the conversational chat interface:

Micro-Feedback Signals

• Thumbs Up / Thumbs Down: Users click feedback icons on response cards. A thumbs-down rating alerts space editors that an answer was incomplete, incorrect, or poorly formatted.
• User SQL Edits: Business users or analysts with edit privileges can inspect generated SQL and modify it directly in the chat interface. When a user manually corrects generated SQL, Genie records the edit as a high-value signal for space refinement.

Diagnostic Analysis of Unhandled Queries

Space managers should regularly inspect the History Log tab within Genie Space Management to review unhandled or failed queries. Common failure patterns include:

1. Missing Business Vocabulary: User asked for a metric using jargon not documented in Space Instructions.
2. Missing Schema Join Path: Genie attempted to join two tables that lacked declared primary/foreign keys or explicit relationship instructions.
3. Complex Aggregation Failure: The query required advanced analytics (e.g., cohort retention) better suited for a Trusted Asset.

Building and Running Genie Benchmarks

To prevent accuracy regression when editing space instructions or adding tables, analysts construct Genie Benchmarks. A benchmark is an automated test suite consisting of pairs of natural language questions and verified Gold-Standard SQL statements.

Benchmark Evaluation Workflow

1. Define Test Suite: Assemble 20 to 50 representative natural language questions covering core business metrics.
2. Attach Gold-Standard SQL: For each question, author and verify the exact ANSI SQL query expected from the system.
3. Run Evaluation: Execute the benchmark suite against the Genie space. Genie generates SQL for each question and compares the output tables against the gold-standard results.
4. Review Accuracy Score: The evaluation returns an overall accuracy percentage and highlights specific queries where generated SQL diverged from the gold standard.

-- Benchmark Test Case Example
-- Question: "What is our average order value (AOV) for Q1 2026?"
-- Gold-Standard SQL:
SELECT 
    AVG(net_amount) AS average_order_value
FROM main.sales_gold.fact_orders
WHERE order_date >= '2026-01-01' AND order_date < '2026-04-01'
  AND order_status = 'COMPLETED';

Running benchmarks before publishing space changes ensures that modifying an instruction to fix one query does not inadvertently break three others.

Monitoring Space Usage with System Tables

Databricks captures full operational telemetry for Genie interactions in Unity Catalog System Tables. Analysts can query system.access.audit and operational views to monitor usage trends, active users, and execution performance.

-- Querying system audit logs to monitor AI/BI Genie space interactions
SELECT 
    event_time,
    user_identity.email AS user_email,
    request_params.space_id AS genie_space_id,
    action_name,
    service_name,
    response.status_code
FROM system.access.audit
WHERE service_name = 'genie'
  AND event_date >= CURRENT_DATE() - INTERVAL 30 DAYS
ORDER BY event_time DESC;

Key Metrics to Track via System Tables

• Active Monthly Users (MAU): Volume of distinct business users interacting with the space.
• Query Latency: Execution duration on the underlying SQL Warehouse.
• Cost Allocation: Compute DBUs consumed by Genie queries on assigned SQL Warehouses.

Refreshing Unity Catalog Metadata After Schema Changes

When underlying Unity Catalog tables gain columns, updated comments, renamed fields, or revised descriptions, Genie’s understanding can drift until metadata is refreshed. The exam guide calls out refreshing Unity Catalog metadata as part of maintaining Genie spaces.

Why refresh matters

• Genie relies on table/column names, comments, synonyms, and curated instructions to generate SQL.
• Stale metadata produces wrong joins, missing filters, or hallucinated columns after upstream schema evolution.
• Certification and tagging changes on source tables also affect which assets should remain in the space.

Maintenance practice

1. After upstream schema or comment changes, open the Genie space dataset configuration and refresh metadata so Genie re-reads Unity Catalog definitions.
2. Re-run benchmarks and spot-check popular questions after the refresh.
3. Update instructions, sample questions, and trusted assets when business definitions changed — not only when physical columns changed.
4. Remove deprecated tables (for example, system.certification_status=deprecated) from the space curation list.

UC schema/comment change → Refresh Genie metadata → Retune instructions/trusted assets → Re-run benchmarks → Monitor feedback

Iterative Curation & Continuous Improvement Workflow

Space maintainers should establish a weekly maintenance schedule to process feedback and publish space updates: