6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions

6.5 Data Discovery: Table and Column Descriptions, and AI/BI Genie Instructions

DP-750 Exam Focus: Two blueprint bullets converge here - "Create, implement, and preserve table and column definitions and descriptions for data discovery" from skill area 2, and "Configure AI/BI Genie instructions for data discovery" from skill area 1. They are the same problem viewed twice: a human searching Catalog Explorer and a natural-language agent generating SQL both depend on the metadata you write.

---

1. Descriptions Are Governance Metadata, Not Documentation

A table with no description is invisible to search, ambiguous in lineage graphs, and unusable by Genie. Unity Catalog exposes descriptions at three levels:

-- Catalog and schema level
COMMENT ON CATALOG prod_retail IS 'Production retail lakehouse. Owner: retail-data-platform.';
COMMENT ON SCHEMA  prod_retail.gold IS 'Curated, business-ready aggregates. SLA: 06:00 UTC daily.';

-- Table level: state the grain, the SLA, and the caveats
COMMENT ON TABLE prod_retail.gold.store_sales_daily IS
  'Grain: one row per store_id per business_date. Revenue is net of returns.
   Restated for 7 days after business_date to absorb late-arriving sales.';

-- Column level
ALTER TABLE prod_retail.gold.store_sales_daily
  ALTER COLUMN net_revenue_usd COMMENT 'Gross sales minus returns and discounts, USD, DECIMAL(18,2).';

ALTER TABLE prod_retail.gold.store_sales_daily
  ALTER COLUMN business_date COMMENT 'Store local calendar date. Not the UTC ingest date.';

-- Descriptions can also be set inline at creation time
CREATE TABLE prod_retail.silver.customers (
  customer_id   BIGINT   COMMENT 'Surrogate key. Stable across source system migrations.',
  email_address STRING   COMMENT 'Lowercased. Masked for non-privileged readers.',
  created_at    TIMESTAMP COMMENT 'UTC. First appearance in the CRM source.'
) COMMENT 'Grain: one row per current customer. SCD Type 1 overwrite.';

What a Good Description Contains

Avoid descriptions that restate the column name (customer_id -> "the customer id"). They consume the reader's attention and teach Genie nothing.

Preserving Descriptions Across Rebuilds

The word preserve in the blueprint bullet matters. A CREATE OR REPLACE TABLE rewrites the table definition and drops comments that were applied afterwards with ALTER. Two defensive patterns:

1. Declare comments in the DDL itself (as in the CREATE TABLE above) so they are part of the table definition that the pipeline recreates.
2. Declare them in Lakeflow Spark Declarative Pipelines with the comment argument on the table decorator or the COMMENT clause in SQL, so the pipeline is the source of truth and every full refresh restores them.

Tags Complement, but Do Not Replace, Descriptions

Section 6.3 covered tag governance and Section 5.2 covered ABAC. The division of labor:

• Descriptions are free text for humans and for Genie. They are not enforceable.
• Tags are structured key-value pairs that ABAC policies, row filters, and column masks can act on programmatically.

ALTER TABLE prod_retail.silver.customers
  SET TAGS ('data_domain' = 'customer', 'contains_pii' = 'true', 'certified' = 'gold');

Catalog Explorer surfaces both: descriptions power keyword search and the object detail pane, tags power faceted filtering and drive ABAC enforcement.

---

2. AI/BI Genie: What It Reads

An AI/BI Genie Agent (formerly called a Genie Space) is a natural-language interface over a curated set of Unity Catalog tables. It converts a business question into SQL, runs it, and returns the result. Its accuracy is a direct function of the context you give it:

---

3. Configuring Genie Instructions

General Instructions

A single plain-text block giving global context: what the business does, what an "active customer" means, which fiscal calendar applies. Databricks guidance is to use them sparingly and to prefer more precise mechanisms:

• Write specific instructions. "When users ask about sales metrics without specifying product name or sales channel, ask them to specify both" beats "ask clarifying questions about sales."
• Describe the business narrative - entities, lifecycles, relationships - rather than dictating SQL behavior. Do not force table selection, hardcode filters, or specify output formatting here.
• Never use general instructions to paper over missing table descriptions, join definitions, or example queries.
• Keep them consistent with every other instruction type. If the text says round to two decimals, the example queries must round to two decimals too.

Example SQL Queries

The highest-value instruction type. Provide complete, correct queries for prompts that are hard to interpret, multi-part, or involve intricate joins - "break down my team's performance", "for recently joined customers, which products perform best". Genie matches a user prompt to a relevant example and learns the pattern for related questions. You can attach usage guidance explaining when a given example is especially relevant.

SQL Functions and Trusted Assets

Trusted assets are the subset of instructions whose logic has been verified by an author, so that when Genie answers from one the response is presented as a verified answer:

• Parameterized example SQL queries - when the exact text of a parameterized query generates the response, users can edit the parameter value and rerun.
• SQL functions registered in Unity Catalog - the right home for a calculation that must always be performed the same way (a churn definition, a margin formula).

Permissions: a user needs at least CAN EDIT on the agent to add or remove trusted assets, and agent users need EXECUTE on any SQL function used as one.

-- Codify a metric once, then register it as a trusted asset in the Genie Agent
CREATE OR REPLACE FUNCTION prod_retail.gold.net_margin_pct(
  revenue DECIMAL(18,2), cogs DECIMAL(18,2))
RETURNS DECIMAL(9,4)
COMMENT 'Net margin percent = (revenue - cogs) / revenue. Returns NULL when revenue is zero.'
RETURN CASE WHEN revenue = 0 THEN NULL ELSE (revenue - cogs) / revenue END;

GRANT EXECUTE ON FUNCTION prod_retail.gold.net_margin_pct TO analytics_readers;

Knowledge Store: Descriptions, Joins, and SQL Expressions

The knowledge store holds agent-local semantic context that does not modify Unity Catalog:

• Table and column descriptions scoped to this agent, for when the shared catalog description is too generic.
• Join relationships - Genie respects primary and foreign keys declared in Unity Catalog, but any missing link must be defined manually. Always state the cardinality (one-to-one, one-to-many, many-to-many); without it Genie can generate joins that explode row counts and double-count metrics.
• SQL expressions - reusable measures, filters, and dimensions that encode business semantics.
• Value dictionaries / example values for categorical columns, so Genie can match "Australia" to the stored code AUS instead of guessing.

The Two Limits

Each Genie Agent enforces two separate caps:

More instructions is not better. Databricks warns that too many instructions reduce effectiveness, especially in longer conversations, because Genie struggles to prioritize.

Benchmarks

A benchmark is a set of test questions with expected SQL, used only to evaluate accuracy. Genie does not use benchmark questions or their SQL as context, so adding a benchmark never improves an answer - it only measures whether your instructions did.

---

4. A Working Order of Operations

1. Point the agent at gold-layer tables. Bronze and silver usually lack curated comments and dimensional joins, so accuracy drops.
2. Make sure every included table and column has a real Unity Catalog description.
3. Declare primary and foreign keys in Unity Catalog; add any remaining joins in the knowledge store with cardinality.
4. Add value dictionaries for categorical columns.
5. Add example SQL queries for the hard questions, and SQL functions for calculations that must never vary.
6. Add a short, specific general instructions block last, only for what nothing above covers.
7. Build a benchmark and measure.

---

5. Exam Traps

• Benchmarks do not train Genie. They evaluate it.
• Knowledge store context is agent-local. It does not rewrite Unity Catalog metadata, so two agents can describe the same table differently.
• A missing join is not fixed by a general instruction. Define the relationship and its cardinality.
• Tags are not descriptions. ABAC policies act on tags; Genie and Catalog Explorer search read descriptions.
• CREATE OR REPLACE TABLE silently drops comments applied later with ALTER. Declare them in the DDL or the pipeline definition.