1.7 Creating a Feature Table and Writing Data to It

1.7 Creating a Feature Table and Writing Data to It

Creating a feature table is a two-step act: define the table with its keys and schema, then write feature values into it on a schedule. Both steps go through FeatureEngineeringClient, and both are directly examinable.

---

Initializing and Creating Feature Tables

Feature tables in Unity Catalog are standard Delta Lake tables augmented with explicit feature metadata (such as primary key constraints). Creating and writing to feature tables is managed via the FeatureEngineeringClient.

Step 1: Client Initialization and Table Creation

from databricks.feature_engineering import FeatureEngineeringClient
from pyspark.sql import functions as F

# Initialize modern Feature Engineering Client
fe = FeatureEngineeringClient()

# Compute feature DataFrame with business aggregation logic
raw_transactions = spark.table("prod_catalog.source_data.transactions")

customer_features_df = raw_transactions.groupBy("customer_id").agg(
    F.count("transaction_id").alias("total_tx_count"),
    F.avg("amount").alias("avg_transaction_amount"),
    F.max("transaction_date").alias("last_transaction_date")
)

# Define the full 3-level table name
feature_table_name = "prod_catalog.ml_features.customer_spending_features"

# Create the feature table in Unity Catalog
fe.create_table(
    name=feature_table_name,
    primary_keys=["customer_id"],
    df=customer_features_df,
    description="Aggregated customer lifetime transaction and spending metrics",
    tags={"domain": "finance", "tier": "gold"}
)

Key Parameters of fe.create_table

• name: Fully qualified 3-level name (catalog.schema.table).
• primary_keys: List of column names that uniquely identify entity records (e.g., ["customer_id"] or composite keys ["user_id", "device_id"]). Primary keys are mandatory for feature lookups.
• timeseries_column: (Optional, singular string) The single column holding the timestamp of each feature observation. Required when the table will be used for point-in-time lookups. Writing timeseries_columns=[...] is a common and incorrect guess — the parameter takes one column name, not a list.
• df: Optional PySpark DataFrame used to populate the table schema and initial data.
• schema: Explicit StructType schema (required if df is omitted).

---

Writing and Updating Feature Tables

Feature values change continuously as new batch or streaming transactions arrive. The FeatureEngineeringClient.write_table API provides controlled update semantics:

# Incremental daily batch update
new_daily_features_df = compute_daily_customer_features()

# Upsert features into Unity Catalog table
fe.write_table(
    name="prod_catalog.ml_features.customer_spending_features",
    df=new_daily_features_df,
    mode="merge"  # Performs Delta Lake MERGE on primary keys
)

Write Modes Supported

• mode="merge" (Recommended): Performs an intelligent Delta Lake MERGE (upsert) operation using the table's defined primary_keys. If an incoming entity record already exists, its feature values are updated; if the entity is new, a new row is inserted.
• mode="overwrite": Completely truncates and replaces the existing feature table contents with the incoming DataFrame. Used for full historical recalculations.

---

Choosing Between the Creation Modes

Rules that decide exam answers

• Primary keys are not optional. They are the join keys for FeatureLookup. A table created without them cannot participate in create_training_set.
• A composite key is a list: primary_keys=["user_id", "device_id"]. The observation DataFrame must supply every key column at lookup time.
• timeseries_column takes one string. It is only needed for point-in-time tables; a static entity table (demographics, product attributes) omits it.
• Merge is keyed, not appended. mode="merge" matches incoming rows to existing rows on the primary keys, updating matches and inserting new entities. It will not produce duplicate rows for the same key, which is exactly why it is the default for incremental refreshes.
• Reading back is ordinary Delta. fe.read_table(name=...) returns the feature table as a Spark DataFrame, and spark.table(name) works too — a feature table is a Delta table with extra metadata, not a special storage format.

---

Unity Catalog Governance, Search & Automated Lineage

Because feature tables in Unity Catalog are native Delta tables, enterprise security policies apply seamlessly:

-- Grant read permissions to data science group
GRANT SELECT ON TABLE prod_catalog.ml_features.customer_spending_features 
TO `data-scientists`;

-- Grant write/update permissions to automated ETL service principal
GRANT MODIFY ON TABLE prod_catalog.ml_features.customer_spending_features 
TO `sp-feature-pipeline`;

Automated Lineage Graph

Unity Catalog captures end-to-end lineage automatically without manual instrumentation:

1. Upstream Lineage: Displays raw bronze/silver Delta source tables queried to build the feature table.
2. Downstream Lineage: Automatically identifies which MLflow training runs, registered models, and scheduled batch inference pipelines consume features from the table.

[ Bronze Raw Logs ] ---> [ Silver Cleaned Data ] ---> [ Feature Table (Unity Catalog) ]
                                                                   |
                                   +-------------------------------+-------------------------------+
                                   v                                                               v
                     [ MLflow Model Training Run ]                                 [ Batch Scoring Pipeline ]
                                   |
                                   v
                     [ Registered Model in UC ]