Testing and Deployment Best Practices

Quick Answer: Use separate Unity Catalog catalogs for dev/staging/prod environments. Parameterize code with environment variables. Deploy via Databricks Asset Bundles with CI/CD. Test with integration tests on sample data in staging before promoting to production.

Environment Isolation

Unity Catalog-Based Isolation

Production:   prod_catalog.schema.table
Staging:      staging_catalog.schema.table
Development:  dev_catalog.schema.table

Parameterization

# Use widgets for environment-specific configuration
dbutils.widgets.text("catalog", "dev_catalog", "Target Catalog")
catalog = dbutils.widgets.get("catalog")

# All table references use the parameterized catalog
spark.sql(f"USE CATALOG {catalog}")
spark.sql(f"SELECT * FROM {catalog}.silver.orders")

-- SQL with parameter
USE CATALOG ${catalog};
SELECT * FROM silver.orders;

Testing Strategies

Unit Tests

# Test individual transformation functions
def test_classify_amount():
    assert classify_amount(1500) == "high"
    assert classify_amount(500) == "medium"
    assert classify_amount(50) == "low"

# Run with pytest
# pytest tests/test_transforms.py

Integration Tests

# Test end-to-end pipeline with sample data
def test_pipeline_end_to_end():
    # 1. Create sample input data
    sample_data = spark.createDataFrame([
        (1, 100, "2026-01-01"),
        (2, 200, "2026-01-02"),
    ], ["order_id", "amount", "order_date"])

    # 2. Write to staging bronze table
    sample_data.write.mode("overwrite").saveAsTable(
        "staging_catalog.bronze.test_orders"
    )

    # 3. Run the pipeline (or the transformation notebook)
    dbutils.notebook.run("./transform", timeout_seconds=300)

    # 4. Verify output
    result = spark.table("staging_catalog.silver.test_orders")
    assert result.count() == 2
    assert result.filter("amount <= 0").count() == 0

Data Quality Tests

-- Validate data quality after pipeline run
-- These can be SQL tasks in a Databricks job

-- Test: No null order IDs
SELECT COUNT(*) AS null_count
FROM prod_catalog.silver.orders
WHERE order_id IS NULL;
-- Expected: 0

-- Test: All amounts are positive
SELECT COUNT(*) AS negative_count
FROM prod_catalog.silver.orders
WHERE amount <= 0;
-- Expected: 0

-- Test: No duplicate orders
SELECT order_id, COUNT(*) AS cnt
FROM prod_catalog.silver.orders
GROUP BY order_id HAVING cnt > 1;
-- Expected: 0 rows

Deployment Workflow

Recommended CI/CD Pipeline

1. Developer creates feature branch
2. Developer tests locally in dev environment
3. Pull request submitted for code review
4. CI pipeline runs:
   a. databricks bundle validate -t staging
   b. Deploy to staging: databricks bundle deploy -t staging
   c. Run integration tests: databricks bundle run test-job -t staging
5. Code review approved
6. Merge to main branch
7. CD pipeline runs:
   a. databricks bundle validate -t prod
   b. Deploy to production: databricks bundle deploy -t prod

Deployment Checklist

Lakeflow Pipeline Modes

On the Exam: Understand the concept of environment isolation using catalogs, parameterization for environment-agnostic code, and the deployment workflow from dev to staging to production using Asset Bundles and CI/CD.