Databricks Notebooks and Code Execution

Quick Answer: Notebooks support cell-by-cell execution with magic commands for multi-language support. Use %run for importing code, dbutils.notebook.run() for orchestrated execution, and understand lazy evaluation — transformations only execute when an action is called.

Multi-Language Support

Magic Commands

Cross-Language Variable Sharing

Variables defined in one language cell are NOT automatically available in cells of another language. To share data:

# Python: register DataFrame as a temp view
df.createOrReplaceTempView("my_data")

%sql
-- SQL: access the temp view
SELECT * FROM my_data WHERE amount > 100

Running Other Notebooks

%run (Inline Execution)

# Execute helpers notebook — all its variables/functions become available here
%run ./helpers

# Now you can use functions defined in helpers.py
result = process_data(df)

• Executes in the same Spark session and process
• Variables and functions from the child notebook are available in the parent
• Cannot pass parameters (use widgets instead)
• Blocking: Waits for the child notebook to complete

dbutils.notebook.run() (Child Job Execution)

# Run a notebook as a separate child job
result = dbutils.notebook.run(
    path="./data_processing",
    timeout_seconds=600,
    arguments={"start_date": "2026-01-01", "catalog": "prod"}
)
print(f"Child notebook returned: {result}")

• Executes in a separate process
• Returns a string value (set via dbutils.notebook.exit())
• Can pass parameters as key-value pairs
• Useful for orchestrating multiple notebooks

Lazy Evaluation

Spark uses lazy evaluation — transformations are not executed immediately. They are only executed when an action is triggered.

Transformations (Lazy — Build Execution Plan)

# These lines do NOT execute yet — they build a plan
df = spark.table("orders")                    # Read plan
filtered = df.filter(col("amount") > 100)     # Filter plan
grouped = filtered.groupBy("customer_id")     # Group plan
result = grouped.agg(sum("amount").alias("total"))  # Aggregation plan

Actions (Eager — Trigger Execution)

# These lines TRIGGER execution of the entire plan
result.show()           # Display results
result.count()          # Return row count
result.collect()        # Return all rows as list
result.write.saveAsTable("summary")  # Write to table
display(result)         # Databricks display function

Why Lazy Evaluation Matters

1. Optimization: Spark optimizes the entire plan before executing (e.g., predicate pushdown)
2. Efficiency: Only the data needed for the final result is read and processed
3. Pipeline composition: You can chain many transformations without intermediate materializations

On the Exam: Understand that transformations are lazy and actions are eager. Know the difference between %run (shared scope, no parameters) and dbutils.notebook.run() (separate process, accepts parameters, returns string).