3.3 Views: Temporary vs Global Temporary Views

In Apache Spark SQL, views allow developers to register DataFrames or SQL transformation logic as queryable relational objects without incurring the storage overhead of writing physical files to cloud storage.

---

1. Relational Views in Apache Spark

A view is a virtual table defined by a logical query plan. Unlike physical tables:

• Physical Tables persist both metadata (in the catalog metastore) and physical data files (Parquet, Delta, ORC) on storage.
• Views store only the query definition. When a view is queried, Spark embeds the view's logical plan directly into the outer query's plan tree. The Catalyst optimizer can then perform cross-boundary optimizations, including predicate pushdown, projection pruning, and join reordering.

Spark SQL supports three distinct view varieties:

1. Standard Temporary Views (Session-scoped)
2. Global Temporary Views (Application-scoped)
3. Permanent Catalog Views (Metastore-scoped)

---

2. Standard Temporary Views (Session-Scoped)

A standard temporary view is tied strictly to the lifecycle of the specific SparkSession that created it. It is not registered in an external metastore (Hive or Unity Catalog) and is stored entirely in driver memory.

Registration Methods in PySpark

# Read a source dataset
orders_df = spark.read.parquet("s3://lakehouse/orders/")

# Create a temporary view; throws AnalysisException if 'temp_orders' already exists
orders_df.createTempView("temp_orders")

# Create or replace an existing temporary view safely
orders_df.createOrReplaceTempView("temp_orders")

Registration via SQL DDL

CREATE OR REPLACE TEMPORARY VIEW temp_orders AS
SELECT order_id, customer_id, total_amount, order_date
FROM bronze_orders
WHERE order_status = 'COMPLETED';

Scope and Lifecycle Boundaries

• Session Bound: Standard temporary views are accessible only within the creating SparkSession. If another session is instantiated using spark2 = spark.newSession(), executing spark2.table("temp_orders") throws an AnalysisException: Table or view not found.
• Automatic Termination: When the SparkSession is stopped via spark.stop() or the notebook/script session finishes, the temporary view is immediately destroyed and deallocated from driver memory.

Programmatic Dropping

To drop a standard temporary view before session termination:

# Programmatic drop via Catalog API
spark.catalog.dropTempView("temp_orders")

In SQL:

DROP VIEW IF EXISTS temp_orders;

3. Global Temporary Views (Application-Scoped)

When multiple SparkSession instances running within the same Spark application (the same Driver JVM) must collaborate on a shared in-memory relation, a standard temporary view is insufficient. Spark provides global temporary views to satisfy this requirement.

Registration Methods in PySpark

# Create an application-wide global temporary view
orders_df.createGlobalTempView("global_orders")  # Throws AnalysisException if exists

# Create or replace safely
orders_df.createOrReplaceGlobalTempView("global_orders")

Registration via SQL DDL

CREATE OR REPLACE GLOBAL TEMPORARY VIEW global_orders AS
SELECT order_id, customer_id, total_amount
FROM bronze_orders
WHERE total_amount > 100.00;

The System-Managed global_temp Database

To prevent naming collisions between session-scoped temporary views and application-wide global temporary views, Spark automatically places all global temporary views into a dedicated system-preserved database named global_temp.

Mandatory Namespace Qualification

Whenever a query references a global temporary view, the developer must explicitly qualify the view name with the global_temp database prefix:

# CORRECT: Fully qualified with global_temp
spark.sql("SELECT * FROM global_temp.global_orders").show()
df_global = spark.table("global_temp.global_orders")

# INCORRECT: Fails with AnalysisException: Table or view not found: global_orders
# spark.sql("SELECT * FROM global_orders").show()

This is one of the most frequently tested topics on Databricks developer examinations. Unqualified access fails because Spark searches the current active database and the local session's temporary views, ignoring the global_temp database unless explicitly instructed.

Lifecycle Boundaries

• Application Bound: Global temporary views remain alive as long as the Spark application driver is running.
• Cross-Session Visibility: Any session created within the driver JVM (via spark.newSession()) can query global_temp.global_orders.
• Explicit Dropping: A global temporary view can be dropped using spark.catalog.dropGlobalTempView("global_orders") or SQL DROP VIEW IF EXISTS global_temp.global_orders;.

4. Multi-Session Code Walkthrough

The following complete PySpark example demonstrates the isolation and sharing boundaries between standard temporary views and global temporary views across sessions:

from pyspark.sql import SparkSession

# 1. Initialize primary SparkSession
spark = SparkSession.builder.appName("ViewScopingWalkthrough").getOrCreate()

# 2. Construct sample DataFrame
data = [(1, "Alice", 500), (2, "Bob", 1200), (3, "Charlie", 800)]
columns = ["id", "name", "score"]
df = spark.createDataFrame(data, columns)

# 3. Register standard and global temporary views
df.createOrReplaceTempView("local_view")
df.createOrReplaceGlobalTempView("shared_view")

# 4. Query within the creating session
spark.sql("SELECT * FROM local_view").show()              # SUCCEEDS
spark.sql("SELECT * FROM global_temp.shared_view").show()  # SUCCEEDS

# 5. Spawn an isolated child session
child_session = spark.newSession()

# 6. Attempt to query local_view from child_session
try:
    child_session.sql("SELECT * FROM local_view").show()
except Exception as e:
    print("child_session cannot access local_view:", type(e).__name__)
    # Output: child_session cannot access local_view: AnalysisException

# 7. Query shared_view from child_session
child_session.sql("SELECT * FROM global_temp.shared_view").show()
# SUCCEEDS: Global temporary views are visible across all sessions

---

5. Architectural Comparison: Views vs Tables

6. Critical Exam Traps and Best Practices

When preparing for the certification exam, be vigilant regarding these subtle behaviors:

1. Omitting the global_temp. Prefix: Never write SELECT * FROM my_global_view. The global temporary database is reserved and cannot be selected as an ordinary current database; use global_temp.<view_name>.
2. createTempView vs createOrReplaceTempView: Calling df.createTempView("v") when v already exists throws an AnalysisException: Temporary table 'v' already exists. Always use createOrReplaceTempView when idempotent view registration is required.
3. View Plan Re-evaluation: Because views store query plans rather than cached data, querying a view three times recalculates the underlying transformations three times. If a view encapsulates an expensive shuffle or aggregations, call spark.catalog.cacheTable("view_name") or df.cache() to prevent redundant cluster computation.