6.3 Complex Nested Types and User-Defined Functions

Modern data pipelines frequently process semi-structured and hierarchical datasets, such as JSON payloads, event logs, and API responses. Apache Spark provides native support for complex data types—including Structs, Arrays, and Maps—allowing developers to manipulate nested records directly within DataFrames. When built-in functions are insufficient, developers can implement custom logic via User-Defined Functions (UDFs). Understanding the internal execution mechanics of complex types and UDFs is essential for writing performant PySpark code.

---

1. Navigating Complex Nested Types

PySpark represents complex hierarchical data using three primary SQL types: StructType, ArrayType, and MapType.

StructType: Hierarchical Objects

A StructType defines a nested record containing named fields, analogous to a JSON object or a C struct. Individual nested fields can be extracted using standard dot notation or the getField() method:

from pyspark.sql import functions as F

# Given a DataFrame with schema: customer: struct<name: string, address: struct<city: string, zip: string>>

# Accessing nested fields using dot notation
df.select(
    F.col("customer.name").alias("customer_name"),
    F.col("customer.address.city").alias("city"),
    F.col("customer.address.zip").alias("postal_code")
)

# Equivalent syntax using getField()
df.select(
    F.col("customer").getField("address").getField("city").alias("city")
)

To construct a new struct from flat columns, use F.struct(*cols):

# Packing flat columns into a nested struct
nested_df = df.withColumn(
    "location",
    F.struct("city", "state", "country")
)

ArrayType: Ordered Sequences

An ArrayType represents an ordered list of elements sharing an identical data type. PySpark provides rich built-in functions for inspecting and manipulating arrays:

# Array inspection and manipulation functions
df.select(
    F.size("tags").alias("tag_count"),                     # Number of elements in array
    F.array_contains("tags", "spark").alias("has_spark"),    # Boolean check for element
    F.element_at("tags", 1).alias("first_tag"),            # 1-indexed element retrieval
    F.array_distinct("tags").alias("deduped_tags"),         # Deduplicated array elements
    F.array_sort("tags").alias("sorted_tags"),              # Naturally sorted array
    F.array_union("tags_a", "tags_b").alias("all_tags")    # Distinct union of two arrays
)

Unnesting Arrays: explode() vs. explode_outer()

To transform array elements into individual DataFrame rows, Spark provides the explode family of functions:

# Exploding an array into separate rows
df.select("order_id", F.explode("item_ids").alias("item_id"))

MapType: Key-Value Pairs

A MapType stores key-value associations. Values are accessed using bracket indexing or functions:

# Extracting map keys, values, and specific entries
df.select(
    F.map_keys("attributes").alias("keys"),
    F.map_values("attributes").alias("values"),
    F.col("attributes")["color"].alias("color_value"),
    F.explode("attributes").alias("attr_key", "attr_value")
)

---

2. Standard Python User-Defined Functions (UDFs)

When built-in Spark SQL functions cannot express custom business logic, developers can create User-Defined Functions (UDFs).

from pyspark.sql.functions import udf
from pyspark.sql.types import StringType

# Define a standard Python UDF using decorator syntax
@udf(returnType=StringType())
def clean_text(text: str) -> str:
    if text is None:
        return None
    return text.strip().lower()

# Applying the UDF to a DataFrame column
df_cleaned = df.withColumn("clean_name", clean_text(F.col("raw_name")))

To make a UDF accessible within Spark SQL string queries, register it with the session catalog:

spark.udf.register("clean_text_sql", clean_text)
spark.sql("SELECT clean_text_sql(raw_name) FROM users_table")

The Architectural Bottlenecks of Standard Python UDFs

Standard Python UDFs are notorious performance bottlenecks. Understanding why they are slow is a core topic on the Databricks developer exam:

1. JVM-to-Python IPC Boundary: Spark's core engine executes inside the Java Virtual Machine (JVM). When a standard Python UDF is invoked, the Executor JVM must spawn an external Python worker daemon process (python3).
2. Scalar execution and batched transport: Ordinary Python UDFs apply scalar logic to each row. Spark batches serialized transport between executor JVM and Python workers; it is not a separate Py4J call per record. Arrow-optimized scalar UDFs are another version-dependent option.
3. Catalyst Optimizer Black Box: To Spark's Catalyst optimizer and Tungsten execution engine, a standard Python UDF is a completely opaque black box. Catalyst cannot inspect the interior logic of the Python code. As a result:
   • Catalyst cannot push filter predicates through or into the UDF.
   • The Python evaluation boundary cannot be fused into a JVM codegen region; codegen can still apply to supported operators before and after it.
   • Spark cannot optimize memory allocation, leading to significant CPU serialization overhead.

---

3. Vectorized UDFs: Pandas UDFs with Apache Arrow

To eliminate the serialization overhead and row-by-row execution penalties of standard UDFs, Spark introduced Vectorized UDFs (also known as Pandas UDFs), powered by Apache Arrow.

import pandas as pd
from pyspark.sql.functions import pandas_udf
from pyspark.sql.types import DoubleType

# Define a Vectorized Pandas UDF (Series to Series)
@pandas_udf(DoubleType())
def calculate_vat(prices: pd.Series) -> pd.Series:
    # Operates on a vectorized Pandas Series in C-speed SIMD
    return prices * 0.20

df_vat = df.withColumn("vat_amount", calculate_vat(F.col("price")))

Arrow execution and the pandas UDF contract

Arrow provides a columnar interchange format. Spark transfers batches between separate JVM and Python processes, then converts inputs/outputs as needed. Do not assume shared address space, zero copying throughout, or automatic SIMD for arbitrary Python code. Vectorized pandas operations often reduce scalar Python overhead; built-in Spark functions remain preferable when they express the logic.

The scalar Series-to-Series function above must return one result per input row, with a pandas representation compatible with the declared DoubleType. Handle nulls deliberately and return the same length and row order. A batch-local mean is not a global mean: each batch sees only part of a partition. The VAT input here is assumed to be a double-valued price column; for exact financial arithmetic choose a deliberate decimal policy.

# Controlling batch sizes for Arrow execution
spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", "10000")

Comparison: Standard UDF vs. Pandas Vectorized UDF

Null safety and choosing the right API

A UDF may be evaluated outside the apparent short-circuit order of surrounding SQL predicates. The function itself must safely handle nulls and invalid input; do not rely on a preceding when() or filter to protect an unsafe division or string operation. Keep dependencies available on executor Python workers, not just the driver. Test both a valid value and None, then inspect the resulting schema. Register an existing decorated UDF without specifying its returnType again; register a plain Python function with an explicit return type.

Pandas API on Spark and pandas UDFs are different tools. The former offers a distributed pandas-like DataFrame interface; the latter plugs Python batch computation into a Spark SQL Column expression. Creating a pandas UDF does not require converting the whole distributed DataFrame to local pandas. A stateless UDF transforms the current inputs; it does not maintain reliable per-customer history between batches. Use a stateful streaming operator and its checkpointed StateStore for that requirement.

Official reference

Arrow and pandas UDFs — source checked October 16, 2026. The examples state their runtime assumptions.