14.2 Pandas-on-Spark Operations, Indexing, and Optimization

Distributed indices and their costs

A pandas index identifies rows for label-based selection and arithmetic. A distributed Spark DataFrame has no implicit stable row order. Pandas API on Spark provides an index abstraction, but its construction and alignment can require distributed work. This distinction explains why familiar small-data code can become slow or change meaning at scale. Choose an existing business key when it represents the intended identity; do not treat generated IDs as durable keys across repartitioning or reloads.

import pyspark.pandas as ps
ps.set_option("compute.default_index_type", "distributed-sequence")

The option name is compute.default_index_type, with an underscore between index and type. In Spark 3.5, distributed-sequence is the default. Check the installed version rather than copying another runtime's defaults. An index is not a database index that accelerates arbitrary lookups. It is a logical row-label column translated into Spark operations.

For a distributed index backed by monotonically_increasing_id(), the current implementation places partition IDs in the upper 31 bits and within-partition record numbers in the lower 33 bits. Thus one partition can contain 0, 1, 2 and another can begin at 8,589,934,592. The values have gaps. They depend on partition IDs and are not a stable business ordering, ranking, or a count of earlier records. Repartitioning can change them. Do not infer a missing record from a gap.

Arithmetic aligns labels, not physical positions

Suppose revenue and cost come from different frames. Subtracting their Series is an index alignment operation, not a promise to zip whatever happens to be the first row of each partition. In Spark 3.5, compute.ops_on_diff_frames defaults to false to guard against expensive accidental joins. If the frames have different anchors, the library can reject the expression. Enabling the option allows supported cross-frame operations, which can add joins and shuffles; exact join behavior depends on the operation.

with ps.option_context("compute.ops_on_diff_frames", True):
    difference = revenue_series - cost_series

Before enabling it, verify that both indices identify the same entities. Two independently generated RangeIndex-like sequences do not prove that customer labels match. When combining data by customer_id, an explicit merge on customer_id is usually easier to reason about. Inspect the resulting Spark plan using psdf.spark.explain(), compare row counts, and check unexpected missing labels or many-to-many key multiplication. An explicit merge can still shuffle; its advantage is deliberate identity semantics.

Batch functions see batches, not the whole dataset

Pandas-on-Spark offers the pandas_on_spark accessor for batch transformation. The DataFrame method is psdf.pandas_on_spark.transform_batch(func), not psdf.transform_batch(func). Its function receives a native pandas batch and must return the same number of rows. Use a transformation with meaning that does not depend on arbitrary batch boundaries.

import pandas as pd
import pyspark.pandas as ps

psdf = ps.DataFrame({"meters": [0.0, 1000.0, 2500.0]})

def to_kilometers(pdf):
    return pd.DataFrame({"kilometers": pdf["meters"] / 1000.0},
                        index=pdf.index)

converted = psdf.pandas_on_spark.transform_batch(to_kilometers)

The expected values are 0.0, 1.0 and 2.5 regardless of how rows are batched. In contrast, computing (pdf - pdf.min()) / (pdf.max() - pdf.min()) inside this function scales each batch separately. It does not compute global min/max scaling, and a constant batch can divide by zero. Compute global statistics explicitly and apply fixed values when global normalization is required. A return annotation must use the supported pandas-on-Spark type-hint conventions; omitting one can require sampling to infer the result schema.

Arrow batch interchange can reduce row-oriented conversion overhead, but it does not make arbitrary Python code automatically vectorized or free of copying. Built-in distributed arithmetic is preferable for a simple division like the example; the batch function is shown to explain the API contract. Pandas UDFs in the Spark SQL API are a separate interface using pandas_udf and Column expressions. Neither interface requires collecting the full dataset into driver pandas.

Practical performance decisions

Use distributed groupby and explicitly keyed merges for large data, and inspect their shuffles. A global sort or an unpartitioned sequence index can become a bottleneck. A rolling window of 7 means seven observations, not automatically seven calendar days; missing days, index ordering and group boundaries matter. Use an event-time Spark window for a time-based requirement when that is the clearer contract.

Methods such as iterrows() transfer rows through a local iterator and usually forfeit distributed transformation benefits. to_pandas() collects the result locally and must fit memory. A supported inplace=True operation updates the pandas-facing object, but does not turn Spark's underlying immutable plan into mutable shared worker memory. Confirm supported arguments rather than assuming every pandas option is implemented.

For each workflow, identify the row key, whether order matters, whether the calculation is batch-local or global, and where materialization occurs. Then inspect the plan and collect only suitably bounded results. Distributed execution adds coordination costs, so native pandas can be the right tool for genuinely small in-memory data; choose from measured size and the required operations.

Official reference

Pandas API options — source checked October 16, 2026. The examples state their runtime assumptions.