7.3 Schema Inspection, Row Iteration, and List Conversion

Inspect structure before values

Exam questions often mix a schema operation with an action. A DataFrame has named SQL columns and a plan; it is not a local list of dictionaries. Read the method's return type and purpose before deciding whether it prints, returns metadata, requests distributed results or builds another DataFrame.

from pyspark.sql import functions as F
people = spark.createDataFrame(
    [(2, "Bo", None), (1, "Ari", 20), (3, "Cy", 30)],
    "id INT, name STRING, score INT"
)
people.printSchema()
schema_object = people.schema
column_names = people.columns
name_type_pairs = people.dtypes

printSchema() prints a schema tree and returns None. schema is a StructType property, not a method called schema(). columns is a list of names; dtypes is a list of name/type-string pairs. Neither columns nor dtypes provides the actual column values. A local dictionary can map field names to types when needed, but it does not validate every record's business meaning. A reader may have already performed schema discovery, so "metadata inspection" does not imply the source was never contacted.

Display versus return

Calling print(people.show()) prints the table and then None. Calling rows = people.limit(2) stores a DataFrame, not two Python rows. A question asking for a list of rows requires an action such as take or collect on a bounded result. A question asking for another distributed DataFrame instead needs a transformation.

Deterministic small conversion

ordered = people.orderBy(F.col("score").desc_nulls_last(), F.col("id").asc())
rows = ordered.limit(2).collect()
row_dicts = [row.asDict(recursive=True) for row in rows]
name_scores = [(row["name"], row["score"]) for row in rows]
# name_scores -> [("Cy", 30), ("Ari", 20)]

The explicit tie breaker id makes the selected order stable if it is unique. A sort by score alone leaves equal-score ordering unspecified. dropDuplicates followed by a sort cannot recover a particular duplicate row already discarded; use row_number over a fully specified window when survivor identity matters. Null ordering is explicit here rather than inferred from an API default.

Row supports field access by name and positional access, but duplicate column names after a join can make conversion ambiguous. Resolve the schema first by joining on named shared keys or selecting aliased columns. asDict(recursive=True) converts nested Row objects to dictionaries, but does not remove array/map values or turn arbitrary complex data into a flat CSV representation. Choose a deliberate serialization format for external consumers.

Iteration and memory

For a result too large for one Python list, toLocalIterator() returns an iterator consumed in the driver/client process. In classic Spark, its memory usage can include the largest partition; prefetching may include another partition. It is not a distributed Python loop with unlimited memory safety. One enormous partition can still cause trouble, and issuing a separate external request for every iterated row may be very slow.

for row in people.select("id", "name").toLocalIterator():
    print(row["id"], row["name"])

This is suitable only for an intentionally manageable local consumer. Use distributed writes for large outputs. toPandas() also collects the full distributed result locally, even with Arrow enabled. Arrow can reduce conversion overhead but does not change the single-machine memory requirement. Under Connect, the receiving process is the client rather than necessarily the remote driver, so assess client memory as well.

Task-side iteration is a different tool

foreach and foreachPartition apply functions in task execution. They do not return a list of processed rows to the driver, and appending to a driver list in those functions does not create shared mutation. foreachPartition can initialize a connection once per task partition instead of once per row, but retries can repeat side effects. Use idempotent writes and consider whether an existing DataFrameWriter connector already handles the target more clearly.

Sorting does not determine concurrency: a dataset with many partitions can still run only as many simultaneous tasks as allocated task slots permit. Conversely, collecting just ten rows does not mean every upstream operation is cheap; an upstream global aggregation or join may still process the entire input. Limit the data logically and inspect the physical plan when cost matters.

Round trip and validation

A small local sequence can become a DataFrame with spark.createDataFrame(sequence, explicit_schema). Local input must fit memory before it is distributed, and the schema should match Python value representations. Empty or all-null collections often require explicit types. Compare expected row count, nullable fields and ordering after a round trip; constructing a new DataFrame from a list does not preserve the original partitioning or sorted presentation contract automatically.

See DataFrame API. For an exam code fragment, name the exact returned object at every step and locate where data crosses from distributed execution into a local collection. That habit prevents confusing schema metadata, lazy transformations and eager actions.