5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design

5.3 Efficient Columnar Formats: Parquet, ORC, and Avro Schema Design

Data Storage Paradigms: Row-Oriented vs. Columnar

Choosing the optimal file format is one of the most impactful architectural decisions in AWS data engineering. Storage formats dictate file serialization efficiency, network I/O, CPU utilization, schema evolution capabilities, and query cost.

Storage formats fall into two broad paradigms:

Row-Oriented Layout (Avro, CSV, JSON):
[Row 1: ColA, ColB, ColC] -> [Row 2: ColA, ColB, ColC] -> [Row 3: ColA, ColB, ColC]

Columnar Layout (Parquet, ORC):
[Row Group 1: ColA values] -> [Row Group 1: ColB values] -> [Row Group 1: ColC values]

• Row-Oriented Formats (JSON, CSV, Apache Avro): Store all column values for a single record sequentially on disk. Highly efficient for write-heavy transactional operations, row-by-row streaming ingestion (e.g., Kafka, Kinesis), and queries that read full object payloads (SELECT *).
• Columnar Formats (Apache Parquet, Apache ORC): Store all values for a single column contiguously on disk within blocks called Row Groups or Stripes. Optimized for read-heavy analytical queries (OLAP), where queries select specific column subsets across millions of records.

---

Deep Dive: Apache Parquet & Apache ORC

Apache Parquet Architecture

Apache Parquet is an open-source, columnar storage format designed for complex nested data structures. A Parquet file contains:

• Header: Contains magic bytes identifying the format.
• Row Groups: Logical horizontal slices of data containing 128 MB to 1 GB of records.
• Column Chunks: Columnar data stored contiguously within a Row Group.
• Pages: Individual 1 MB data pages inside column chunks containing compressed values and encoding dictionaries.
• File Footer: Crucial metadata section stored at the end of the file. Contains table schemas, Row Group metadata, column statistics (min/max values, null counts), and byte offsets for column chunks.

Apache ORC Architecture

Apache ORC (Optimized Row Columnar) is widely used in Apache Hive and Presto workloads. ORC divides files into Stripes (typically 64 MB to 256 MB). Each stripe contains:

• Index Data: Min, max, sum, and count statistics for every 10,000 rows.
• Row Data: Columnar data streams.
• Stripe Footer: Contains encoding types and column stream locations.
• PostScript: Embedded footer containing file metadata, compression buffer size, and schema definitions.

---

Mechanics of Projection and Predicate Pushdown

Columnar formats achieve massive query acceleration in Amazon Athena, AWS Glue, EMR Spark, and Redshift Spectrum through two key optimization techniques:

1. Projection Pushdown

When an analytical query runs against a 100-column table but selects only two columns (SELECT user_id, order_total FROM orders), projection pushdown allows the query engine to:

1. Fetch and parse the Parquet file footer metadata.
2. Calculate the exact S3 byte ranges containing only user_id and order_total column chunks.
3. Issue targeted S3 Byte-Range GET requests to retrieve only those specific column byte streams, ignoring the other 98 columns entirely. This reduces network I/O and data scanning volume by up to 98%.

2. Predicate Pushdown (Row Group Skipping)

When a query contains filter predicates (e.g., WHERE order_total > 5000), predicate pushdown evaluates the filter criteria against the min/max statistics embedded inside file footers and row group metadata before reading raw data bytes:

• If a Row Group's metadata indicates min_order_total = 10 and max_order_total = 4200, the engine instantly skips that entire 128 MB Row Group without reading its data pages.
• Entire files or row groups are bypassed at zero scan cost.

---

Deep Dive: Apache Avro & Schema Evolution

Apache Avro is a row-oriented, binary serialization format driven by JSON-defined schemas. Unlike CSV or JSON, Avro stores data in a compact binary payload while enforcing strict schema definitions (.avsc).

Avro Schema Evolution Rules

Avro is the preferred storage format for landing raw event streams (e.g., in Kafka/Kinesis pipelines) because of its robust support for Schema Evolution:

• Backward Compatibility: A new schema can read data written by an older schema. (Adding optional fields with default values).
• Forward Compatibility: An older schema can read data written by a newer schema. (Removing fields that have default values).
• Full Compatibility: Schemas are both backward and forward compatible.

Because Avro binary payloads embed or reference schema definitions in the AWS Glue Schema Registry, applications can evolve payload fields over time without breaking downstream ETL ingest pipelines.

---

Compression Codecs: Snappy vs. ZSTD vs. GZIP

Splittability Impact on Spark Parallelism

A file format or compression codec is splittable if a distributed framework (Spark/EMR) can divide a single large file across multiple map task threads. Unsplittable raw GZIP files force a single Spark worker node to process the entire file sequentially, creating CPU bottlenecks. Parquet and ORC internal row groups restore splittability even when using compressed blocks.

---

Comprehensive Storage Format Selection Matrix

---

Code & Schema Example: Avro Schema & PySpark Parquet Conversion

{
  "type": "record",
  "name": "TelemetryEvent",
  "namespace": "com.aws.datalake",
  "fields": [
    {"name": "device_id", "type": "string"},
    {"name": "event_timestamp", "type": "long"},
    {"name": "temperature", "type": ["null", "double"], "default": null},
    {"name": "status_code", "type": "string", "default": "OK"}
  ]
}

# PySpark ETL: Read Avro/JSON Ingest Stream -> Convert to Compressed Parquet
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("AvroToParquetETL").getOrCreate()

# Read incoming streaming landing data
raw_df = spark.read.format("avro").load("s3://landing-zone-bucket/avro-events/2026/08/13/")

# Transform & write to analytics data lake as ZSTD-compressed Parquet
raw_df.write \
    .mode("append") \
    .format("parquet") \
    .option("compression", "zstd") \
    .option("parquet.block.size", 134217728) # 128 MB Row Group size \
    .save("s3://analytics-data-lake-bucket/fact_telemetry/")