7.1 List, Dictionary, and Set Comprehensions

List, Dictionary, and Set Comprehensions

Python embraces both object-oriented and functional programming paradigms. Among Python's most expressive and widely utilized functional features are comprehensions. A comprehension is a concise, declarative construct that creates a new collection—such as a list, dict, or set—by applying an expression to each item across one or more iterables while optionally filtering elements.

---

1. Comprehension Foundations and Set-Builder Paradigm

Comprehensions originate from mathematical set-builder notation. In mathematics, one defines a set of squares as:

$\{ x^2 \mid x \in \mathbb{N}, x < 10 \}$

In traditional imperative programming, generating a transformed collection requires initializing an empty accumulator, writing an explicit for loop, and repeatedly invoking mutation methods like .append() or .add():

# Imperative approach (verbose and slower)
squares = []
for x in range(10):
    if x % 2 == 0:
        squares.append(x ** 2)
print(squares)  # [0, 4, 16, 36, 64]

In Python, the list comprehension expresses this transformation declaratively in a single line:

# Declarative list comprehension
squares = [x ** 2 for x in range(10) if x % 2 == 0]
print(squares)  # [0, 4, 16, 36, 64]

The Bytecode Performance Advantage

Comprehensions are not merely syntactic sugar—they are significantly faster than equivalent imperative for loops in CPython:

1. Specialized Bytecode Instruction: Comprehensions use the dedicated C-level LIST_APPEND, MAP_ADD, or SET_ADD bytecode instructions directly on the collection being built.
2. No Method Lookup Overhead: In an imperative loop, Python must resolve the .append attribute on the list object on every single iteration (LOAD_METHOD / CALL_METHOD). Comprehensions bypass Python-level attribute lookups entirely.
3. Optimized Pre-allocation: The CPython Virtual Machine optimizes internal array resizing for comprehensions when iterable lengths are known.

---

2. List Comprehension Syntax and Conditional Logic

A list comprehension is always enclosed in square brackets [...] and contains an output expression followed by at least one for clause and zero or more optional clauses.

A. Basic Syntax

[expression for item in iterable]

numbers = [1, 2, 3, 4, 5]
doubled = [n * 2 for n in numbers]
print(doubled)  # [2, 4, 6, 8, 10]

# Transforming strings
words = ["python", "pcap", "certification"]
uppercased = [w.upper() for w in words]
print(uppercased)  # ['PYTHON', 'PCAP', 'CERTIFICATION']

B. Conditional Filtering: Trailing if

To select a subset of elements, append an if clause to the end of the comprehension:

[expression for item in iterable if condition]

The condition is evaluated for each item in the iterable. If the condition evaluates to True, the item is passed to expression and inserted into the resulting list; if False, the item is discarded.

raw_data = [12, -7, 34, 0, -15, 88, -1, 42]
positives = [x for x in raw_data if x > 0]
print(positives)  # [12, 34, 88, 42]

# Multiple filtering conditions (equivalent to logical and)
even_positives = [x for x in raw_data if x > 0 if x % 2 == 0]
print(even_positives)  # [12, 34, 88, 42]

C. Conditional Value Transformation: Inline Ternary if/else

A major point of confusion on the PCAP exam is the distinction between filtering elements and conditionally transforming elements:

• Filtering (if at the end): Dictates which items enter the new list. Items that fail the condition are excluded.
• Ternary Value Expression (if-else before for): Dictates what value each item transforms into. All items from the iterable are retained, but their mapped values differ based on the condition.

[expr_if_true if condition else expr_if_false for item in iterable]

scores = [72, 45, 88, 91, 58, 64]

# Label each score as 'Pass' or 'Fail'
status = ["Pass" if s >= 60 else "Fail" for s in scores]
print(status)  # ['Pass', 'Fail', 'Pass', 'Pass', 'Fail', 'Pass']

# Replace negative numbers with zero while keeping positive values unchanged
readings = [10, -5, 20, -1, 0, 30]
normalized = [x if x > 0 else 0 for x in readings]
print(normalized)  # [10, 0, 20, 0, 0, 30]

Combining Ternary Transformation with Filtering

You can combine both constructs in a single comprehension:

# Only consider numbers > 0 (filter), then square even numbers and negate odd numbers (transform)
vals = [1, 2, 3, 4, 5, -2, -4]
result = [x ** 2 if x % 2 == 0 else -x for x in vals if x > 0]
print(result)  # [-1, 4, -3, 16, -5]

[!WARNING] Syntax Trap: Writing [x for x in vals else 0] or [x for x in vals if x > 0 else 0] is a SyntaxError. The else keyword can only appear as part of a ternary expression before the for keyword.

---

3. Nested Comprehensions and Multi-Dimensional Structures

Comprehensions support multiple for clauses to iterate over nested sequences or multi-dimensional structures.

The Left-to-Right Loop Evaluation Order

When multiple for clauses appear in a comprehension, they execute in left-to-right order, matching the exact nesting order of standard for loops:

# Imperative nested loop
result = []
for outer in ["A", "B"]:
    for inner in [1, 2]:
        result.append(f"{outer}{inner}")

# Equivalent comprehension (outer loop first, inner loop second)
combos = [f"{outer}{inner}" for outer in ["A", "B"] for inner in [1, 2]]
print(combos)  # ['A1', 'A2', 'B1', 'B2']

Flattening a 2D Matrix

A classic application tested on the PCAP exam is flattening a two-dimensional grid into a one-dimensional list:

matrix = [
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]
]

# Flatten: for row in matrix, then for item in row
flattened = [item for row in matrix for item in row]
print(flattened)  # [1, 2, 3, 4, 5, 6, 7, 8, 9]

# Flatten with condition: only even numbers
evens_flat = [item for row in matrix for item in row if item % 2 == 0]
print(evens_flat)  # [2, 4, 6, 8]

Nested List Comprehensions: Matrix Construction and Transposition

When constructing a multi-dimensional list (a list of lists), the output expression of the comprehension is itself another list comprehension:

rows, cols = 3, 4
# Construct a 3x4 zero-initialized matrix
grid = [[0 for _ in range(cols)] for _ in range(rows)]
print(grid)
# [[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]

# Transpose a 3x2 matrix into a 2x3 matrix
matrix_3x2 = [[1, 2], [3, 4], [5, 6]]
transposed = [[row[i] for row in matrix_3x2] for i in range(2)]
print(transposed)  # [[1, 3, 5], [2, 4, 6]]

The List Multiplication Reference Pitfall

Understanding why nested comprehensions are required for matrix creation is a critical Python concept:

# DANGEROUS: Replicates the SAME list reference 3 times!
bad_grid = [[0] * 3] * 3
bad_grid[0][0] = 99
print(bad_grid)  # [[99, 0, 0], [99, 0, 0], [99, 0, 0]] - ALL rows modified!

# CORRECT: Evaluates the inner list comprehension 3 independent times
good_grid = [[0] * 3 for _ in range(3)]
good_grid[0][0] = 99
print(good_grid)  # [[99, 0, 0], [0, 0, 0], [0, 0, 0]] - Only row 0 modified

---

4. Dictionary and Set Comprehensions

Python extends the comprehension syntax to dictionaries and sets using curly braces {}.

A. Dictionary Comprehensions

A dictionary comprehension requires a key: value expression pair separated by a colon:

{key_expr: value_expr for item in iterable if condition}

names = ["alice", "bob", "charlie", "david"]
# Map name to its character length
name_lengths = {name: len(name) for name in names}
print(name_lengths)
# {'alice': 5, 'bob': 3, 'charlie': 7, 'david': 5}

# Square mapping for even numbers
even_squares = {x: x ** 2 for x in range(10) if x % 2 == 0}
print(even_squares)
# {0: 0, 2: 4, 4: 16, 6: 36, 8: 64}

Inverting Dictionaries and Handling Key Collisions

Dictionary comprehensions provide a clean idiom for swapping keys and values ({v: k for k, v in d.items()}):

ports = {"HTTP": 80, "HTTPS": 443, "SSH": 22, "FTP": 21}
inverted = {port_num: service for service, port_num in ports.items()}
print(inverted)
# {80: 'HTTP', 443: 'HTTPS', 22: 'SSH', 21: 'FTP'}

[!IMPORTANT] Duplicate Key Overwrites: If multiple keys in the original dictionary share the same value, inverting the dictionary causes duplicate keys in the new dictionary. Python dictionaries enforce unique keys, so later keys will silently overwrite earlier ones:

grades = {"Alice": "A", "Bob": "B", "Charlie": "A"}
inv_grades = {grade: student for student, grade in grades.items()}
print(inv_grades)  # {'A': 'Charlie', 'B': 'Bob'} ('Alice' was overwritten!)

B. Set Comprehensions

A set comprehension evaluates an expression inside curly braces without colons. Sets automatically deduplicate elements and maintain no guarantee of order:

{expression for item in iterable if condition}

sentence = "the quick brown fox jumps over the lazy dog and the fox is quick"
# Extract unique word lengths greater than 3 characters
word_lengths = {len(w) for w in sentence.split() if len(w) > 3}
print(word_lengths)  # {4, 5}

# Deduplicating transformed items
raw_tags = [" python ", "PYTHON", "Java ", "python", "JAVA", "C++"]
cleaned_tags = {tag.strip().lower() for tag in raw_tags}
print(cleaned_tags)  # {'python', 'java', 'c++'}

---

5. Scope Isolation and Performance in Python 3

One of the most consequential architectural changes between Python 2 and Python 3 is how comprehensions handle variable scoping.

Loop Variable Leakage vs Scope Isolation

In Python 2, list comprehensions leaked their iteration variables into the enclosing scope, frequently overwriting existing local variables. In Python 3, all comprehensions (list, dict, set, and generator expressions) are compiled into their own private nested function-level scopes (<listcomp>, <dictcomp>, <setcomp>):

x = "GLOBAL_X"
items = [x * 2 for x in range(4)]

print("Comprehension result:", items)  # [0, 2, 4, 6]
print("Value of x after comprehension:", x)  # GLOBAL_X (Preserved!)

The iteration variable x inside the comprehension lives exclusively within the temporary comprehension scope and does not modify the global x.

Summary Comparison of Comprehension Types