8.2 Generator Functions and Generator Expressions

Generator Functions and Generator Expressions

Writing custom iterator classes with explicit __iter__() and __next__() methods requires manual state management, tracking index pointers, and explicit exception handling. Python introduces generators as a concise, elegant, and memory-efficient way to implement iterators. Generators allow developers to write iterative streams using standard procedural function syntax, leveraging the interpreter's ability to suspend and resume execution frames.

---

1. Generator Functions and the yield Keyword

A generator function is any Python function that contains at least one yield statement in its body.

The Fundamental Difference: return vs yield

• return: Terminates function execution completely, destroys the local stack frame, and returns a single value to the caller.
• yield: Produces a value to the caller and suspends the function's execution state, freezing all local variables and the instruction pointer. The next time the generator is requested for a value, execution resumes immediately after that yield statement.

def simple_generator():
    print("--> Starting execution")
    yield "First"
    print("--> Resuming after first yield")
    yield "Second"
    print("--> Resuming after second yield")
    yield "Third"
    print("--> Function finished")

Instantiation vs Execution

A major concept tested on the PCAP exam is what happens when a generator function is called:

[!IMPORTANT] Calling a generator function does NOT execute any code inside the function body. Instead, it immediately instantiates and returns a generator object (<class 'generator'>). The code inside the function body begins executing only when next() is called on the generator object for the first time.

# Calling the function DOES NOT print '--> Starting execution'
gen = simple_generator()
print(type(gen))  # <class 'generator'>

# Code executes up to the first yield
val1 = next(gen)  # Prints: '--> Starting execution'
print("Received:", val1)  # Received: First

# Code resumes from previous yield up to second yield
val2 = next(gen)  # Prints: '--> Resuming after first yield'
print("Received:", val2)  # Received: Second

---

2. Execution Lifecycle, Frame Suspension, and Resumption

To understand how generators operate internally, consider the lifecycle of a generator object:

Created (Suspended at start) ──▶ next() ──▶ Running ──▶ yield ──▶ Suspended
                                              │
                                              ▼
                                  return / End of Body
                                              │
                                              ▼
                                      Closed / Exhausted
                                   (Raises StopIteration)

State Preservation

When a generator yields, Python retains its entire stack frame in heap memory:

1. Local Variables: All local bindings retain their exact values across suspensions.
2. Instruction Pointer: Python marks the precise bytecode instruction following the yield statement.
3. Active Control Flow: Active loops (for, while), conditionals (if/else), and context managers remain paused in mid-execution.

def count_accumulator(step):
    total = 0
    current = 1
    while current <= 3:
        total += (current * step)
        yield total
        current += 1

acc = count_accumulator(10)
print(next(acc))  # total = 10, current = 1 -> yields 10
print(next(acc))  # resumes: current becomes 2, total = 10 + 20 = 30 -> yields 30
print(next(acc))  # resumes: current becomes 3, total = 30 + 30 = 60 -> yields 60

---

3. Generator Termination and the return Statement

A generator terminates when its execution flow reaches the end of the function body or encounters a return statement. In either case, the generator raises a StopIteration exception.

Return Values in Generators (Python 3)

In Python 3, a generator function can include a return value statement. When executed:

1. The generator raises StopIteration.
2. The returned value is stored in the .value attribute of the raised StopIteration exception instance.

def countdown_with_msg(n):
    while n > 0:
        yield n
        n -= 1
    return "Blastoff!"

gen = countdown_with_msg(2)
print(next(gen))  # 2
print(next(gen))  # 1

try:
    next(gen)
except StopIteration as e:
    print("StopIteration caught! Return value:", e.value)
    # Prints: StopIteration caught! Return value: Blastoff!

[!NOTE] When iterating over a generator using a standard for loop, the for loop automatically intercepts StopIteration and silently discards the e.value return payload.

---

4. Generator Expressions: Syntax, Laziness, and Comprehension Comparison

Python provides a high-level shorthand syntax for creating generators on the fly: generator expressions (often abbreviated as genexps).

Syntax

A generator expression uses the exact same syntax as a list comprehension, but is enclosed in parentheses (...) instead of square brackets [...]:

(expression for item in iterable if condition)

# List comprehension: Eagerly evaluates and creates a list in memory
list_comp = [x ** 2 for x in range(5)]
print(list_comp)        # [0, 1, 4, 9, 16]
print(type(list_comp))  # <class 'list'>

# Generator expression: Lazily evaluates on demand
gen_exp = (x ** 2 for x in range(5))
print(gen_exp)          # <generator object <genexpr> at 0x...>
print(type(gen_exp))    # <class 'generator'>
print(next(gen_exp))    # 0
print(next(gen_exp))    # 1

Parentheses Elision in Function Calls

When a generator expression is passed as the sole positional argument to a function, the enclosing parentheses of the generator expression may be omitted for cleaner syntax:

# Explicit double parentheses
total1 = sum((x ** 2 for x in range(10)))

# Idiomatic single parentheses (outer parentheses serve function call)
total2 = sum(x ** 2 for x in range(10))
print(total1, total2)  # 285 285

Eager vs Lazy Evaluation: Memory Benchmarking

The fundamental architectural advantage of generator expressions is lazy evaluation:

import sys

# 1,000,000 integers in a list comprehension
list_data = [x for x in range(1_000_000)]
print("List memory:", sys.getsizeof(list_data), "bytes")
# ~8,448,728 bytes (~8.4 MB)

# 1,000,000 integers in a generator expression
gen_data = (x for x in range(1_000_000))
print("Generator memory:", sys.getsizeof(gen_data), "bytes")
# ~208 bytes (Constant footprint regardless of sequence length!)

---

5. Infinite Streams, Pipelines, and Data Processing

Because generators produce items on demand without allocating collections in memory, they can represent mathematically infinite sequences.

Infinite Number Generator

def infinite_fibonacci():
    """Generates an infinite stream of Fibonacci numbers."""
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

fib_gen = infinite_fibonacci()

# Extract first 6 Fibonacci numbers
first_six = [next(fib_gen) for _ in range(6)]
print(first_six)  # [0, 1, 1, 2, 3, 5]

Generator Pipelines

Generators can be chained together to construct multi-stage data processing pipelines, similar to Unix shell pipes (cat | grep | awk). Each stage processes items lazily one by one:

def read_numbers(limit):
    for n in range(limit):
        yield n

def filter_evens(numbers):
    for n in numbers:
        if n % 2 == 0:
            yield n

def square_numbers(numbers):
    for n in numbers:
        yield n ** 2

# Pipeline: Source -> Filter -> Transform
raw_stream = read_numbers(10)
even_stream = filter_evens(raw_stream)
squared_stream = square_numbers(even_stream)

print(list(squared_stream))  # [0, 4, 16, 36, 64]

---

6. Re-entrancy, Single-Pass Nature, and Common Traps

On the PCAP exam, several common pitfalls related to generator mechanics frequently appear:

Trap 1: The Consumed Generator Trap

Just like iterators, generators are single-pass. Once consumed, they cannot be iterated again:

gen = (x for x in range(3))
print("Sum 1:", sum(gen))  # Sum 1: 3 (0 + 1 + 2)
print("Sum 2:", sum(gen))  # Sum 2: 0 (Already exhausted!)

Trap 2: Indexing and len() on Generators

Generators do not support random access or length inquiry. Attempting to use index brackets [] or len() raises a TypeError:

gen = (x * 10 for x in range(5))

try:
    print(len(gen))
except TypeError as e:
    print("Error:", e)  # object of type 'generator' has no len()

try:
    print(gen[0])
except TypeError as e:
    print("Error:", e)  # 'generator' object is not subscriptable

Trap 3: Lazy Side Effects

Expressions or function calls inside a generator expression are not evaluated until the generator is stepped with next():

def transform(x):
    print(f"Processing {x}")
    return x * 2

# Creating the generator DOES NOT call transform()
gen = (transform(x) for x in [1, 2, 3])
print("Generator created")

# Side effects occur only as items are consumed
print(next(gen))  # Prints 'Processing 1', then 2