8.1 Iterators and the Iteration Protocol

Iterators and the Iteration Protocol

Iteration is one of the foundational paradigms of Python programming. From traversing elements in a list to streaming gigabytes of log entries line by line, Python uses a unified, standard mechanism known as the Iteration Protocol. Understanding the internal mechanics of this protocol—how Python distinguishes between containers that hold data and objects that traverse data—is a core requirement for the PCAP certification.

---

1. Iterables vs. Iterators: The Fundamental Distinction

Many Python programmers use the terms iterable and iterator interchangeably, but in Python's object model, they represent two distinct concepts with completely different responsibilities.

What is an Iterable?

An iterable is an object capable of returning its members one at a time. It serves as a data container or sequence representation. Examples of built-in iterables include:

• Sequences: list, tuple, str, bytes, range
• Non-sequence collections: dict, set, frozenset
• File streams and generator objects

An object is an iterable if it satisfies either of two conditions:

1. It implements the __iter__() method, which returns an iterator object.
2. It implements the legacy sequence protocol method __getitem__() accepting integer indices starting from 0 and raising IndexError when out of bounds.

What is an Iterator?

An iterator is a stateful stream object that produces the next value in a sequence upon demand. It maintains internal state pointing to the current position within the iteration traversal.

An object is an iterator if and only if it implements the Iterator Protocol:

1. __next__(): Returns the next element in the stream, or raises the built-in StopIteration exception when no more elements remain.
2. __iter__(): Returns the iterator object itself (return self).

Concrete Comparison

| Attribute / Behavior | Iterable (e.g., list, str) | Iterator (e.g., list_iterator) | | :--- | :--- | :--- | :--- | | Primary Role | Holds or represents data | Tracks traversal state across data | | Required Methods | __iter__() (or __getitem__()) | __iter__() AND __next__() | | Stateful Traversal | Stateless container | Stateful (consumed as traversed) | | Multiple Passes | Reusable (can iterate many times) | Single-pass (exhausted after traversal) | | Direct next() Call | Raises TypeError ('list' object is not an iterator) | Returns next item or raises StopIteration |

# A list is an iterable, NOT an iterator
numbers = [10, 20, 30]

try:
    next(numbers)
except TypeError as e:
    print(e)  # 'list' object is not an iterator

# Calling iter() on the iterable produces an iterator
num_iter = iter(numbers)
print(type(num_iter))  # <class 'list_iterator'>

print(next(num_iter))  # 10
print(next(num_iter))  # 20
print(next(num_iter))  # 30

try:
    next(num_iter)
except StopIteration:
    print("Iterator is fully exhausted!")

---

2. The Iterable and Iterator Protocols

Python's iteration mechanics rely on object-oriented special methods (dunder methods).

The Iterable Protocol: __iter__()

When Python needs to iterate over an object obj (for example, at the start of a for loop or when calling list(obj)), it invokes obj.__iter__(). This method must return an iterator instance.

class NumberCollection:
    def __init__(self, data):
        self.data = data

    def __iter__(self):
        # Return a new iterator over self.data
        return iter(self.data)

The Iterator Protocol: __next__() and __iter__()

To qualify as an iterator, a class must implement two methods:

1. __next__(self): Advances the iterator state and returns the next item. When the stream is exhausted, it must raise StopIteration.
2. __iter__(self): Returns self. This allows an iterator to be passed directly into loops, list constructors, and any function expecting an iterable.

[!IMPORTANT] Why Iterators Must Implement __iter__() Returning self: In Python, every iterator is also an iterable because it implements __iter__(). This design ensures that functions accepting general iterables (such as for item in x:) can accept both containers (like [1, 2, 3]) and active iterators (like iter([1, 2, 3])) without distinction.

it = iter([1, 2, 3])
print(it is iter(it))  # True: iter() on an iterator returns the iterator itself

---

3. The Built-in iter() and next() Functions

Python provides two built-in functions that form the official API for interacting with the iteration protocol.

The iter() Function

The built-in iter() function can be called in two distinct forms:

Form 1: Single-Argument iter(iterable)

Invokes iterable.__iter__() to obtain an iterator. If __iter__() is not defined, Python attempts to use __getitem__() starting at index 0.

names = ["Alice", "Bob", "Charlie"]
it = iter(names)
print(next(it))  # Alice

Form 2: Two-Argument iter(callable, sentinel)

When called with two arguments, the first argument must be a zero-argument callable (a function, method, or lambda), and the second argument is a sentinel value.

The resulting iterator calls callable() on each iteration and yields the return value. As soon as the callable returns a value equal to sentinel, the iterator raises StopIteration without yielding the sentinel.

import random

# Function that returns a random integer between 1 and 6
def roll_die():
    return random.randint(1, 6)

# Iterate roll_die() until it rolls a 6 (the sentinel)
# Rolls before 6 are produced; the 6 itself stops iteration
die_iterator = iter(roll_die, 6)
for roll in die_iterator:
    print(f"Rolled: {roll}")

This form is frequently used when reading fixed-size chunks from file streams or network sockets until an empty byte string b'' is encountered:

# Read 64-byte blocks until EOF (empty bytes sentinel b'')
# with open('data.bin', 'rb') as f:
#     for chunk in iter(lambda: f.read(64), b''):
#         process(chunk)

The next() Function

The built-in next() advances an iterator and retrieves its next value:

next(iterator[, default])

• Without default: Calls iterator.__next__(). If the iterator raises StopIteration, the exception propagates out.
• With default: If the iterator raises StopIteration, next() intercepts the exception and returns the specified default value instead of raising an error.

items = ["alpha", "beta"]
it = iter(items)

print(next(it, "END"))  # alpha
print(next(it, "END"))  # beta
print(next(it, "END"))  # END (StopIteration was caught internally)
print(next(it, "END"))  # END

---

4. How for Loops Work Under the Hood

In Python, a for loop is syntactic sugar for a while loop that orchestrates iter(), next(), and StopIteration exception handling.

Consider this standard loop:

for item in [10, 20, 30]:
    print(item)

Under the hood, the CPython interpreter executes the following exact operational sequence:

# 1. Obtain an iterator from the iterable
_iterator = iter([10, 20, 30])

# 2. Repeatedly advance the iterator in a loop
while True:
    try:
        item = next(_iterator)
    except StopIteration:
        # 3. Cleanly exit the loop when StopIteration is raised
        break
    
    # 4. Execute the loop body
    print(item)

This desugaring reveals why any custom object implementing __iter__() (or __getitem__()) works seamlessly with for loops, comprehensions, in checks, unpacking, and functions like sum(), min(), and max().

---

5. Designing Custom Iterable and Iterator Classes

There are two primary structural patterns for building custom iteration in Python classes:

Pattern A: Separate Iterable and Iterator Classes (Recommended for Reusability)

Separating the data container (Iterable) from the traversal state (Iterator) ensures that the collection can be iterated multiple times simultaneously without interference.

class CountdownIterator:
    """Iterator tracking countdown traversal state."""
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

class Countdown:
    """Iterable container representing a countdown range."""
    def __init__(self, start):
        self.start = start

    def __iter__(self):
        # Return a brand new iterator on every iteration request
        return CountdownIterator(self.start)

cd = Countdown(3)
print(list(cd))  # [3, 2, 1]
print(list(cd))  # [3, 2, 1] (Reusable! Can iterate again)

Pattern B: Self-Contained Iterator Class (Single-Pass)

When an object is solely an iterator, it implements both __iter__ and __next__ within the same class definition:

class FibonacciIterator:
    """Iterator producing Fibonacci numbers up to a maximum limit."""
    def __init__(self, limit):
        self.limit = limit
        self.a = 0
        self.b = 1
        self.count = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.count >= self.limit:
            raise StopIteration
        value = self.a
        self.a, self.b = self.b, self.a + self.b
        self.count += 1
        return value

fib = FibonacciIterator(5)
for val in fib:
    print(val, end=" ")  # 0 1 1 2 3
print()

---

6. Iterator Exhaustion and Single-Pass Streams

A critical property tested on the PCAP exam is iterator exhaustion.

• Containers (like list, tuple, set, dict) are reusable iterables. Every time iter(container) is called, a new iterator object is instantiated at the beginning of the sequence.
• Iterators (like generator objects, map, filter, zip, enumerate, or custom iterators) are stateful streams. They can only traverse forward and cannot be rewound.

Once an iterator has raised StopIteration, it is exhausted. Passing an exhausted iterator to a subsequent for loop or constructor yields zero elements:

raw_data = [1, 2, 3, 4]

# list_iterator is an iterator
it = iter(raw_data)

# First pass consumes all items
first_pass = list(it)
print("First pass:", first_pass)    # [1, 2, 3, 4]

# Second pass on the SAME iterator finds it already exhausted
second_pass = list(it)
print("Second pass:", second_pass)  # [] (Empty!)

# Built-in functional tools return single-pass iterators
m = map(lambda x: x * 2, [1, 2, 3])
print(1 in m)        # False (m checked 2, 4, 6 and became exhausted)
print(list(m))       # [] (Nothing left!)