Generators

In the last lesson you built an iterator the hard way — two classes, __iter__, __next__, a hand-raised StopIteration. A generator is that same iterator written as an ordinary function with one new keyword, yield. All the protocol plumbing comes for free, and you get something more: lazy evaluation. A generator produces its values one at a time, only when asked, so the full sequence never has to exist in memory at once. That last property is the real prize, and it is what lets a 16 GB laptop process a 100 GB file.

yield, watched closely

The strange and wonderful thing about yield is that it pauses a function mid-execution and resumes it later, with every local variable intact. The print statements below are there so you can see exactly when the body runs:

def count_up_to(n):
    print("  [gen] starting")
    i = 0
    while i < n:
        print(f"  [gen] about to yield {i}")
        yield i             # pause here, hand the value to the caller
        print(f"  [gen] resumed after {i}")
        i += 1
    print("  [gen] done")

g = count_up_to(3)          # NOTHING runs yet — the body is paused at the top
print("calling next:")
print("got", next(g))       # runs until the first yield
print("calling next:")
print("got", next(g))       # resumes, runs to the next yield

calling next:
  [gen] starting
  [gen] about to yield 0
got 0
calling next:
  [gen] resumed after 0
  [gen] about to yield 1
got 1

Read the interleaving carefully, because it is the whole idea. Calling count_up_to(3) printed nothing — the body did not start until the first next(g). Then each next() ran the function up to the next yield, froze it there (note “resumed after 0” appears only on the second call), and handed back the yielded value. A generator function is effectively a coroutine: it lives in suspended animation between calls, remembering exactly where it paused.

Generator expressions — the one-liner

You already met generator expressions briefly with comprehensions. Set beside a list comprehension, the difference is eager versus lazy:

squares_list = [x * x for x in range(10)]
print(type(squares_list).__name__, squares_list)

squares_gen = (x * x for x in range(10))
print(type(squares_gen).__name__)
print(list(squares_gen))

# sum() pulls one value at a time — no list is ever built.
print(sum(x * x for x in range(10)))

list [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
generator
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
285

The bracketed version built and stored the whole list; the parenthesised version built a generator that produced nothing until list(...) pulled the values through. That last line is the point: sum consumed the squares one at a time. Run the very same sum(x * x for x in range(10_000_000)) and it still holds only a single number at any instant — whereas the bracketed [x * x for x in range(10_000_000)] would allocate all ten million at once.

The memory difference, pictured

The contrast is worth seeing directly. A list keeps every value it will ever produce; a generator keeps only a recipe and its current position, so its size does not grow with the data at all:

Streaming pipelines — generators feeding generators

This is the pattern that makes generators indispensable. Each stage is a generator that reads from the stage before it, so data flows through the whole chain one record at a time and memory never holds more than a single record — no matter how large the source:

from io import StringIO

# Stand in for a huge CSV with StringIO — its API is identical to an open file.
raw = StringIO('''id,user,amount,status
1,aarav,120.50,ok
2,priya,9.99,FAILED
3,diego,250.00,ok
4,sofia,15.00,failed
5,liam,88.10,OK
''')

# Stage 1: yield raw lines, dropping the header.
def read_lines(f):
    next(f)                 # skip the header row
    for line in f:
        yield line.rstrip("\n")

# Stage 2: parse each line into a dict.
def parse(lines):
    for line in lines:
        id_, user, amt, status = line.split(",")
        yield {"id": int(id_), "user": user,
               "amount": float(amt), "status": status.lower()}

# Stage 3: keep only successful transactions over $50.
def filter_high_value(records):
    for r in records:
        if r["status"] == "ok" and r["amount"] > 50:
            yield r

# Wiring it up runs nothing yet — it just connects the stages.
pipeline = filter_high_value(parse(read_lines(raw)))

# Pulling from the pipeline drives the whole chain, one record at a time.
for record in pipeline:
    print(record)

{'id': 1, 'user': 'aarav', 'amount': 120.5, 'status': 'ok'}
{'id': 3, 'user': 'diego', 'amount': 250.0, 'status': 'ok'}
{'id': 5, 'user': 'liam', 'amount': 88.1, 'status': 'ok'}

Three records survived: the two FAILED/failed rows were filtered out, and liam’s OK was lower-cased to ok before the test. The crucial detail is when the work happens — building pipeline ran nothing; only the for loop pulled a record through all three stages, then the next, then the next. Swap StringIO for open("transactions.csv") and this identical code streams a terabyte with memory flat at one record.

yield from — delegating to another generator

When one generator wants to re-yield everything another generator produces, yield from does it without an explicit loop:

def odds(n):
    for i in range(n):
        if i % 2:
            yield i

def evens(n):
    for i in range(n):
        if i % 2 == 0:
            yield i

def both(n):
    yield from evens(n)     # yield every value evens(n) produces
    yield from odds(n)      # then every value odds(n) produces

print(list(both(6)))

[0, 2, 4, 1, 3, 5]

yield from g is, to a first approximation, for x in g: yield x — but shorter, and it also forwards send() and exceptions to the inner generator correctly, which matters in the rare cases you reach for those.

When NOT to use a generator

A generator is the wrong tool when you need any of the things a one-shot lazy stream cannot give you: its length up front (there is no len() on a generator), random access by index (g[5] does not work), or to iterate the same data more than once (a generator is one-shot — once drained, list(g) returns [], and there is no rewind). For all of those, a list is the right answer.

The rule of thumb is clean: if the data fits comfortably in memory and you will touch it more than once, use a list; if it is huge, or you only ever stream through it once, use a generator.

In one breath

• yield turns a function into a generator — an iterator with the boilerplate removed.
• Calling a generator function runs nothing; the body advances one yield per next() and freezes between.
• A generator holds a recipe plus its position — constant memory, regardless of how many values it yields.
• Chain generators into a pipeline to stream data larger than RAM, one record at a time.
• Generators are one-shot and have no len() or indexing — use a list when you need those.

Practice

What’s next

You can already write lazy pipelines. Decorators let you wrap any of those stages with retry, caching, or timing — without touching their code.