What is the difference between a generator and a list, and when should you prefer a generator?

How to think about it

A list is a finished container — every value computed and sitting in memory. A generator is a recipe — it computes the next value only when something asks for it. That single difference in when the work happens is the whole question, and the memory gap it opens is dramatic: a list of ten million squares costs tens of megabytes, while the equivalent generator costs about a hundred bytes, because at any instant it is holding one number, not ten million.

The price for that thrift is that a generator is single-pass. Walk it once and it’s spent — no second loop, no indexing. If you need the values more than once, you either keep a list or rebuild the generator.

The syntax is one character apart

A list comprehension builds everything now; swap the [] for () and you get a lazy generator expression instead:

import sys

big_list = [x * x for x in range(10_000_000)]
print(sys.getsizeof(big_list))   # tens of MB — all ten million stored

big_gen = (x * x for x in range(10_000_000))
print(sys.getsizeof(big_gen))    # ~100 bytes — nothing computed yet

A generator function uses yield to suspend and resume, which is what lets you stream a file far larger than memory:

def read_chunks(filepath, size=4096):
    with open(filepath, "rb") as f:
        while chunk := f.read(size):
            yield chunk

for chunk in read_chunks("dataset.bin"):
    process(chunk)   # only one chunk in memory at a time

A worked example

Three things at once: the memory gap on a small list, lazy production value-by-value, and the single-pass trap that bites everyone:

import sys

# Memory: same 1000 values, wildly different footprints
small_list = [x * x for x in range(1000)]
small_gen  = (x * x for x in range(1000))
print(f"list size : {sys.getsizeof(small_list):,} bytes")
print(f"gen size  : {sys.getsizeof(small_gen):,} bytes")

# Lazy: a generator hands over one value per next()
def count_up(limit):
    n = 0
    while n < limit:
        yield n
        n += 1

gen = count_up(5)
print()
print("Pulling values one at a time:")
print(" next():", next(gen))
print(" next():", next(gen))
print(" rest  :", list(gen))         # drain whatever's left

# Single-pass: a drained generator is empty forever after
gen2 = (x for x in range(5))
print()
print("First pass :", list(gen2))
print("Second pass:", list(gen2))     # already exhausted

# Pipeline: each stage is a generator, the whole chain is constant-memory
def evens(n):     return (x for x in range(n) if x % 2 == 0)
def squares(seq): return (x * x for x in seq)

print()
print("Pipeline result:", list(squares(evens(10))))

list size : 8,856 bytes
gen size  : 112 bytes

Pulling values one at a time:
 next(): 0
 next(): 1
 rest  : [2, 3, 4]

First pass : [0, 1, 2, 3, 4]
Second pass: []

Pipeline result: [0, 4, 16, 36, 64]

The numbers tell the story: a thousand stored integers take 8,856 bytes, the generator that can produce the same thousand takes 112, because it stores a recipe, not results. And look at the second pass — []. The generator wasn’t reset; it was consumed.

They compose into pipelines

The real payoff is composability. Stack generators and the whole chain runs in constant memory, with the final consumer — here sum — pulling every stage one value at a time:

lines   = (line.strip() for line in open("log.txt"))
records = (line.split(",") for line in lines if line)
values  = (float(r[2]) for r in records)
total   = sum(values)   # the entire pipeline runs in constant memory

The idea underneath

Every for loop, every sum, list, and join, drives a sequence by calling next() over and over. A generator simply chooses to do one value’s worth of work per call instead of all of it upfront. That’s the entire model; yield is just Python’s way of letting a function pause and pick up where it left off.