Iterators

You have written hundreds of for loops, and each one quietly leaned on a protocol you may never have seen. Every for loop in Python is sugar over exactly two methods — __iter__ and __next__ — and once you can see that machinery, a great deal of Python’s “magic” turns into the same small trick repeating. Files, dictionaries, strings, generators, zip, range: every one of them is just an object that answers those two methods. Let us pull the loop apart and watch it work.

The for-loop, unwrapped

Here is what for x in nums: actually does, written out by hand:

nums = [10, 20, 30]

# What the for-loop really does, step by step:
it = iter(nums)            # ask nums for an iterator
while True:
    try:
        x = next(it)       # ask the iterator for the next value
    except StopIteration:  # the iterator says "I'm done"
        break
    print(x)

10
20
30

Two built-ins and one exception are the whole story. iter(obj) calls obj.__iter__() and hands back an iterator. next(it) calls it.__next__() to pull the next value. And when there is nothing left, __next__ raises StopIteration — a special signal exception meaning “no more values” — which the for loop quietly catches to end the loop. That is the entire protocol, drawn out below:

Iterable versus iterator — the distinction that trips people

This is the part worth slowing down on, because the two words sound interchangeable and are not. An iterable is anything you can call iter() on — a list, tuple, dict, string, file, or generator. An iterator is the thing iter() gives back: a stateful, one-shot object that remembers where it is and produces the next value on demand. A list is iterable; it is not itself an iterator.

nums = [1, 2, 3]

# A list is iterable but is NOT an iterator.
print(hasattr(nums, "__iter__"))   # yes
print(hasattr(nums, "__next__"))   # no — you cannot call next() on a list

it = iter(nums)                    # now we hold an iterator
print(hasattr(it, "__next__"))     # yes
print(next(it), next(it))          # pulls 1, then 2

# An iterator is one-shot: it remembers it already gave out 1 and 2.
for x in it:
    print("remaining:", x)

for x in it:                       # second loop: nothing left
    print("again:", x)
print("exhausted")

True
False
True
1 2
remaining: 3
exhausted

Read the tail of that output carefully. After next(it) twice and one for loop, the iterator is spent — the second for loop prints nothing at all. Contrast that with a plain list, which you can loop over a thousand times, because each for calls iter() afresh and gets a brand-new iterator. This is exactly why a zip(...) or map(...) result seems to “vanish” if you iterate it twice: those are iterators, and one pass exhausts them.

Everything is iterable

Once you see the protocol, the variety collapses. Strings yield characters, dictionaries yield keys, files yield lines — all the same two methods:

for c in "abc":
    print(c)

user = {"name": "Aarav", "age": 28}
for k in user:
    print(k, "->", user[k])

# enumerate and zip are themselves iterators layered over other iterables.
names = ["Aarav", "Priya", "Diego"]
scores = [82, 91, 77]
for i, (name, score) in enumerate(zip(names, scores), start=1):
    print(f"{i}. {name}: {score}")

a
b
c
name -> Aarav
age -> 28
1. Aarav: 82
2. Priya: 91
3. Diego: 77

Build your own — a paginated API client

Real APIs hand back data a page at a time, and a good client hides that completely: the caller should just write for record in client: and never think about page boundaries. The iterator protocol is exactly how you provide that:

class PaginatedAPI:
    """A pretend API that returns records in pages of 3."""
    def __init__(self, total):
        self.total = total

    def __iter__(self):
        # Hand back a FRESH cursor each time someone loops over us.
        return _APICursor(self.total)

class _APICursor:
    def __init__(self, total):
        self.total = total
        self.idx = 0
        self.page = []

    def _fetch_next_page(self):
        if self.idx >= self.total:
            return []
        start = self.idx
        page = [{"id": i, "value": i * 10}
                for i in range(start, min(start + 3, self.total))]
        self.idx += len(page)
        print(f"  [HTTP] fetched page starting at id={start}")
        return page

    def __iter__(self):
        return self

    def __next__(self):
        if not self.page:
            self.page = self._fetch_next_page()
        if not self.page:
            raise StopIteration
        return self.page.pop(0)

api = PaginatedAPI(total=7)
for record in api:
    print("got:", record)

  [HTTP] fetched page starting at id=0
got: {'id': 0, 'value': 0}
got: {'id': 1, 'value': 10}
got: {'id': 2, 'value': 20}
  [HTTP] fetched page starting at id=3
got: {'id': 3, 'value': 30}
got: {'id': 4, 'value': 40}
got: {'id': 5, 'value': 50}
  [HTTP] fetched page starting at id=6
got: {'id': 6, 'value': 60}

Trace the [HTTP] lines and the design reveals itself: a new page is fetched only when the current one runs dry, so seven records arrive across three lazy fetches — the caller’s for loop never knew pages existed. Notice the deliberate two-class split: the iterable PaginatedAPI returns a fresh _APICursor each time it is looped, while the cursor is the actual iterator. If you instead put __next__ directly on PaginatedAPI and returned self from __iter__, a second loop over the same object would silently see nothing — the same one-shot exhaustion you saw with zip.

In one breath

• A for loop is iter() once, then next() until StopIteration.
• An iterable answers __iter__; an iterator also answers __next__ and is one-shot.
• Each for over a list gets a fresh iterator; zip/map/enumerate are iterators and exhaust after one pass.
• Build a reusable iterable by returning a fresh iterator object from __iter__, not self.
• Iterators stream — they are how you process data larger than memory.

Practice

What’s next

A generator is an iterator with all the boilerplate stripped away — one yield replaces the entire two-class machine you just wrote. That is generators, next.