Multiprocessing

When your work is CPU-bound and stays in Python bytecode, threads all share the one GIL and give you nothing. The answer is processes. Each process gets its own Python interpreter, its own GIL, and its own memory — so they genuinely run in parallel across your cores. Nothing is free, of course: starting a process is comparatively slow, and you can only send picklable data across the boundary between them. Seeing that trade clearly is the whole lesson.

When you need processes

The litmus test is the GIL lesson’s table. Reach for processes when the work is pure-Python CPU — tokenising text, parsing strings, custom math in a loop, hashing many small items — and it does not already release the GIL inside a C extension (NumPy, Polars, and Pillow do, so use threads for those), and you actually want more than one core. When unsure, time it: run serial, run with a thread pool, and if threads bought you nothing, you are CPU-bound in pure Python and processes are the tool.

ProcessPoolExecutor — same shape, different worker

concurrent.futures gives you ThreadPoolExecutor and ProcessPoolExecutor with identical APIs, so you can often swap one for the other and measure whether it helps:

from concurrent.futures import ProcessPoolExecutor
import re, time

WORD_RE = re.compile(r"\w+")

def count_words(doc):                       # module-level so it can be pickled
    return sum(1 for _ in WORD_RE.finditer(doc))

docs = [f"the quick brown fox {i} " * 5000 for i in range(8)]

t0 = time.perf_counter()
serial = [count_words(d) for d in docs]
print(f"serial:  {time.perf_counter() - t0:.2f}s")

t0 = time.perf_counter()
with ProcessPoolExecutor(max_workers=4) as pool:
    parallel = list(pool.map(count_words, docs))
print(f"4 procs: {time.perf_counter() - t0:.2f}s")

assert serial == parallel

On a four-core machine, that benchmark shows roughly a 3–4× speedup on real CPU work — minus the genuine cost of spawning processes and shipping data to them. The assert serial == parallel is the reassurance that you parallelised the work without changing the answer.

The pickling gotcha

To hand work to another process, Python must serialise the function and its arguments to bytes, using a format called pickle, so the child can reconstruct them. Everything you send must therefore be picklable — and the classic surprise is that lambdas and locally-defined functions are not, because the child cannot import them by name:

# BAD — a lambda cannot be pickled; this would raise PicklingError on a real pool:
#   pool.map(lambda x: x * x, range(10))

# BAD — a function defined INSIDE another function is also unpicklable:
def make_worker():
    def inner(x):
        return x * x
    return inner

# GOOD — a module-level function has a fully importable name.
def square(x):
    return x * x

print("ok:", square(7))

ok: 49

The rule that falls out: anything you send to a process pool must be defined at module level, in a .py file the child can import. Module-level functions and classes, and the built-in types, all pickle fine. This is also why nearly every multiprocessing script carries an if __name__ == "__main__": guard — on spawn-based platforms the child re-imports your script, and without the guard it would run your top-level pool code too, spawn its own children, and fork-bomb your machine:

# scripts/parallel_word_count.py
from concurrent.futures import ProcessPoolExecutor

def count_words(doc):        # module-level — picklable
    return len(doc.split())

if __name__ == "__main__":   # only the parent runs this
    docs = [...]
    with ProcessPoolExecutor() as pool:
        results = list(pool.map(count_words, docs))

A realistic example — parallel feature extraction

The everyday shape in data work: a large list of inputs, a heavy per-item function, and a wish to use every core. Imagine extracting hand-crafted features from a million chat messages — pure-Python CPU work, exactly what processes are for:

import re

URL_RE = re.compile(r"https?://\S+")
WORD_RE = re.compile(r"\w+")

def extract_features(msg):
    words = WORD_RE.findall(msg)
    return {
        "n_chars": len(msg),
        "n_words": len(words),
        "n_urls": len(URL_RE.findall(msg)),
        "avg_wordlen": round(sum(len(w) for w in words) / max(len(words), 1), 2),
        "upper_ratio": round(sum(1 for c in msg if c.isupper()) / max(len(msg), 1), 3),
    }

corpus = [
    f"Hey check out https://example.com/{i} this is message {i} OK??"
    for i in range(2000)
]

# On a real CPython interpreter you would parallelise it:
#     with ProcessPoolExecutor() as pool:
#         feats = list(pool.map(extract_features, corpus, chunksize=200))
feats = [extract_features(m) for m in corpus]

print("rows:", len(feats))
print("sample:", feats[0])

rows: 2000
sample: {'n_chars': 58, 'n_words': 12, 'n_urls': 1, 'avg_wordlen': 3.58, 'upper_ratio': 0.052}

Two production details are embedded above. The chunksize argument to pool.map matters: each chunk is one inter-process round-trip, so too small and you drown in per-message overhead, too big and you lose load balancing — aim for roughly 10–100 chunks per worker. And the module-level regex compiles (URL_RE, WORD_RE at the top) avoid recompiling on every call and avoid re-shipping the pattern with each task.

Shared memory, briefly

Each process has its own memory, so handing a 10 GB DataFrame to four workers normally copies it four times. There are two escape hatches — multiprocessing.shared_memory (a zero-copy buffer for raw bytes and NumPy arrays) and multiprocessing.Manager (shared lists, dicts, and locks via a slow proxy protocol). But for the overwhelming majority of jobs the right move is simpler: chunk the input small enough that each task’s data is cheap to ship, and aggregate the results in the parent. Reach for shared memory only when profiling proves the inter-process copying is the real bottleneck.

In one breath

• Processes each have their own interpreter and GIL, so pure-Python CPU work runs truly in parallel.
• ProcessPoolExecutor mirrors ThreadPoolExecutor’s API — swap and measure.
• Everything you send must be picklable; define worker functions at module level (no lambdas or closures).
• Guard the entry point with if __name__ == "__main__": or risk a fork bomb on spawn platforms.
• Keep per-task data small (tune chunksize); reach for shared memory only when IPC is proven to dominate.

Practice

What’s next

Threads and processes are both thread-of-execution models. asyncio is the third option — one thread, one process, cooperatively scheduled — and it is the right tool for very high fan-out I/O.