Jobs, stages, and tasks

The last lesson ended on a verb it never explained: the driver schedules tasks. We counted those tasks — 200 partitions, 80 slots, three waves — as if they simply existed, waiting to be handed out. But you wrote six lines of DataFrame code, not 200 tasks; something turned one into the other, and decided how many stages and where to wedge the network reshuffle. We asked what that translation actually is. This lesson is the hierarchy it produces — the same one you read in the Spark UI when a job is slow.

“The pipeline took 47 minutes last night, can you find why?” The Spark UI has 8 jobs, each with 4 stages, each with 200 tasks — about 6,400 boxes to look at. The wrong move is to scroll through all of them. The right move is to know the hierarchy well enough to ask the UI a question: which Job took the longest, which Stage inside that Job was the bottleneck, and is the slow Stage skewed or just slow?

The Spark UI shows three nested concepts: Jobs, Stages, and Tasks. They’re not interchangeable jargon — they’re three real layers of how Spark executes work. Once you know the rule that creates each layer, the UI suddenly makes sense.

The hierarchy

Job        = one action (count, show, write, collect)
Stage      = a chunk of the Job between two shuffles
Task       = one partition's worth of work in a stage

Or in words you’d use at the whiteboard:

• Every action triggers a Job. No action = no execution.
• Each shuffle splits a Job into a new Stage. Stages contain consecutive narrow transformations.
• Each partition becomes one Task within a stage. Tasks are the units of work scheduled onto executor cores.

Walking through an example

Consider this DataFrame pipeline:

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.getOrCreate()

orders   = spark.read.parquet("s3://bucket/orders/")   # narrow
users    = spark.read.parquet("s3://bucket/users/")    # narrow
filt     = orders.filter(F.col("status") == "PAID")    # narrow
joined   = filt.join(users, "user_id")                 # WIDE — shuffle
agg      = joined.groupBy("country").count()           # WIDE — shuffle
agg.write.parquet("s3://bucket/out/")                  # ACTION — triggers Job

What happens when .write.parquet(...) runs:

1. The driver builds the logical plan for the chain above.
2. The action triggers one Job.
3. Spark splits the Job into stages at every shuffle:
   • Stage 0: read orders + filter (narrow chain)
   • Stage 1: read users (narrow chain, runs in parallel with Stage 0)
   • Stage 2: join’s shuffle-read + the rest of the join (depends on 0 + 1)
   • Stage 3: groupBy’s shuffle-read + write (depends on Stage 2)
4. Each stage runs one task per partition. If orders has 100 partitions and users has 50, Stage 0 has 100 tasks and Stage 1 has 50 tasks. After the shuffle, Spark’s default partition count is 200 (configurable via spark.sql.shuffle.partitions), so Stages 2 and 3 each have 200 tasks.

That’s a lot of moving parts for 6 lines of code. The Spark UI shows them all.

Narrow vs wide — the rule that creates stages

A narrow transformation can be computed on one partition without looking at any other partition:

• filter, select, withColumn, where
• map, flatMap, mapPartitions
• Most casts and arithmetic

A wide transformation needs data from many partitions to compute each output partition — which means data has to shuffle across the network:

• groupBy, agg, reduceByKey
• join (unless it’s a broadcast join — covered later)
• repartition, distinct, orderBy (global)
• window functions over partitions

Stage boundary = shuffle. That’s the only rule. The reason: a shuffle requires every upstream task to finish and write its output before any downstream task can read it — so Spark must checkpoint between stages. Within a stage, narrow transformations pipeline freely with no barrier.

df = spark.read.parquet("s3://bucket/orders/")
filt = df.filter(F.col("status") == "PAID")
result = filt.select("user_id", "amount")
result.write.parquet("s3://bucket/out/")    # ACTION

Tasks — one per partition

A task is the smallest unit Spark schedules. Each task processes one partition of data. The task count for a stage equals the partition count entering that stage.

orders.rdd.getNumPartitions()    # how many tasks Stage 0 will have

Tasks are scheduled onto cores of executors. If you have 10 executors × 4 cores = 40 parallel slots, and a stage has 200 tasks, the stage runs in roughly 200 / 40 = 5 waves.

If you see a stage with 1 task taking 10 minutes while all others finished in 30 seconds, that’s data skew — one partition is huge. You’ll learn to fix this in the skew lesson.

Reading the Spark UI

The Spark UI’s main page is a list of Jobs. Click into one and you see its Stages. Click a Stage and you see its Tasks (in a sortable table or as a visualization).

The two screens you’ll use most:

• Jobs — was this job slow? How long? Which stages dominated?
• Stages → Event Timeline — are tasks balanced or is one a straggler? Are tasks even running, or are executors idle?

A typical investigation flow:

1. “My job took 30 minutes.” → Open the Job, find which Stage was 25 of those minutes.
2. Open that Stage’s Task table. Sort by duration descending.
3. If one task dominates → skew. If all tasks take the same long time → underpowered cluster or expensive logic. If there are thousands of tiny tasks → too many partitions, increase partition size.

A toy DAG, in pure Python

To make the hierarchy concrete, here’s the same Job-Stage-Task split as a Python sketch. No Spark — just the mental model.

# Toy model: how Spark decides job/stage/task structure.
class Op:
    def __init__(self, name, kind, parents=()):
        self.name = name
        self.kind = kind   # 'narrow' or 'wide'
        self.parents = parents

def stages_of(action_op):
    # Walk the DAG from the action back to sources.
    # New stage every time we cross a 'wide' op.
    stages, current = [], []
    def walk(op):
        nonlocal current
        if op.kind == "wide":
            stages.append(current[::-1])
            current = []
        current.append(op.name)
        for p in op.parents:
            walk(p)
    walk(action_op)
    stages.append(current[::-1])
    return [s for s in stages if s][::-1]

read_o = Op("read orders", "narrow")
filt   = Op("filter PAID", "narrow", parents=[read_o])
read_u = Op("read users",  "narrow")
join   = Op("join",        "wide",   parents=[filt, read_u])
agg    = Op("groupBy",     "wide",   parents=[join])
write  = Op("write",       "narrow", parents=[agg])

for i, stage in enumerate(stages_of(write)):
    print(f"Stage {i}: {' -> '.join(stage)}")

Stage 0: read users -> read orders -> filter PAID -> join
Stage 1: groupBy
Stage 2: write

The one rule worth taking away is visible right there: the boundaries fall exactly at the two wide ops. join ends one stage, groupBy ends the next — every narrow op (the reads, the filter) pipelines freely inside a stage, and only a shuffle forces a wall. (This toy is a deliberate skeleton, so its grouping is rougher than the diagram above: it sweeps the two source reads into one stage and yields three stages rather than Spark’s four. Spark’s real planner separates the parallel reads, splits each shuffle into a write side and a read side, and chooses broadcast vs. sort-merge — but the rule that creates a boundary, “cross a wide op → new stage,” is exactly this.)

Where the Spark UI lives

When you run pyspark, the UI is at http://localhost:4040. On Databricks, it’s the Spark UI tab in the cluster page. On EMR, it’s behind the History Server. Memorize how to find it — you’ll be in it often.

In one breath

A line of PySpark becomes three nested layers: an action (count, show, write, collect) triggers one Job; the Job splits into Stages at every shuffle — the single rule, where a wide transformation (groupBy, join, orderBy) needs data from many partitions and forces a network reshuffle, while consecutive narrow ops (filter, select, map) pipeline inside one stage; and each Stage runs one Task per partition, scheduled onto executor cores (parallelism = executors × cores). Reading the Spark UI is just walking this hierarchy down — slowest Job → slowest Stage → the Task table, where one task dwarfing the rest means skew.

Practice

Before the quiz, label the example pipeline yourself: for read → filter → join → groupBy → write, mark each step narrow or wide, and from that alone predict how many shuffle boundaries (and therefore roughly how many stages) the Job has. Then the debugging reflex the lesson drills: a stage shows 99 tasks done in 10 s and task #100 still running at 15 min — name the diagnosis in one word, and the exact Spark-UI screen and sort that would confirm it.

A question to carry forward

Read back over this lesson and one phrase keeps doing quiet, load-bearing work: nothing runs until an action. The whole rookie-mistake callout turned on it — a notebook cell defines a ten-step DataFrame chain in 50 milliseconds, then the next cell’s .write takes 20 minutes, because the transformations never executed; they only built a plan. We leaned on that fact to explain Jobs, but we never asked why Spark works this way at all.

And it is a genuinely strange design. Most code you’ve ever written runs line by line, top to bottom, the instant you call it. Spark deliberately refuses — it lets you stack filter, join, groupBy without computing a single row, hoarding the operations until you finally force its hand. That is not laziness for its own sake; it is the source of Spark’s speed, the thing that lets it see your entire pipeline before running a step and rewrite it for the better. So the question to carry forward is: why does Spark wait — what does it gain by being lazy and holding the whole DAG until an action, and how does that deferral turn into optimization? That is the DAG and lazy evaluation, and it is the next lesson.