Driver and executors

The background chapter kept pointing at one box and walking past it — “Spark Core: the engine, task scheduling, shuffles” — and promised that when a job runs, something turns one line of DataFrame code into work spread across a thousand machines. This chapter opens that box. We asked the first question: what are the actual processes executing a job, and who is in charge? The answer is three actors, and confusing them is the difference between a five-minute fix and a five-hour one.

A junior engineer’s first PagerDuty page: “Spark job OOM, ETA?” They SSH in, see a giant red error, and copy-paste it into the team chat: “is this the driver or an executor?” If you can’t answer that question in five seconds, you can’t fix the bug. Three actors run every Spark job — and knowing who did what is the difference between debugging in minutes and debugging in hours.

The three actors

The driver

The driver is your code — the Python (or Scala) program that calls SparkSession, builds DataFrames, and triggers actions.

from pyspark.sql import SparkSession

# This runs in the driver
spark = SparkSession.builder.appName("revenue").getOrCreate()

orders = spark.read.parquet("s3://bucket/orders/")    # builds plan only
top = orders.groupBy("country").count()               # builds plan only

top.show()                                            # NOW driver coordinates execution

The driver:

• Holds the SparkSession and the logical plan (the DAG — directed acyclic graph — of transformations)
• Asks the cluster manager for executors
• Schedules tasks onto executor cores
• Collects results back when you call .collect(), .show(), .toPandas(), etc.

Crucially, the driver is a single process on one machine. It is not distributed. Everything that returns to your Python session (.collect(), the result of .show(), the size of a list comprehension on a DataFrame’s rows) flows through the driver.

The executors

Executors are the worker processes that actually run your filters, joins, and aggregations.

Each executor:

• Is a separate JVM process on some worker machine
• Has a fixed number of cores (concurrent tasks) and a memory budget
• Reads data from storage (S3, HDFS, Kafka)
• Holds partitions of your DataFrames in memory while it works on them
• Reports task results back to the driver

A common executor config:

spark = (SparkSession.builder
    .config("spark.executor.instances", "20")        # 20 executors
    .config("spark.executor.cores", "4")             # 4 tasks each at once
    .config("spark.executor.memory", "16g")          # 16 GB JVM heap each
    .getOrCreate())

This cluster has 20 executors × 4 cores = 80 tasks in parallel. If your DataFrame has 200 partitions, the work runs in roughly 200 / 80 = 3 waves.

The cluster manager

The cluster manager just finds machines. You ask for 20 executors with 4 cores and 16GB each; the cluster manager looks at the pool of available machines and launches the JVM processes there.

Four cluster managers exist:

• YARN — classic Hadoop-era choice, still common on-prem
• Kubernetes — the modern default for cloud-native Spark
• Mesos — deprecated; you may see it in legacy stacks
• Standalone — Spark’s own simple scheduler; fine for tests or small clusters

You usually don’t interact with the cluster manager directly. You set a few config values (spark.master=yarn, spark.kubernetes.container.image=...) and it just works.

SparkSession — the handle

SparkSession is the single entrypoint to everything: reading data, running SQL, accessing configuration. It exists only in the driver.

spark = SparkSession.builder \
    .appName("revenue") \
    .config("spark.sql.shuffle.partitions", "200") \
    .getOrCreate()

# Driver: orchestrates
df = spark.read.parquet("s3://bucket/orders/")

# Executors: do the work when an action is called
df.groupBy("country").count().show()

Older code uses SparkContext (sc) instead — it’s still accessible as spark.sparkContext and is needed for the RDD API. For DataFrame work, you can ignore it.

Two OOMs you’ll hit (and how to tell them apart)

The single most common production failure is out-of-memory. There are two flavors, and they need opposite fixes.

Driver OOM

You wrote df.collect() or df.toPandas() on a billion-row DataFrame. All those rows now have to fit in your driver’s RAM. They don’t. Boom.

# Driver OOM landmine
big_df = spark.read.parquet("s3://bucket/clicks/")
rows = big_df.collect()                              # pulls EVERYTHING to driver
df_pd = big_df.toPandas()                            # same problem

Symptoms:

• Stack trace starts in the driver, mentions OutOfMemoryError
• Spark UI shows tasks finished, driver dies last

Fix: never collect a big DataFrame. Use .show(), .limit().collect(), or write to storage and read from there.

Executor OOM

One partition is much bigger than the others (data skew), or your group-by produces a single key with millions of values. That executor’s RAM can’t hold it.

# Executor OOM landmine
clicks.groupBy("country").agg(F.collect_list("event_id")).show()
# If India has 200M events, that one executor task tries to hold
# them all in a list. OOM.

Symptoms:

• Stack trace shows the failing task on an executor
• Spark UI shows one task taking 100x longer than the others, then dying
• Often labeled “ExecutorLostFailure” or “Container killed by YARN/k8s”

Fix: more executor memory, or fix the skew (covered in the skew lesson).

In one breath

Every Spark job runs on three actors: the driver is your program (one process, one machine — it holds the SparkSession, builds the DAG, asks for resources, schedules tasks, and collects results), the executors are the worker JVMs that actually run filters/joins/aggregations on partitions held in their memory across cores (so parallelism = executors × cores), and the cluster manager (YARN/Kubernetes/Standalone) just finds machines and launches them; the two OOMs you’ll hit are opposite bugs — driver OOM from pulling a huge result back with .collect()/.toPandas() (no amount of cluster scale-up helps), and executor OOM from a skewed partition or a giant per-key aggregate.

Practice

Before the quiz, become the on-call engineer. A red stack trace arrives. Describe the one thing you’d look for to decide in five seconds whether it’s a driver OOM or an executor OOM — and for each, the opposite fix. Then the math that catches people: a cluster of 10 executors × 4 cores runs a stage of 200 partitions — how many tasks run at once, how many waves, and why does adding a 1000th executor do nothing for a job that ends in .collect() on a billion rows?

A question to carry forward

Notice the verb that did all the work in this lesson and was never explained: the driver schedules tasks onto executor cores. We counted those tasks — 200 partitions, 80 slots, 3 waves — as if “tasks” were just sitting there waiting to be handed out. But where do they come from? You wrote six lines of DataFrame code, not 200 tasks. Something took orders.filter(...).join(users).groupBy("country").count() and decided it becomes this many tasks, in this many batches, with a network reshuffle wedged in the middle.

That translation — from a line of code to the units of work the driver schedules — is its own hierarchy, and it is what you actually read in the Spark UI when a job is slow. So the question to carry forward is: when the driver turns your DataFrame chain into work, what are the layers it creates — the Job, the Stages, the Tasks — and what rule decides where one stage ends and the next begins? That is jobs, stages, and tasks, and it is the next lesson.