What is lazy evaluation in Spark, and how does it differ from transformations vs actions?

How to think about it

Lazy evaluation means Spark records transformations in a plan instead of immediately computing rows. An action, such as count, show, collect, or write, finally asks Spark to execute the required plan as a distributed job.

The important distinction is this: a transformation describes what should happen; an action asks Spark to make it happen.

Why Spark waits

Imagine processing 1 TB of Parquet event data. A first line filters clicks. The next selects two columns. The next groups by day. If Spark executed eagerly, every line could materialize an intermediate result, write it to disk or memory, and then read it again for the next line.

That would be a great way to spend a lot of money moving data around.

Instead, Spark builds a directed acyclic graph, or DAG, which is a one-way graph of dependent computation with no cycles. Each transformation adds another node or edge to that graph. Spark waits until it knows which result you actually need.

That delay gives Spark a larger optimization window. It can see the complete chain and remove work that does not affect the final answer. For a DataFrame or SQL query, Spark’s Catalyst optimizer analyzes the logical plan, which is the high-level description of the query, and turns it into a physical plan, which specifies the actual operators and execution strategy.

The optimizer may:

• push a filter into the Parquet reader so irrelevant row groups are skipped;
• prune columns so unused data is never read;
• combine compatible operations;
• choose a broadcast join when one side is small enough;
• use runtime statistics through adaptive query execution to adjust parts of the plan while the job runs.

These optimizations are possible because Spark has the whole computation before it starts processing the data.

There can still be small amounts of driver-side work while a plan is being constructed or analyzed. Spark may list files, inspect metadata, or validate parts of an expression. “Lazy” means Spark has not performed the distributed row computation yet. It does not mean the Python process does literally nothing.

A concrete example

Suppose an e-commerce company stores 1 TB of events in Parquet. Each row has 200 columns, but the daily click report needs only event_type, user_id, and event_ts.

from pyspark.sql import functions as F

events = spark.read.parquet("s3://acme-events/")

clicks = (
    events
    .filter(F.col("event_type") == "click")
    .select("user_id", "event_ts")
)

daily = (
    clicks
    .withColumn("day", F.to_date("event_ts"))
    .groupBy("day")
    .count()
)

daily.write.mode("overwrite").parquet(
    "s3://acme-reports/daily-clicks/"
)

The first six statements do not scan all event rows on the cluster.

spark.read.parquet creates a relation describing the files. filter returns a new DataFrame describing the filter. select, withColumn, and groupBy add more description to the plan. The .count() attached to groupBy is easy to misread: it returns another DataFrame and is still a transformation.

The final write is the action. At that point, Spark can build and execute the required plan:

1. Read the Parquet files.
2. Read only the columns needed by the final result, where column pruning is possible.
3. Apply the click filter, with predicate pushdown where the Parquet source and file statistics support it.
4. Derive a day from each timestamp.
5. Shuffle rows by day.
6. Compute the grouped counts.
7. Write the output.

If the input has 2,048 scan partitions and the job is configured with 200 shuffle partitions, the first stage may run roughly 2,048 scan tasks and the aggregation stage may run roughly 200 reduce tasks. The exact numbers depend on file sizes, partitioning, configuration, and adaptive query execution.

The output might contain only 30 rows for a 30-day report. Spark still has to inspect every relevant input row to know those counts. A small output does not imply a small computation.

Transformations versus actions

Operation	Type	What happens immediately
df.filter(...)	Transformation	Returns a new DataFrame plan
df.select(...)	Transformation	Extends the plan; no row scan
df.groupBy(...).count()	Transformation	Builds an aggregate plan
df.show(10)	Action	Runs a job to produce rows
df.count()	Action	Computes the number of rows
df.write.parquet(...)	Action	Executes and materializes the result

A transformation produces another distributed dataset description. For DataFrames, that usually means another DataFrame. For RDDs, transformations include operations such as map, filter, and flatMap.

An action produces a value, sends output somewhere, or performs a side effect. count returns a number. collect returns rows to the driver. show displays rows. write sends results to storage.

Actions do not all process the same amount of data. first or take(10) may stop after finding enough rows for a simple query. count and write normally need all relevant input. A grouped query such as daily.show(10) still needs to complete the aggregation before it knows the first ten groups.

Only the ancestors needed for the requested result execute. Creating an unused DataFrame does not cause its plan to run:

unused = events.filter(F.col("event_type") == "refund")
final = events.select("user_id")

final.count()

The refund filter is not part of final’s lineage, so Spark does not execute it.

What happens at the action

When daily.write(...) runs, the driver submits the query for execution. Spark analyzes the DataFrame plan, applies available optimizations, chooses physical operators, and gives the scheduler a job.

A partition is a chunk of the distributed dataset. Spark normally creates tasks that process partitions. Operations such as filter, select, and withColumn are usually narrow transformations: each output partition can be computed from one input partition without moving records between machines.

groupBy, join, and repartition are commonly wide transformations. They require a shuffle, which is the redistribution of records across the cluster so that related keys meet in the same partition. The shuffle is not performed when you call groupBy; it occurs when an action executes the plan. But the eventual shuffle can still be expensive.

This is why “transformations are free” is the wrong explanation. Transformations are deferred, not free. They describe work that an action may later perform.

The same lazy lineage idea applies to RDDs, although RDD transformations generally do not receive the same schema-aware Catalyst optimizations as DataFrames. A Python map is an arbitrary function from Spark’s point of view, so Spark cannot usually look inside it and infer that a column can be pruned or a predicate can be pushed into a file reader.

The production catch: actions can recompute work

Suppose daily is expensive. This code performs the lineage twice:

daily.count()                         # action one
daily.write.mode("overwrite").parquet(
    "s3://acme-reports/daily-clicks/"
)                                      # action two

Unless the result is cached or otherwise materialized, the second action can reread the source and repeat the shuffle. The first action was only a check; it did not permanently save the result.

The practical symptom appears in the Spark UI: two actions show repeated input scans and repeated shuffle read or write, and the second action takes nearly as long as the first.

If several downstream results reuse the filtered data, persist that reusable intermediate:

clicks = (
    events
    .filter(F.col("event_type") == "click")
    .select("user_id", "event_ts")
    .persist()
)

clicks.count()  # first action materializes the persisted data

daily = (
    clicks
    .withColumn("day", F.to_date("event_ts"))
    .groupBy("day")
    .count()
)

unique_users = clicks.select("user_id").distinct()

daily.write.mode("overwrite").parquet("s3://acme-reports/daily-clicks/")
unique_users.write.mode("overwrite").parquet("s3://acme-reports/click-users/")

persist is not itself an action. It marks the data for storage. The first action computes the required partitions, and later actions can reuse them. Persistence consumes memory or disk, can be evicted, and may make a one-use computation slower. Cache an expensive result that is reused, not every DataFrame that looks important.

For diagnosis, inspect the physical plan with df.explain(mode="formatted") and check the Spark UI. Look for input bytes, shuffle volume, task duration, and skew. A notebook cell that builds ten transformations in one second has not proved the job is cheap. It has only proved that Spark successfully wrote down the recipe.

The senior-level nuance

Lazy evaluation does not guarantee every optimization. A filter may not be pushed into a source if the source cannot apply it. A user-defined function, or UDF, can hide the expression from the optimizer and force Spark to process rows in a less efficient way.

A common symptom is a job reading close to the full 1 TB even though the code selects two columns and filters one event type. The physical plan may show no pushed filters, or a Python UDF operator where a built-in Spark SQL function could have been used. Prefer built-in functions such as F.to_date and F.col when they express the logic; Spark can reason about those expressions.

There is also a reliability reason for lineage. Lineage is Spark’s record of how a partition can be recomputed. If an executor loses a partition, Spark can rerun the relevant upstream transformations instead of requiring a complete copy of every intermediate result. This supports fault tolerance, but it does not make arbitrary side effects safe. A task may be retried, so writing to an external system inside foreach or a transformation can produce duplicates unless the operation is idempotent.

Finally, Spark is not automatically the right tool for every workload. For a single 50 MB file and one simple aggregation, starting a distributed Spark job may cost more time and infrastructure than using a local DataFrame engine. And if a program requires immediate, ordered side effects after every line, a lazy distributed dataflow is the wrong abstraction.

What they’ll ask next

Does groupBy cause a shuffle immediately?
No. groupBy builds a plan. When an action runs, Spark’s physical plan may include a shuffle because rows with the same key must be brought together. The data movement is deferred, not avoided.

What happens if I call two actions on the same DataFrame?
Spark may recompute the shared lineage for each action. Use cache or persist when the intermediate is expensive and reused, then verify in the Spark UI that later actions actually read the cached data.

Does lazy evaluation apply equally to RDDs and DataFrames?
Both are lazy, but DataFrames expose structure that Catalyst can optimize. RDD transformations can still be pipelined and fault-tolerant, but arbitrary Python functions limit query-level optimizations such as column pruning and predicate pushdown.

Say this in the interview

“Spark transformations lazily build a lineage and DAG, while actions trigger the required job; that separation lets Spark optimize the complete DataFrame plan, avoid unused work, and recompute failed partitions, although repeated actions can recompute the plan unless I persist a reused intermediate.”