The Spark ecosystem map

The last lesson left the data safe and distributed — and completely inert. A billion rows scattered as replicated blocks across a cluster do nothing at all until something reads and computes over them. That something, we said, is “Spark” — but that one word hides a whole zoo, and before we can open the engine and watch a job run, we have to know what “Spark” even refers to. This lesson is that map.

A new analyst joins your team and asks, “what do we use Spark for?” You say “everything.” They ask which Spark, and you realise: the nightly batch job on EMR is Spark SQL on Parquet, the Kafka clicks pipeline is Structured Streaming writing Delta on S3, the recommender job on Databricks is MLlib feature engineering feeding into PyTorch on a separate GPU box. Same word, four different stacks.

When someone says “we use Spark,” they could mean a dozen different things. There’s the engine (one Apache project), the language API you write code in (Scala/Python/Java/R/SQL), the table format that manages your data (Delta, Iceberg, plain Parquet), and the platform that runs it all (Databricks, EMR, Dataproc, self-managed).

Let’s map the territory.

Inside Apache Spark

Spark itself is one project with five built-in modules. You’ll use some daily, others rarely.

Module	What it does	Use it for
Spark Core	The base engine — RDDs, task scheduling, shuffles	Almost never write Core directly anymore
Spark SQL	DataFrames, Catalyst optimizer, ANSI SQL parser	95% of jobs you’ll write
Structured Streaming	Streaming as “infinite DataFrames”	Real-time pipelines, CDC, IoT
MLlib	Distributed ML algorithms	Large-scale training, mostly being replaced
GraphX	Distributed graph processing	Rare; mostly replaced by GraphFrames or external tools

You’ll spend ~95% of your time in Spark SQL (which gives you the DataFrame API). Streaming is the second-most-common, and the rest are niche.

Spark Core — the engine you don’t write to

Spark Core is the original API: RDDs (Resilient Distributed Datasets — fault-tolerant collections spread across executors). It’s a low-level interface where you write transformations directly on those distributed collections.

# RDD-era code — you'll see this in legacy codebases
rdd = sc.textFile("s3://bucket/logs/")
counts = (rdd.flatMap(lambda line: line.split())
             .map(lambda word: (word, 1))
             .reduceByKey(lambda a, b: a + b))

Almost nobody writes new code at this level since Spark 2.0. The DataFrame API is faster and easier to read: because DataFrames expose named columns and typed operations, Catalyst can see what the code does and rearrange it — filter early, skip unused columns, pick the right join strategy. RDD lambdas are opaque Python functions; the optimizer can’t look inside them. RDDs survive for two reasons: legacy code, and the occasional case where you need a transformation Spark SQL doesn’t support.

Spark SQL — the one you’ll live in

Spark SQL is the high-level API: DataFrames, Datasets, and ANSI SQL. Everything you write in PySpark uses it under the hood.

# DataFrame API — what 95% of your PySpark code looks like
from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.getOrCreate()
orders = spark.read.parquet("s3://bucket/orders/")
top = (orders.filter(F.col("status") == "PAID")
             .groupBy("country")
             .agg(F.sum("amount").alias("revenue"))
             .orderBy(F.desc("revenue")))
top.show()

You can write the same thing as SQL — Spark SQL parses ANSI-ish SQL and produces the same physical plan:

orders.createOrReplaceTempView("orders")
top = spark.sql("""
  SELECT country, SUM(amount) AS revenue
  FROM orders
  WHERE status = 'PAID'
  GROUP BY country
  ORDER BY revenue DESC
""")

DataFrames and SQL are equivalent. Pick whichever reads better for the team — most production codebases mix both.

Two engine features you get for free in Spark 3.x and don’t have to configure:

• Catalyst rewrites your plan (predicate pushdown, column pruning, join reordering).
• Adaptive Query Execution (AQE) is on by default and re-plans at runtime — coalescing tiny shuffle partitions, switching sort-merge to broadcast when one side turns out to be small, and splitting skewed partitions automatically.

Structured Streaming — Spark SQL for unbounded data

Structured Streaming reframes streaming as “a DataFrame that keeps growing.” The same code you’d write for batch works for stream, with two changes: readStream instead of read, and a sink that writes incrementally.

# A Kafka -> S3 streaming pipeline
events = (spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("subscribe", "clicks")
    .load())

(events.selectExpr("CAST(value AS STRING) AS json")
       .writeStream
       .format("parquet")
       .option("path", "s3://bucket/clicks/")
       .option("checkpointLocation", "s3://bucket/ckpt/")
       .trigger(processingTime="1 minute")
       .start())

It runs as micro-batches (default trigger: as fast as possible; or fixed intervals like every 1 minute). For sub-second latency, look at Continuous Processing mode (experimental) or use Flink.

MLlib and GraphX — declining

MLlib ships with distributed implementations of classification, regression, clustering, ALS for recommendations, and a Pipelines API modeled on scikit-learn. It’s fine, but two trends pulled it sideways:

• Most ML training fits on one big GPU box, not a Spark cluster
• For models that do need distributed training, Horovod, Ray, or PyTorch DDP are the dominant choices

MLlib still has a place for feature engineering at scale and for legacy pipelines, but new ML work rarely starts there.

GraphX is even more niche. If you’re doing graph analytics, you’ll probably use GraphFrames (a DataFrame-based companion) or jump to specialized engines like Neo4j or KuzuDB.

The PySpark API

Spark is written in Scala (on the JVM). It exposes the same DataFrame API in Scala, Java, Python (PySpark), R, and SQL.

PySpark is by far the most popular for data engineering and analytics. Under the hood, your Python code drives a JVM Spark process via Py4J — but you rarely notice, except in two cases:

1. UDFs in Python can be slow (serialization across the JVM ↔ Python boundary). Use Pandas UDFs (covered later) to fix this.
2. Error messages sometimes have Scala stack traces. Read past them to find the Python line.

The wider ecosystem

Spark doesn’t exist in isolation. Here’s how it relates to the other names you’ll see in a modern data stack:

Tool	Role	Relationship to Spark
Delta Lake	Table format on data lakes	Adds ACID transactions, time travel, schema evolution to Parquet on S3 — Spark’s native partner
Apache Iceberg	Open table format	Vendor-neutral alternative to Delta; same idea, broader engine support (Trino, Flink, DuckDB read it)
Apache Hudi	Table format with strong upserts	Best fit for streaming CDC use cases
Databricks	Managed Spark platform	The company founded by Spark’s creators; commercial cloud product
AWS EMR / GCP Dataproc	Managed Spark on cloud	Cheaper than Databricks for batch, fewer features
DBT	SQL-first transformation framework	”T” of ELT; orchestrates Spark SQL (or Snowflake/BigQuery) queries
Airflow / Dagster / Prefect	Workflow orchestrators	Schedule Spark jobs as part of a larger DAG

A typical stack: Kafka → S3 (raw) → Spark/Delta (bronze/silver/gold) → DBT for marts → Snowflake/Trino for BI. Spark is the workhorse middle layer.

In one breath

“Spark” is really four stacked questions: the engine (Apache Spark, one project with five modules — Core for RDDs/scheduling/shuffle, Spark SQL for DataFrames where you’ll spend ~95% of your time, Structured Streaming for unbounded data as growing DataFrames, and the declining MLlib and GraphX), the language binding you write (PySpark, Scala, Java, R, SQL — all the same DataFrame API), the table format managing your data (Delta, Iceberg, Hudi — ACID and time travel over Parquet), and the platform running it (Databricks with its Photon engine, EMR, Dataproc, self-managed); nobody writes RDDs anymore because Catalyst can only optimize the DataFrame API, not opaque lambdas.

Practice

Before the quiz, untangle the opening scenario: “we use Spark” turned out to mean a batch SQL job, a streaming pipeline, and an ML feature build on three platforms. For each of those, name the Spark module doing the work. Then the distinction the lesson calls the rookie mistake: a Databricks tutorial’s code runs 2–3× faster than the identical code on your self-managed EMR cluster, byte-for-byte the same PySpark — what is different, and why is “Spark” and “Databricks” not the same thing?

Quick check

0/3

Q1Which Spark module powers the DataFrame API you use day-to-day?

Spark CoreSpark SQLStructured StreamingMLlib

Check answer

Q2Why do most teams not write new code against RDDs anymore?

RDDs were deprecatedCatalyst can optimize DataFrames but not RDDs, so DataFrames are usually faster — and the API is much cleanerRDDs only support JavaRDDs don't support shuffles

Check answer

Q3What does Delta Lake add on top of Parquet files in S3?

CompressionACID transactions, schema evolution, and time travelFaster reads in pandasEncryption at rest

Check answer

A question to carry forward

That completes the background. You know what big data is, where it lives, and what “Spark” actually names. But notice that every layer in this lesson’s map was described from the outside — as a box on a diagram, a module you call, a platform you pick. The one box we kept pointing at and walking past was Spark Core: “the engine — task scheduling, shuffles.” We called it the part you never write to, and moved on. That box is where the real understanding lives.

Because here is the thing the whole map quietly assumed: when you call orders.groupBy("country").agg(...) on a billion rows, something turns that one line into work that runs across a thousand machines, survives a node dying, and shuffles matching keys together — and none of that happens by magic. So the question to carry forward, out of the map and into the engine, is the one this background chapter was built to earn: when a Spark job runs, what are the actual processes executing it across the cluster, who’s in charge, and how does the work get divided among them? That is the driver-and-executors architecture, and it opens the next chapter — the one where we finally go inside the machine.