Hadoop — the three pieces you should know

The last lesson left us with a thousand separate machines and a pile of hard questions: where does a file bigger than any single disk live? When a box dies mid-job, how is the data not lost? Who hands out the work? We said none of this was invented by Spark — Spark inherited it — and promised to meet the system that first made a thousand commodity boxes behave like one giant, fault-tolerant computer. This is that system.

Hadoop is the system that made “big data” cheap. Before Hadoop (2006-ish), processing terabytes meant buying a custom data warehouse from a vendor. After Hadoop, you rented commodity boxes and ran open-source software. That’s the entire revolution in one sentence.

You’ll rarely deploy raw Hadoop today, but the three pieces show up everywhere — and Spark inherits most of them.

The three pieces

These three were the original “Hadoop stack.” Vendor distributions (Cloudera, Hortonworks, MapR) bundled them plus Hive, Pig, HBase, Oozie, and a dozen others. Most of those bundles are dead. The mental model is not.

HDFS in 30 seconds

HDFS splits a big file into fixed-size blocks (usually 128MB), replicates each block to 3 different machines, and tracks where everything lives in a central NameNode. The actual machines storing blocks are DataNodes.

If a DataNode dies, no data is lost — the other 2 replicas survive, and HDFS quietly re-replicates the missing block.

You’ll learn HDFS in detail in the next lesson. For now, the one-line takeaway: HDFS made petabyte-scale storage cheap by using commodity disks plus replication, instead of expensive SAN hardware.

MapReduce — the original distributed compute

MapReduce is a programming model (a constrained way of expressing a computation so that any cluster can run it automatically in parallel) with two phases:

1. Map — apply a function to every record, emit (key, value) pairs
2. Shuffle — group all values with the same key onto the same machine
3. Reduce — aggregate the values for each key

That’s it. Word count is the “hello world”:

# MapReduce, in 10 lines of pure Python.
from collections import defaultdict

documents = [
    "spark is fast spark is in memory",
    "hadoop is on disk mapreduce is slow",
    "spark replaced mapreduce for most jobs",
]

# MAP — emit (word, 1) for every word in every doc
mapped = []
for doc in documents:
    for word in doc.split():
        mapped.append((word, 1))

# SHUFFLE — group by key (in real Hadoop, this is the expensive part)
shuffled = defaultdict(list)
for key, value in mapped:
    shuffled[key].append(value)

# REDUCE — sum the values per key
reduced = {key: sum(values) for key, values in shuffled.items()}

for word, count in sorted(reduced.items(), key=lambda x: -x[1])[:5]:
    print(f"{count:>3}  {word}")

  4  is
  3  spark
  2  mapreduce
  1  fast
  1  in

The shape is the lesson, not the word counts. Map flattened three documents into a stream of (word, 1) pairs without caring about order; shuffle gathered every pair with the same key onto one pile; reduce summed each pile — is to 4, spark to 3, mapreduce to 2. Notice that Map never had to see the whole dataset at once, and neither did Reduce — each worked on a slice. That is the entire reason the model parallelizes: scatter the Map across a thousand machines, let the shuffle route matching keys together, and run Reduce on the gathered piles. The expensive, network-bound step hiding in the middle is the shuffle — remember that word; it will haunt the rest of this section.

That’s the model that ran billions of jobs from 2006-2014 and powered the early data warehouses at Yahoo, Facebook, and Google. It works. It just writes everything to disk between Map and Reduce — because MapReduce has no concept of keeping data in RAM across stages, every intermediate result is durably written to HDFS before the next stage can start reading it. If your job needs three MapReduce stages, it writes everything to disk three times.

Why Spark replaced MapReduce

Spark’s pitch was simple: keep intermediate data in RAM between stages, instead of dumping it to HDFS every time. For iterative algorithms (gradient descent, PageRank, anything that loops 20+ times), that change made Spark 10-100x faster.

The trade-off: Spark needs more memory per node. In 2014 that was a real constraint. Today RAM is cheap and the trade-off is obvious.

Two other Spark wins:

• Higher-level API. Writing a 3-stage Spark job in PySpark is ~10 lines. The same job in raw Hadoop MapReduce was 200+ lines of Java.
• Lazy DAG optimization. Spark sees the whole job before running and can rearrange it (predicate pushdown, column pruning). MapReduce ran each stage blindly.

By 2018 nobody started a new project on raw MapReduce. It’s now a legacy maintenance burden.

YARN — the cluster scheduler

YARN’s job is to answer: “I have 100 machines and 50 jobs. Which job gets which CPUs and memory?” It’s a resource scheduler.

You ask YARN for “10 containers, each with 4 cores and 16GB RAM,” and YARN finds 10 machines with capacity, launches your processes there, and tracks them.

YARN has now been largely replaced by Kubernetes for the same job. Databricks, EMR Serverless, and Dataproc all run on Kubernetes under the hood. But the mental model is identical — your Spark driver asks the cluster manager for resources, and gets containers back.

Where you’ll actually see Hadoop in a job

• Legacy on-prem stacks at banks, telcos, governments — large Cloudera/Hortonworks clusters still run nightly batch ETL
• Migration projects — moving HDFS + Hive + MapReduce to S3 + Iceberg + Spark/Trino is a multi-year, well-paid line of work
• Job descriptions — “Hadoop experience” usually means “you know HDFS layout, can read a YARN log, and have written MapReduce or Spark”
• Architecture interviews — explaining why Spark replaced MapReduce is a standard question

In one breath

Hadoop made big data cheap by replacing vendor warehouses with commodity boxes running open-source software, in three pieces: HDFS stores files split into replicated 128 MB blocks across many machines (a NameNode indexes where each block lives, DataNodes hold the bytes), MapReduce is the map→shuffle→reduce programming model that runs a job in parallel over that data, and YARN schedules which machines get which CPUs and RAM — and the cloud has since replaced all three (S3 for HDFS, Spark for MapReduce because it keeps intermediate data in RAM instead of writing to disk between every stage, Kubernetes for YARN), while the one thing that survives is the partition-plus-parallel-compute mental model itself.

Practice

Before the quiz, reason about the change that made Spark win. A job runs three MapReduce stages back to back. The lesson says MapReduce writes everything to disk three times — explain exactly when and why those writes happen, and what Spark does differently to turn 10 disk round-trips into RAM hand-offs. Then map the three Hadoop pieces onto their modern cloud replacements, and name the single mental model that none of those replacements threw away.

A question to carry forward

We just toured all three pieces at a jog, and twice did the same thing: gave HDFS a single paragraph and a promise that “you’ll learn it in detail in the next lesson.” There’s a reason it earns the deep dive and the others don’t. MapReduce and YARN are about computing — and the cloud swapped them out wholesale for Spark and Kubernetes. But the storage layer is more fundamental: before any engine can process data across a thousand machines, the data has to survive across a thousand machines first — disks that fail, files larger than any one disk, blocks that must never be lost.

So the question to carry forward is the foundation everything else in this section reads from: how does a filesystem take one enormous file, scatter it in pieces across hundreds of unreliable machines, and still hand it back to you whole even when boxes are dying — and why is its design (blocks, replication, a NameNode index) the template that S3 and every modern object store quietly copied? That is HDFS architecture, and it is the next lesson.