HDFS — blocks, NameNode, replication

The last lesson gave the storage layer a single paragraph and a promise: that we’d see how a filesystem takes one enormous file, scatters it across hundreds of unreliable machines, and still hands it back whole when boxes are dying. That promise is this lesson — and the answer turns out to be three ideas so simple that every storage system since, S3 included, quietly copied them.

HDFS solves one problem: store a file too big for any single disk. The trick is to chop the file into chunks, scatter the chunks across many machines, and keep a redundant copy of each chunk so a dead machine doesn’t lose your data.

It’s a beautifully simple idea. Almost every distributed storage system after HDFS borrowed it.

The picture

• A 384MB file is split into 3 blocks of 128MB each (B1, B2, B3)
• Each block is replicated 3 times across DataNodes (default replication=3)
• The NameNode remembers: “file /data/clicks.csv = blocks [B1, B2, B3]; B1 is on DataNodes 1+2+5; B2 is on 2+3+7; …”
• The actual bytes never touch the NameNode

If DataNode 1 dies, B1 and B3 still have 2 surviving replicas. HDFS notices and asks a healthy node to make a new copy. No data loss.

Blocks — and why 128MB

The default block size is 128MB. (Older clusters: 64MB. Some tuned clusters: 256MB or 512MB.) Why so large?

• Sequential read throughput. Hard disks are ~150MB/s sequential but only ~5-10MB/s random. Big blocks mean fewer seeks.
• NameNode memory. Every block adds ~150 bytes of metadata in the NameNode’s RAM. A 1PB cluster with 64MB blocks needs ~16M block entries; with 128MB blocks, only ~8M.
• Task locality. Spark/MapReduce schedule one task per block. Too small = too many tasks (overhead). Too big = stragglers.

Compare this to a normal filesystem block (4KB on most Linux setups). HDFS chose a block 32,000x larger because it was optimized for streaming reads of huge files, not random access to small ones.

NameNode vs DataNode

The split of responsibilities matters:

Component	What it holds	What happens if it dies
NameNode	All filesystem metadata in RAM	The whole cluster is offline until you fail over to a standby NameNode
DataNode	Actual block bytes on local disk	HDFS re-replicates the missing blocks from surviving replicas

The NameNode is the single point of failure of classic HDFS. Real clusters run two NameNodes (active + standby) sharing edits through a quorum of JournalNodes. This was the source of many production outages in the 2010s.

Why does the NameNode keep metadata in RAM? Because every file operation hits it, and a disk lookup per operation would be too slow. The trade-off is that the NameNode’s RAM is a hard limit on cluster file count — usually around 100-500M files.

Replication and rack awareness

The default replication factor (number of identical copies of each block stored on separate nodes) is 3. Three is the sweet spot: 2 copies can’t survive a simultaneous rack failure, and 4+ copies double the write amplification for modest extra durability. HDFS doesn’t just put 3 copies on random nodes — it uses rack awareness to spread replicas across failure domains:

• 1st replica on the local node (or a random one if the writer isn’t a DataNode)
• 2nd replica on a different rack
• 3rd replica on the same rack as the 2nd, but a different node

The idea: a whole rack can lose power (network switch dies, rack PDU trips) and you still have at least one surviving copy. This is the same idea AWS uses with availability zones — different physical locations on the same metro network.

You’d configure it in hdfs-site.xml:

<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>

Higher replication = more durability + more cost. Some teams use 2 for cold archive, 3 for warm, and erasure coding (effective ~1.4x overhead) for very large cold data.

A toy implementation

The mechanics of “split, hash, place” fit in a tiny Python sketch:

# A tiny HDFS-shaped storage model.
import hashlib

class TinyHDFS:
    def __init__(self, datanodes, block_size=128, replication=3):
        self.datanodes = {d: {} for d in datanodes}   # node -> {block_id: bytes}
        self.namenode = {}                            # file -> [(block_id, [nodes])]
        self.block_size = block_size
        self.replication = replication

    def _pick_nodes(self, block_id):
        # Hash the block id, then pick the next N nodes in a ring.
        nodes = sorted(self.datanodes)
        h = int(hashlib.md5(block_id.encode()).hexdigest(), 16)
        start = h % len(nodes)
        return [nodes[(start + i) % len(nodes)] for i in range(self.replication)]

    def write(self, path, data):
        blocks = []
        for i in range(0, len(data), self.block_size):
            block_id = f"{path}#{i // self.block_size}"
            chunk = data[i : i + self.block_size]
            nodes = self._pick_nodes(block_id)
            for n in nodes:
                self.datanodes[n][block_id] = chunk
            blocks.append((block_id, nodes))
        self.namenode[path] = blocks

fs = TinyHDFS(datanodes=["n1", "n2", "n3", "n4", "n5"])
fs.write("/logs/today.csv", "a" * 350)   # 350 bytes, 128-byte blocks -> 3 blocks

for path, blocks in fs.namenode.items():
    print(path)
    for bid, nodes in blocks:
        print(f"  {bid:30s} -> {nodes}")

/logs/today.csv
  /logs/today.csv#0              -> ['n3', 'n4', 'n5']
  /logs/today.csv#1              -> ['n1', 'n2', 'n3']
  /logs/today.csv#2              -> ['n1', 'n2', 'n3']

There’s the whole filesystem in eight lines of output. The 350-byte file split into three blocks (#0, #1, #2), and each block was copied to three of the five nodes — that’s the NameNode’s entire job, recording that #0 lives on n3/n4/n5 while the bytes themselves sit out on the DataNodes. Now read it as a survival story: kill node n3 and every block still has at least two living copies elsewhere, so nothing is lost; real HDFS simply notices the under-replicated block and asks a healthy node to make a fresh third copy. Real HDFS adds rack awareness, lease management, append support, and a thousand other details — but the core is exactly this: split, hash to nodes, replicate.

How HDFS maps to cloud storage today

HDFS is increasingly replaced by object storage (S3, GCS, ABS). The mental model translates well:

HDFS concept	S3 equivalent
File path	Object key (the /path/to/file.csv part)
Block	Multi-part upload chunk (5-100MB typically)
NameNode metadata	S3 list API (slower, has consistency caveats)
Replication	Built-in multi-AZ redundancy (AWS does not publish the copy count, but guarantees 11 nines of durability)
Rack awareness	Cross-AZ replication

The big difference: S3 has no NameNode. That’s both a feature (infinite scale, no single point of failure) and a problem (listing millions of files in a prefix is slow). The modern solution is a table format — Iceberg, Delta, Hudi — that stores its own metadata index in object files, replacing the NameNode’s role.

In one breath

HDFS stores a file too big for one disk by splitting it into large (128 MB) blocks, replicating each block to 3 machines spread across racks for fault tolerance, and keeping a single NameNode index in RAM that maps files to blocks to DataNodes (the bytes never touch the NameNode) — so a dead DataNode loses nothing and HDFS quietly re-replicates, while the NameNode is the single point of failure and its RAM caps the cluster’s file count (which is why HDFS hates many small files); the cloud replaced HDFS with object stores like S3 that have no NameNode, so table formats (Iceberg, Delta, Hudi) now play the metadata-index role.

Practice

Before the quiz, reason about the two design choices that look strange until you see why. First, the 128 MB block — 32,000× a normal 4 KB filesystem block: give the two reasons such a huge block is better for HDFS’s workload, and what breaks if you store ten million 1 KB files instead. Second, the NameNode keeps all metadata in RAM, never on disk for reads — explain the speed reason for that and the hard limit it imposes on the cluster.

Quick check

0/3

Q1Why is the default HDFS block size 128MB instead of 4KB?

HDFS can't handle small blocksLarge blocks reduce NameNode memory pressure and favor sequential disk throughputIt's a Hadoop tradition with no real reasonLarger blocks compress better

Check answer

Q2A DataNode crashes. What happens to its data?

Lost permanentlyRecovered automatically from other replicas; HDFS makes new copies elsewhereRecovered only if you restart the NameNodeRestored from the NameNode's backup

Check answer

Q3In a cloud-native data lake on S3, what plays the role of HDFS's NameNode?

S3 itself — it has a built-in NameNodeA table format layer like Apache Iceberg or Delta Lake (or Hive Metastore)KubernetesThere's no equivalent — listings are always fast

Check answer

A question to carry forward

That closes the storage half of the foundation. Between this lesson and the last, you now know where big data lives — scattered as replicated blocks across a cluster, indexed by a NameNode, or sitting in S3 behind a table format. The data is safe, durable, and distributed. It is also completely inert: a billion rows on a thousand disks that do nothing until something reads and computes over them.

That something is, of course, “Spark” — except that word is doing an enormous amount of hiding. Ask three engineers what they mean by “we use Spark” and you’ll get the batch SQL job, the streaming pipeline, and the ML feature build, running on three different platforms with three different table formats. So before we open the engine and watch a job actually execute, the question to carry forward is one of orientation: when someone says “Spark,” what are all the things they might mean — the engine, the modules inside it, the language you write, the table format, the platform — and how do they fit together? That map is the Spark ecosystem, and it’s the next lesson, the one that closes this background chapter before we go inside the machine.