Data Engineering

3articles in this topic.

Data EngineeringAug 15, 2026

The small files problem eats your read throughput

A 400 GB table that takes forty minutes to scan is not a 400 GB problem. It is a 1.2-million-file problem, and every one of those files charges the same fixed toll.

11 min read Read

Data EngineeringAug 15, 2026

The shuffle is the bill

Two Spark jobs read the same data and write the same answer. One costs six times more. The difference is almost always a wide transformation nobody noticed.

12 min read Read

Data EngineeringJun 2, 2026

When O(n squared) quietly kills your data pipeline

An O(n squared) operation is invisible at a thousand rows and catastrophic at a million — and it almost always reaches production disguised as clean-looking code.

10 min read Read