Data Engineering
3 articles in this topic.
Data Engineering Aug 15, 2026
The small files problem eats your read throughput
A 400 GB table that takes forty minutes to scan is not a 400 GB problem. It is a 1.2-million-file problem, and every one of those files charges the same fixed toll.
11 min read
Read
Data Engineering Aug 15, 2026
The shuffle is the bill
Two Spark jobs read the same data and write the same answer. One costs six times more. The difference is almost always a wide transformation nobody noticed.
12 min read
Read
Data Engineering Jun 2, 2026
When O(n squared) quietly kills your data pipeline
An O(n squared) operation is invisible at a thousand rows and catastrophic at a million — and it almost always reaches production disguised as clean-looking code.
10 min read
Read