datarekha
Section 4 chapters · 18 lessons

PySpark

Understand the cluster, the DAG, the Catalyst optimizer, and the shuffle that ate your job. Learn PySpark the way platform engineers actually run it.

  1. 01

    Big Data Background

    4 lessons
  2. 01 What 'big data' actually means Beginner 5 min
  3. 02 Hadoop overview Beginner 7 min
  4. 03 HDFS architecture Beginner 7 min
  5. 04 The Spark ecosystem Beginner 5 min
  6. 02

    Spark Architecture

    4 lessons
  7. 05 Driver & executors Intermediate 7 min
  8. 06 Jobs → stages → tasks Intermediate 7 min
  9. 07 DAG & lazy evaluation Intermediate 6 min
  10. 08 Shuffles Advanced 8 min
  11. 03

    DataFrames

    5 lessons
  12. 09 DataFrame intro Beginner 6 min
  13. 10 Schemas Intermediate 6 min
  14. 11 Joins in Spark Intermediate 8 min
  15. 12 Window functions Advanced 7 min
  16. 13 Pandas UDFs Advanced 7 min
  17. 04

    Internals & Optimization

    5 lessons
  18. 14 Catalyst optimizer Advanced 8 min
  19. 15 Reading execution plans Advanced 7 min
  20. 16 Adaptive Query Execution Advanced 6 min
  21. 17 Partitioning Advanced 7 min
  22. 18 Skew & salting Advanced 7 min
Skip to content