Writing
Notes on what actually breaks in production. Spark skew and shuffles, Databricks cost, Delta and Iceberg maintenance, and the reliability gaps nobody sees until a deadline slips.
Your Iceberg metadata directory is growing faster than your data
Every commit writes a snapshot and nothing cleans them up by default. Streaming ingest can leave tens of thousands of files in one directory.
Read →You can be productive in Spark for months before you understand what it is doing
I showed a mentee a query plan for the first time. He asked what an Exchange was. Forty minutes later we were still on that one question.
Read →Your Delta table is Iceberg-compatible. Your readers just don't know it yet
Iceberg compatibility on a Delta table is not instant. The metadata sync lags, and downstream readers can silently consume a stale snapshot.
Read →Spark is silently killing your join performance
The default broadcast threshold is 10MB. Your dimension tables are bigger, so Spark falls back to a shuffle join and your five-minute job runs for forty.
Read →A job that completes is not a job that is healthy
A Spark job crept from 1 hour to 3. No alerts, no errors. The story of silent degradation and what to track so it does not happen to you.
Read →UDF is the most expensive three letters in your PySpark job
Every row crosses the JVM-Python boundary twice. On a billion-row table that overhead doesn't add minutes, it adds hours. Here is the fix order.
Read →Most Databricks cost guides treat every optimization the same
Idle compute is where most of the money goes, not slow queries. The order you fix things in decides whether the bill actually moves.
Read →Snowflake vs Databricks is a hiring decision, not a data decision
The feature matrix everyone compares is the wrong lens. What you are really choosing is which problem your team is equipped to carry.
Read →