Most Databricks cost guides treat every optimization the same
Most Databricks cost guides treat every optimization the same. They’re not.
I’ve watched teams spend weeks tuning query performance while their all-purpose cluster ran 24/7 burning money overnight. Fix the right things first.
Idle compute is where most of the money goes. Not slow queries. Not Delta table bloat. Clusters sitting there, fully provisioned, waiting for someone to run a notebook.
Start here:
- Job clusters over all-purpose for every scheduled pipeline. The cluster starts when the job triggers, terminates when it finishes, and you pay for nothing in between
- Auto-termination on shared all-purpose clusters, 5 to 10 minutes idle max, not 2 hours
- Spot instances with fallback, typically cheaper than on-demand, and Spark handles node loss automatically
Once idle time is under control, look at compute efficiency:
- Autoscaling for pipelines where data volume changes between runs. A fixed cluster size is a guess, autoscaling adjusts to the actual workload
- Photon for heavy SQL and Delta
MERGEoperations. The vectorized engine cuts CPU usage meaningfully there. For general transformation work it rarely justifies the premium - Delta table maintenance matters more than most people think. Uncompacted small files force Spark to open thousands of files per scan, which adds up fast on large tables
Caching is the one thing people overuse. It’s only worth it for data you query repeatedly in the same session. Caching everything is just paying to hold data in memory you’ll never read twice.
The order matters. Idle compute first, then query efficiency, then storage hygiene. Most teams get this backwards and wonder why the bill doesn’t move.