What I work on
Engagements start from a symptom, not a retainer. You tell me what is slow, what is expensive, or what you are afraid to migrate. I find the real cause and fix it. If it is not something I can help with, I say so.
Apache Spark performance tuning
Most Spark jobs are slow for a handful of repeatable reasons. I read the query plan, find the shuffle or skew that is actually costing you, and fix it without throwing more hardware at the problem.
- Skew and shuffle pathologies after joins and aggregations
- Broadcast join thresholds and partition sizing
- Killing expensive Python UDFs and cache misuse
- Reading query plans to find the number that actually matters
Databricks cost optimization
I tear down the bill line by line, fix idle compute first, then query efficiency, then storage hygiene. Most teams do this in the wrong order and wonder why the number never moves.
- Idle compute and all-purpose clusters running around the clock
- Job clusters, auto-termination, and spot instances with fallback
- Photon and autoscaling where they pay for themselves, not everywhere
- Delta table maintenance and small-file cleanup
Platform migrations
The hardest part of a migration isn't moving the data, it's proving to the business that the numbers didn't change. I lead migrations with validation first and design for the rebuild after this one.
- Migrations onto Snowflake and Databricks
- Validation-first approach so the numbers provably match
- The boring patterns that survive the next platform change
- De-risking the cutover before it reaches production
Platform architecture and reliability
A job that completes is not a job that is healthy. I design for observability and durability so a pipeline degrading quietly gets caught by a baseline, not by an angry stakeholder.
- Data contracts, SLAs, and pipeline trust
- Delta and Iceberg maintenance, retention, and metadata hygiene
- Catching silent degradation before a missed deadline finds it
- Boring, durable design over clever, fragile design
How engagements usually go
- 1
Diagnose
A short, paid look at the query plans, the bill, or the migration plan. You get a clear read on where the money and the time actually go.
- 2
Fix
I do the work or pair with your team to do it. Config changes, refactors, validation harnesses, whatever the diagnosis pointed at.
- 3
Hand back
You keep the patterns and the reasoning, not just the fix. The point is that the next engineer can read the plan too.
Not sure which one you have?
Most teams describe a symptom, not a root cause. Tell me what you are seeing and I will tell you which of these it actually is.