A job that completes is not a job that is healthy

Our Spark job went from 1 hour to 3 hours. No alerts. No errors.

Someone complained about a missed deadline, that’s how we found out.

The culprit: a custom decorator on every transformation method that called .count() after each step to log row counts.

Spark is lazy by design. It builds an execution plan and runs it only when an action is triggered.

Our decorator triggered a full materialization after every single step. As data grew, so did the cost. The job still succeeded, so nobody noticed.

This is the part most teams miss. A job that completes is not a job that’s healthy. Without tracking duration over time, you have no baseline. Without a baseline, you have no signal.

Reliability is not just about failures. It’s about silent degradation you never see coming.

A few things worth tracking on every pipeline:

• Job duration trend over time
• Alert thresholds based on a historical baseline, not fixed values

How does your team monitor pipeline health beyond success and failure status?