The mistake I made first: setting grace periods as a fraction of the interval instead of p95 runtime plus jitter. Got paged at 3 AM for jobs that were 40 seconds late, then started ignoring the alerts — which defeats the whole point.