22 cron jobs, one denied permission, and the alert that never came

A job can execute on schedule, exit with an error, and still look healthy to anything that only checks whether it ran. That gap is where outages hide the longest. Here is the one-day loop that closed it: real per-job heartbeats, alerts in the channel people already read, and a Slack command loop that talks back.

A macOS system update quietly revoked Full Disk Access from a Python interpreter. Nothing announced this. No dialog, no log line an operator would think to check. The interpreter just started failing every filesystem call it used to make cleanly, on every job that used it.

The jobs kept "running." launchd fired them on schedule, and a shallow health check, the kind that asks only "did the process execute," would have shown green the entire time. What actually happened is one specific job, a daily activity digest, exited with an error and produced nothing, while the file it should have written kept an old, unrelated modification time that looked fresh enough to pass a lazy staleness check. The job was blind for roughly 26 hours before anyone noticed, and the thing that caught it wasn't monitoring. It was a human asking why a digest looked thin.

That is the failure mode worth naming: a system can be down in a way that looks up. Anything that checks "did it run" instead of "did it produce a real result" will miss it every time.

Two Pythons, one permission grant

The fix took longer than it should have for an unglamorous reason: macOS ships more than one Python. The system CommandLineTools Python and the one a package manager installs are different binaries, each needing its own Full Disk Access grant in System Settings. Revoking or re-granting one silently leaves the other broken, and a job might call either depending on how its shebang resolves. Getting the fleet clean meant clearing stale entries for both binaries and re-granting each explicitly, then re-running every affected job to confirm a clean exit rather than trusting the checkbox.

That part was a permissions fix. The more durable part was making sure the next silent failure, whatever causes it, can't stay silent for 26 hours again.

Heartbeats, not just error counts

Error tracking answers "did something throw an exception." It does not answer "did the 6 a.m. job run at all." Those are different questions, and a cron fleet needs the second one answered independently of the first.

The fix was turning on per-job heartbeat monitoring, one monitor slug per scheduled job, so a job that fails to check in is itself the alert, with no dependence on that job's own error handling working correctly. Standing the fleet up this way surfaced an immediate, unglamorous constraint: the monitoring vendor's free tier covers exactly one heartbeat monitor, and the fleet runs 22. Every job beyond the first was silently dropped rather than monitored, which is its own quiet version of the same failure mode this whole effort was trying to close.

The honest number: full heartbeat coverage across 22 scheduled jobs costs about $45 a month on a plan-plus-metered-overage structure, not the platform's higher advertised tier, which bundles anomaly detection and dashboard features a 22-job fleet doesn't need. Picking the plan that matches the actual requirement instead of the one a pricing page defaults to is most of the savings.

Put the alert where the eyes already are

A heartbeat monitor is only useful if a human sees it fail. Alerts that land in an inbox get triaged into an inbox's usual half-life, which is to say eventually. The fix was routing every alert rule into the Slack channel the team already has open during the workday, so a failing job produces a message in the same window as everything else that day, not a separate surface someone has to remember to check.

That closed the read side. The same day also closed the write side: a Slack command listener wired to a scheduled agent runner, so a message in that channel can trigger a fresh diagnostic run and get a reply back in the same thread, without anyone opening a terminal. A monitoring channel that only reports is a dashboard with extra steps. A monitoring channel a person can talk to is closer to having someone on call, at the cost of an automation and no on-call rotation.

What one day of closing the loop actually produced

Measured the same day, against the live Sentry API rather than from memory: 38 issues across the affected projects moved to resolved, the majority of them the backlog that heartbeat coverage and the permission fix had been masking. All 22 cron monitors were confirmed checking in clean. Three scheduled automations went live: a daily smoke test of the client-facing surface, a weekly content-vault pass, and the Slack command listener itself. Nine tickets got filed to keep the decisions traceable; four closed the same day, alongside one older ticket this fix finally let the team close out too.

None of this is exotic. Heartbeat monitoring, alert routing, and a chat-native command surface are all individually ordinary tools. The result came from wiring them into one loop instead of running each in isolation, on the same infrastructure this firm runs its own operations on before it goes anywhere near a client's.

The reusable shape

The pattern generalizes past this one fleet: any scheduled-job system with silent-failure risk needs the same three pieces, a heartbeat per job independent of that job's own error handling, alerts routed to a channel people actually read instead of one they have to remember to check, and ideally a way to talk back to the system from that same channel. That is the version of this loop worth carrying into a client's operating substrate once it has run on RE's own infrastructure long enough to trust it, which is exactly the dogfood-before-productize rule this firm holds itself to everywhere else.