Skip to content

Silent failure

A failure that produces no error and no alert, so the system reports success while the work it was supposed to do never happened.

An error is visible. It has a status, a stack trace, and a red run in the execution list. A silent failure has none of those, which is why it is usually discovered by a person asking why something stopped arriving, weeks after it stopped.

Common shapes, all of which leave a green run behind:

  • An unhandled branch. A router has three cases and an unconnected fallback. Records that match nothing pass through an empty path and disappear. The run completes, so the platform marks it successful.
  • A trigger that stopped firing. Nothing failed, because nothing ran. A deactivated workflow, a lost webhook registration, or a schedule in a shifted timezone produces no failed runs at all, because it produces no runs.
  • A 200 with an empty body. The HTTP call succeeded and returned zero records. The loop iterates zero times. The difference between “legitimately empty” and “the query broke” is invisible unless you assert on it.
  • Credentials that fail soft. A token that quietly loses a scope returns partial data instead of a 401, and the workflow processes the partial result as though it were complete.
  • A queue nobody drains. Failures are written to a table for later handling, and later never arrives.

The fix is to assert, and to assert from outside. Inside the workflow, check the shape of the result: if a weekday fetch returns zero invoices, throw. At the end of every successful run, write a heartbeat with a timestamp and record counts. Then run a separate scheduled check that reads that heartbeat and alerts when it is stale.

That last step is the one that matters most, because a workflow that is not running cannot report that it is not running. Everything else is the system grading its own work. See observability for the wider practice, and dead letter queue for where the failures should land.