Why your automation fails quietly, and what to do about it

The automations that hurt are not the ones that crash. A crash produces a stack trace, a red run in the execution list, an email to somebody. A crash is a gift: it tells you where and roughly when.

The expensive failures produce nothing. The workflow reports success. The dashboard is green. Six weeks later somebody asks why finance stopped receiving the Monday reconciliation file, and you discover it stopped in the second week of the quarter.

This note is about that second category: how silent stops happen, and what to add to a system so they stop being silent.

An error is not the same as a stop

Two different things get lumped together as “the automation broke”.

An error is a run that started and threw. There is a record of it with a failed status. You can count these and alert on them.

A stop is the absence of a run. Nothing threw, because nothing executed. There is no failed run to count, because there is no run. Monitoring that only watches for errors is blind to this whole class of problem, and in long-lived automations it is the more common one.

The shift is to stop asking “did anything fail?” and start asking “did the thing that should have happened, happen?”

Five ways systems go quiet

The unhandled branch

A router or switch node has three defined cases and a fallback that goes nowhere. Ninety-five percent of records match one of the three. The remaining five percent hit the empty fallback, the run completes, and the platform marks it green because completing an empty branch is not an error.

This is the most common silent failure in visual automation tools, because they make it easy to draw a branch and easy to leave one end unconnected. The fix is mechanical: every switch gets a default path, and that path does something visible. Write the unmatched item to a table, post it to a channel, increment a counter. Anything except nothing.

A trigger that stopped firing

Polling triggers depend on a schedule that itself depends on the platform being awake. Webhook triggers depend on the sender still knowing your URL. Both can stop without any signal on your side, because from your side nothing happened, which is exactly what nothing looks like.

Causes seen in practice: a workflow deactivated during debugging and never reactivated, a redeploy that lost the webhook registration, or a schedule set in a timezone that shifted.

A 200 with an empty body

An upstream API returns HTTP 200. Your HTTP node is happy. The body is {"data": []} or, worse, {"results": null}. Your loop iterates zero times. Your summary step reports “0 records processed” to nobody. The run is green.

Empty is a legitimate answer sometimes, which is what makes this hard. Sunday has no orders. That is fine. What is not fine is a system that cannot tell the difference between “legitimately empty” and “the query broke and now returns nothing”.

Expired credentials that fail soft

Most credential expiry produces a clean 401. Some does not. A token that quietly loses a scope, or a service account removed from one folder but not another, returns a partial result instead of an error, and the workflow processes that partial result as though it were the whole thing.

A queue that drains nowhere

You push failed items to an error table so they can be handled later. Nobody built the later. Because the main path succeeded in enqueuing the failure, the run is green, and the failure has become a row that no process reads.

Making failure visible

The general principle: an automation should assert what it expects, and something outside the automation should assert that the automation ran.

A checklist that covers most of it.

• Every branch terminates in something observable. No unconnected outputs. Default cases log or alert.
• Assert on the shape of the result, not just the status code. A 200 with zero records where you expected records is a failure condition you write yourself.
• Add a heartbeat. At the end of a successful run, write a timestamp somewhere durable.
• Watch the heartbeat from outside. A separate scheduled check that alerts when the timestamp is older than the expected interval plus a margin. This is what catches trigger stops, because it does not depend on the stopped workflow running.
• Record counts, not just outcomes. How many records in, how many out, at each stage. Silent data loss shows up as a mismatch long before anyone notices missing output.
• Give every queue a reader and an age alert. Rows older than a day are an alert, not a backlog.
• Alert to a place with an owner. A channel nobody is named for is another queue that drains nowhere.

A worked example

A nightly workflow pulls yesterday’s invoices from a billing API, matches them against a bank feed, and writes unmatched items to a review sheet.

The naive version has one failure signal, the run status. It goes green on an empty API response, green on a routing branch that drops credit notes, and produces nothing at all if the schedule stops.

The instrumented version adds four things and no new infrastructure:

1. After the fetch, a check node: if invoices.length === 0 and yesterday was a weekday, throw with the message expected invoices, got 0. Now an empty response is an error, not a success.
2. The matcher’s switch gets a default branch that appends unmatched types to the review sheet with a reason column, so credit notes surface instead of vanishing.
3. The last step writes { workflow: 'invoice-match', ranAt: <timestamp>, fetched: n, matched: m, unmatched: u } to a small status table.
4. A separate workflow runs at 09:00 and reads that table. If ranAt is more than 26 hours old, or fetched is zero on a weekday, it posts to the operations channel.

Point four matters most, because it is the only check that still works when the main workflow is not running. Everything else is the workflow reporting on itself, and a system that is not running cannot report that it is not running.

The rule worth keeping

Design the failure path at the same time as the happy path, not after the first incident. Ask of every step: if this silently returns nothing, who finds out, and how long does it take? If the honest answer is “a person notices eventually”, that step is not finished.