The 7 ways automated workflows fail in production

An automation that works on the day it ships is a low bar. The interesting question is whether it still works, correctly and unattended, eighteen months later — after two API deprecations, a vendor changing an email template, and the person who built it moving on.

These are the seven ways it goes wrong, in rough order of how often they show up in audits.

1. Silent failure#

The run errors. The platform records it in a log nobody opens. Six weeks later somebody notices the numbers do not add up. This is the single most common failure mode and it is a configuration problem, not a technical one — almost every platform can alert on failure, and almost nobody switches it on.

The fix is a failure path that reaches a human who can act: an error branch that posts to a channel with the run ID, the input that caused it, and the error message. Not an email to a shared inbox.

2. The run that never happened#

Subtler and worse. A webhook stops firing. A scheduled trigger silently detaches after a credential expires. There is no error, because there is no run — and error alerting catches nothing.

3. Partial execution#

Step 3 of 6 fails. Steps 1 and 2 already wrote data. The record now exists in a state the rest of the system does not expect — an order created but never invoiced, a contact enriched but never assigned.

• Do the reads and validation first, the writes last. Fail before you have changed anything.
• Where multiple writes are unavoidable, write a status field the workflow updates as it progresses, so a recovery run knows where it stopped.
• Never leave a half-written record with no marker distinguishing it from a complete one.

4. Duplicate execution#

A retry fires after a timeout that actually succeeded. A webhook is delivered twice. The customer gets two invoices, or two welcome emails, or is charged twice.

The answer is idempotency: design each step so running it twice produces the same result as running it once. In practice that means deriving a stable key from the input (an order ID, a message ID, a hash of the payload) and checking whether that key was already processed before doing the work.

# Not idempotent — a retry creates a second record
create_invoice(order)

# Idempotent — a retry is a no-op
key = "invoice:" + order.id
if not seen(key):
    create_invoice(order)
    mark_seen(key)

Idempotency is also what makes recovery safe: when a run fails halfway, you can simply re-run the whole thing rather than reasoning about which steps completed.

5. Schema drift#

An upstream system renames a field, changes a date format, or starts returning null where it used to return an empty string. The automation keeps running and starts writing garbage — which is worse than stopping, because nothing alerts and the bad data spreads.

• Validate the shape of incoming data at the entry point and fail loudly when it does not match.
• Treat an unexpected null as an error, not as a value to write.
• Prefer explicit field mapping over passing whole objects through — you find out about changes at the boundary rather than three systems downstream.

6. Rate limits and volume#

The automation works fine on 20 records a day and falls over on the day 2,000 arrive after a backlog clears. Rate limits are the usual cause; so are per-run timeouts on hosted platforms.

• Batch and paginate rather than looping one call per record where the API supports it.
• Implement exponential backoff on 429 and 5xx responses — and cap the retries so a genuinely broken endpoint does not retry forever.
• Test with a volume spike deliberately, before reality provides one.

7. Business-rule drift and orphaning#

The last two are organisational rather than technical, and they are what actually kills automations long-term. The process changes and the automation keeps doing the old thing perfectly. Or the person who built it leaves, nobody else understands it, and it becomes a black box that everyone is afraid to touch and nobody dares switch off.

Every automation is a small piece of software that nobody has agreed to maintain. Naming an owner is the cheapest reliability work available.

The design checklist#

• Does a named human get alerted when this fails?
• Would I be alerted if it silently stopped running at all?
• Is every write step safe to run twice?
• Does it fail before writing anything, rather than halfway through?
• Does it validate the shape of its inputs?
• Does it survive 100× the normal volume arriving at once?
• Could someone who has never seen it fix it from the documentation?

Seven questions. Answering them at design time costs an hour; answering them after an incident costs considerably more. The ROI model assumes the automation keeps working — this is the work that makes that assumption true.