Why it matters: Automation quietly encodes assumptions about the world, and when those assumptions go stale the system keeps running as if nothing changed, often for days before anyone notices.
The Details:
- Automated systems freeze a snapshot of "how things work" at the moment they were built. A human doing the same task manually would notice a shifting pattern or an odd data point; a script just executes the rule it was given, correct or not. The failure mode is not a crash, it is confident, tireless execution of an assumption that already broke.
- The useful split is between automating execution and automating judgment. Shipping code once tests pass is execution and a safe target. Deciding whether those tests actually cover the right scenarios is judgment, and that still needs a person. Watch for growing piles of special-case logic; that pattern usually means something was never as rule-bound as it looked.
- Checkpoints fail not because they exist but because they are placed badly. A pause point stuck too early, before enough information exists, or one that forces someone to scroll a wall of logs, becomes a rubber stamp instead of real oversight. A well-built checkpoint gives a short summary of what is about to happen and what is uncertain, so a person can approve or block in seconds rather than review everything after the fact.
- The 2012 Knight Capital incident shows the stakes: a configuration error in trading automation produced hundreds of millions in bad trades within 45 minutes because there was no checkpoint before the action became irreversible. The same gap shows up in ordinary CI/CD pipelines when a bad commit ships because the automation was never told what to actually check.
Bottom Line: The safest automation is not the one with the most coverage, it is the one whose assumptions are written down and whose failure points still have a human standing at the door.
