The situation
A door-to-door sales company runs its order flow through group chats: reps post sales as freeform text, and an inherited automation uses an LLM to extract the orders into the tracking sheet the business runs on. One carrier's orders were not automated at all, so an assistant entered every one by hand.
The part nobody knew: the LLM extraction step had failed 95 times in 30 days. Silently. The longest ran about 26 hours, and a still spreadsheet looks exactly like a slow sales day. The cause was never confirmed. Nobody was alerted, because nothing was watching.
What I did
I went in read-only first. I traced the automation end to end and found the "one system" was actually three disconnected data paths, which explained why some submissions never reached the sheet at all. I audited the sheet itself: 877 rows missing order numbers, 94 duplicates, and over 1,300 rows flagged for review sitting in the main tabs as normal lines. The quality rules ran after the model, to grade its output, not to stop it.
Then the fix direction: replace the fragile LLM extraction with deterministic code. The replacement removes the step that kept failing: plain code reads the order, so a malformed model answer can no longer stop the sheet.
The number
The acceptance test costs nothing: the extractor's counts must match the assistant's own posted monthly tallies before anything deploys. The rule the engagement runs on: nothing deploys until the machine's counts match the human's counts.
Anonymized: the client has not yet approved being named. Every number is from the engagement's own logs and diagnosis documents.
The most dangerous AI system is not the one that fails loudly. It is the one that fails 95 times in a month and keeps looking like a slow sales day. If something in your business runs on a person remembering, the assessment week finds it and prices the fix, by Friday.
Book a 30-minute callMore work: the analyst that refuses to guess · the investigations OS · the research desk that runs itself