The orders that stopped going missing

The situation

A door-to-door sales company runs its order flow through group chats: reps post sales as freeform text, and an inherited automation uses an LLM to extract the orders into the tracking sheet the business runs on. One carrier's orders never got automated at all, so an assistant hand-enters roughly 396 orders a month.

The part nobody knew: the LLM extraction step had failed 95 times in 30 days. Silently. The longest outage ran over five hours, and a still spreadsheet looks exactly like a slow sales day. The pipeline eventually recovered on its own, most likely because the AI provider updated their model. Nobody fixed it, because nobody knew it was broken.

What we did

We went in read-only first. We traced the automation end to end and found the "one system" was actually three disconnected data paths, which explained why some submissions never reached the sheet at all. We audited the sheet itself: 877 rows missing order numbers, 94 duplicates, and over 1,300 rows flagged for review sitting in the main tabs as normal lines. The quality gate that was supposed to block bad rows had never blocked anything.

Then the fix direction: replace the fragile LLM extraction with deterministic code. Measured against 400 real orders, the deterministic extractor caught the order number on 271 of 271 in the first carrier sample.

The number

The acceptance test costs nothing: the extractor's counts must match the assistant's own posted monthly tallies before anything deploys. First measurement came in within two percentage points of the human tally. The rule the engagement runs on: nothing deploys until the machine's counts match the human's counts.

Anonymized: the client has not yet approved being named. Every number is from the engagement's own logs and diagnosis documents.

More work: the analyst that refuses to guess · the investigations OS · the research desk that runs itself