It had to pass a hard test: the client's own engineers reviewed it and merged it into their live system.
The situation
A data platform had a fraud problem: bad actors polluting the data its buyers pay for. Their quality team investigated flagged tickets by hand, with an analyst running database searches for every ticket. Their verdict on earlier tools was blunt: broad exploration, when what they needed was a deep answer on the one flagged question.
What I did
I built a system that works on their own data. It reads a flagged ticket, pulls the evidence rows before the AI is asked anything, and walks a 5-step flow, with a person in the loop, to a report card their buyers see.
What made it trustworthy is not the AI. It is the checks around it:
- Missing evidence cannot produce a "clear" verdict.
- Every number in a report is checked against the real query counts.
- When the device evidence contradicts itself, the report stops instead of going out with the contradiction inside it.
For a time, alongside it: a weekly threat report. One week's run read 8,763 messages across 19 fraud-community channels and fed their detection rules and a brief for their leadership.
The number
Their own engineers reviewed the finished work and merged it into their live system: 369 files went onto their own changed codebase, with 18 conflicts fixed by hand, and a later merge landed with zero conflicts and 2,498 tests passing. The engagement is monthly and still running.
Anonymized at the founder's direction. Every number is from the engagement record
More work: the pipeline that failed 95 times silently · the investigations OS · the research desk that runs itself