The situation
An energy supplier's AI incubator wanted a system that could find and assess new product opportunities on its own: market research proposing candidates, customer research sizing them, compliance research checking them, and a final step merging the three verdicts into a one-page brief. I led delivery, from written requirements to runs on production infrastructure. (The client is unnamed here: the engagement terms restrict it.)
The problem
- AI research systems are easy to demo and hard to trust: left alone, they invent numbers, cite pages they never opened, and smooth over disagreement.
- Mid-build, the system reported finding nothing for weeks, with each run costing $26 and taking 15 minutes.
What I did
- Wrote the requirements first: 89 prioritised requirements, each mapped one-to-one into the delivery backlog before the build began.
- Enforced the rules in code rather than asking nicely in prompts: no number the data didn't produce, every claim linked and dated, and a citation must name a page the run actually opened.
- Kept judgement human where it matters: a severe compliance finding interrupts a person, and disagreement between the research streams stays visible instead of being averaged away.
- Diagnosed the two bugs hiding each other: a source list the search code silently read as empty, and a loop where the system re-read its own conversation nine times per request. Fixing them took a run from $26 and 15 minutes to $0.87 and two.
- The first run after the fix produced a real, dated, five-source product proposition nobody had suggested, with a compliance finding attached that would otherwise have been discovered expensively later.
- Gated every release with 340 automated tests, and ran the system on a schedule on company infrastructure, with every step traceable.
Getting an AI to find things is easy. Getting it to tell you what it doesn't know is the real job.