The playbook, nine months later
I promised an update on the find, prove, scale playbook. Here's
Last November I wrote about my find, prove, scale playbook and said I'd post an update on how the first project went. That took longer than I planned, but the results are in and I'm really happy with them.
The project was improving conversion rate in one of our high-volume operations. We set the target before launch, about a five point lift in paid conversion. Comparing the first seven months of this year against the same months last year, conversion is up 4.9 points. AI now handles about a fifth of the operation's volume, and the program has brought in over twenty times what it cost to run.
A few things surprised me along the way.
The monitoring system turned out to be the bigger asset
The second thing I built was a monitoring layer. AI reviews the output of the first system, scores it against what we care about, and flags problems to the right people on a schedule. I built it as a safety net for the main project. It's become the thing we lean on most.
Before, checking quality meant sampling a handful of cases whenever someone had the time or an inkling about something. Now, AI reads all of it every day. The system is self-improving, and things that need human involvement get auto-pushed to the proper team member. The AI reviews are actually becoming business data we're using to make production decisions on, which is super exciting.
AI in production needs a daily owner
In May I got pulled onto other priorities, and for a couple of months a lot of volume ran through AI without anyone managing quality day to day. By August some buyers had paused over quality concerns, and we pulled volume back to protect those relationships. The technology didn't break. It kept running without anyone owning it, and quality drifted the way it drifts on any team without a manager.
That's the biggest lesson of the nine months for me. An AI system that touches revenue needs someone whose actual job is to watch it every day. We're fixing that now, and it'll be the first thing I set up on the next project.