From AI experiments to AI products
A new job where AI is the whole job, and the first thing I've had
I changed jobs recently. After a long time in the solar industry as a Product Manager, I now get to work directly on AI at a new company. This is actually a super fun role I'm really excited about. My job is to find operational bottlenecks across the business, design pilots against them, and turn the ones that work into production systems. Both for internal work and for revenue-facing operations.
I came in expecting to spend months listening and mapping workflows before touching anything. The data pointed at a real bottleneck almost immediately, though, and working on it made me write down how I want to run AI projects here. I ended up with three steps: find, prove, scale.
Why this works now
Big companies plan so much because testing used to be expensive. When a pilot means engineers, integrations, and a chunk of someone's roadmap, spending weeks on a detailed plan before committing makes total sense. Meticulous planning protects companies from wasting resources.
With AI, I can get a working test running in days, and it costs almost nothing. I'd rather run five small tests and put everything behind whichever ones deliver than spend a month writing a plan for one of them. A failure costs us days and we learn something either way, which is more than I can say for most planning documents I've written.
I know this runs against how most large companies operate. It ran against my own instincts too, after years of writing business cases to get anything built. Business cases still have their place. What I'm really trying to say is that the way I see development and efficiency gains coming with AI is through rapid testing, at a speed we've never been able to accomplish before. So instead of derisking through research first and long meticulous planning, we can now derisk through lightning fast execution, from a corporate perspective at least.
Find
Testing cheap doesn't mean I test everything. There are more AI ideas floating around here than I could build in years, so before anything gets a slot, I score it. Business value first, in dollars where it's able to be calculated. Then whether it's work AI is actually good at, which to me means judgment calls, prediction, personalization, content generation, and anything involving unstructured data. Rules-based work with clean inputs usually doesn't need AI at all.
Here's my scorecard I came up with (ok, some of this came from Claude, but I do think it's genuinely useful and I'll continue to use it to make decisions). Every opportunity or AI ask goes through this scorecard, and anything that scores above a 70 gets tested.
- Business value, out of 30 - expected revenue, cost, speed, quality, experience, or risk impact. In dollars where it's able to be calculated.
- AI fit and technical feasibility, out of 20 - is this judgment, prediction, personalization, content, or unstructured-data work, and can we actually build it?
- Data readiness, out of 15 - enough clean, accessible data to ground the model and act on its output.
- Workflow ownership and adoption readiness, out of 15 - a clear business owner and a team ready to change how the work gets done.
- Risk and controllability, out of 10 - legal, privacy, security, and reputational exposure, and how well we can contain it.
- Reusability and strategic fit, out of 10 - whether the data, controls, and components carry over to the next opportunity.
It takes a few minutes per idea and it saves me a lot of arguments, because when someone asks why we're building this instead of their thing, I can walk them through the same scores.
The first thing I'm working on in this role is conversion rate. We have a high-volume operation where a big share of what comes in never converts. The people were doing their jobs fine. The systems in the middle moved work along without capturing the value in it. So the first test is AI going directly after conversion, which makes it a direct product improvement measured in revenue, and exactly the kind of project I hoped this job would be.
Prove
Every test starts with a baseline. Before we launch anything, I want written down how the current process works, what its KPIs are for the last 12 months, its operating cost, who built it, and who manages it. This also gets used to build the case for whether AI is actually helping or not.
The build itself follows two rules. First, redesign the workflow around the tool instead of bolting AI onto the existing process. Bolted-on AI inherits every problem the workflow already had. Second, ship the simplest version that could work. Simple ships in days, and when something goes wrong, you can see where.
From there it's measurement: value, quality, cost, and risk, against a scale-or-stop threshold we agreed on before launch. I like to set the threshold up front so that once the pilot numbers start rolling in, we don't have to play the game of figuring out if we're okay extending or not. ## Scale
I don't have personal evidence on this one yet, but I see this as pretty clear: we take the ad hoc pilot I've built and get engineering to build it for real, integrated properly into our tooling so it can be productized, and pieces can be reused for other pilots.
We set up the measurement in revenue and margin rather than usage on purpose. That's what the business actually cares about, and it's the standard I want to be held to.
I'll update this in another few months to see how things are working with this path.