What did it look like before?
When a product stops growing, someone has to say why. The answer usually sits in one of five places.
Each one needs a different fix. Pick the wrong one and you lose weeks.
Why build this one?
The answer decides what the team works on next for that product, and a wrong answer doesn't show up straight away. It shows up weeks later.
I also didn't want to swap one black box for another. The goal I wrote down at the start was a tool whose answers the team could understand and question, rather than just trust.
I built this one myself because I already knew the context behind the question. As an operator I knew what the team was really trying to find out, and why each check mattered.
How did I approach it?
I started from the question I wanted answered, and worked back to the data. A few ideas shaped everything after that:
- One answer.A list of twenty possible issues just moves the thinking back to the person. The tool has to commit to the main thing holding the product back.
- Rule things out first.Before blaming anything, it checks what clearly isn't the problem. Whatever's left is where to look.
- Check our own work before blaming the product.If the ads aren't set up well, you can't tell whether the product page is any good. So the page only gets blamed once the ads are ruled out.
- Ignore thin evidence.A great result on a handful of clicks is luck. Anything with too little data behind it doesn't count.
- Fixed rules make the call.Claude reads product pages and writes up the explanation, but the verdict itself comes from rules I can check and change.
- Every “no” has a reason.When the tool rejects something, it says why in plain words, so someone on the team can disagree with it.
What broke, and what I changed
The first versions got things wrong in ways that taught me more than the parts that worked.
Two searches meant the same thing and got opposite answers.
One was kept and one was thrown out, only because one happened to share a word with the product's title. Nothing was wrong with the data. The rule treated every word in the title as equally important, when some words say what the product is and others only describe it. Once I split those apart, both searches got the same answer, and the ones that had slipped through were caught.
A helper step was quietly loosening the rules.
I'd added a step to rescue searches that had been thrown out unfairly. It worked, but it also let in things it shouldn't, and nobody was checking what it added. Now it can only make small additions, and every one of them shows up in the output with the search that triggered it.
Strict went too far.
I started strict on purpose, accepting that some good searches would be lost. Then it started throwing out the most obvious search for the product, the one buyers type first. I added a narrow exception for that case and left the rest strict.
Big products broke the AI step.
With hundreds of searches to sort, Claude's answer got cut off halfway and the whole step failed. Splitting the work into batches fixed it, and if one batch fails now, the rest still go through.
Small numbers looked like signals.
An ad position with almost no spend, or a search with a few lucky sales, kept getting picked as the best performer. Anything below a minimum amount of evidence is now left out of those comparisons.
How did the team take to it?
The team got a short guide covering two things: how to run it, and how to tell when its answer looks wrong.
There was no pushback. The team already knew why each check was there, and anything the tool throws out comes with a reason, so nobody had to take it on faith. It was useful from the first day.
What changed
Nobody's asked to use it. People pick it up because it saves them most of an afternoon.
What's still not right
- Products that already lead their market confuse it. With nothing ahead of them to compare against, it flags too many things as possible problems.
- I built a whole extra section that runs on every product and costs money each time, and it never shows up anywhere. I only found out when I sat down to document the tool.
- Some rules were written around the first products I tested and don't fit every category yet.
- On very large products it's slow, because it writes results one line at a time.
Read next
Is your team answering the same question by hand?
Tell me about it, and I'll tell you whether a system would help.