Product direction: why most AI adoption in product orgs fails

Every product organization I've encountered over the past 18 months has done some version of the same thing: they gave their PMs AI tools. ChatGPT, Claude, Copilot. Some set up shared prompts and templates. A few ran workshops. They called it transformation.

Very few fundamentally changed how product decisions get made.

The biggest improvements came from automating the manual work PMs do day to day: writing PRDs, summarizing research calls, cleaning up meeting notes, running faster analysis on data. These are real improvements. But the bottleneck in creating valuable products is not solved by writing PRDs faster. It is about knowing what to write.

AI is making product building cheap and fast. What it has not done across the board is make product decisions better. In most product orgs, those two facts have not been reconciled. Leah Tharin has written about this clearly in her piece "Direction over Speed" — the framing captures the problem precisely.

The actual bottleneck

In most B2B SaaS product organizations, prioritization works like this: a PM spends 15 to 20 hours per cycle manually reviewing product analytics, skimming support tickets, pulling notes from sales calls, and conducting customer interviews and discovery calls. At the end of that process, they have a rough mental model of what customers are saying, their pain points, and a sense of which opportunities deserve focus. That model goes into a planning conversation, where it competes with other rough mental models from other PMs, plus whatever the CPO or CEO heard from a big customer last Tuesday.

The output is a roadmap. And because the process is slow, manual, and subjective, teams regularly build things they believed customers wanted rather than things customers demonstrably need. They find out they were wrong six months later, in production, after engineering has invested.

AI is making shipping faster. It has not fixed the process that decides what to ship. The cost of being wrong is going up.

When you move fast in the wrong direction

Before AI coding tools, the shipping constraint masked the direction problem. You couldn't move fast enough to discover how often you were building the wrong thing. You'd find out eventually, but slowly, and the feedback loop was long enough that the failure was ambiguous: was it the wrong problem, the wrong solution, the wrong timing?

With AI-enabled teams, you can ship weekly. You find out quickly, and often.

This is why in our AI process for PMs at Siteimprove, we are focusing first on using AI to help PMs make better decisions about what to build. Clarifying direction. Pressure-testing thinking. Building conviction and confidence before committing to build. This has always been what world-class product orgs focus on — Steve Blank and Eric Ries popularized getting out of the building and rapid customer discovery specifically to disprove ideas quickly before engineering invests. The difference now is that with AI, we can cycle through many more ideas faster, determine what we should not build, and find the right opportunities to invest in within days rather than weeks or months.

AI made building cheap. It also made being wrong expensive. Most product orgs haven't updated their prioritization process to account for either.

What direction actually means in practice

Direction is the answer to three questions, asked before engineering writes a line of code.

Is this a real problem? Signal strength: how many customers experience it, how valuable is it if solved, how much ARR is in play, how many independent sources are pointing at the same thing. One loud enterprise customer is a single data point. Convergence across a broad cross-section of your ICP — from product analytics, support, sales, and customer interviews — is what you are looking for. That convergence used to take significant manual effort to establish.

Is this the right problem for us to solve? Strategic alignment: does it fit our current strategy, does it move the metrics that matter, does it create value we can capture. A real problem is not automatically a right problem for us.

Are we solving it in a way customers will actually use? Validation: tested with something a customer can react to, not a document they have to imagine. This is where many teams are applying AI most directly today — reducing the time from decision to working prototype.

Most teams get the first question wrong most often. They build from a few loud voices or a compelling internal narrative instead of synthesizing the available evidence. The other two questions don't get properly asked because the first one never got properly answered.

The intelligence layer

The highest-leverage intervention in an AI-native product system is not making PMs faster at generating outputs. It is making signal synthesis faster and more accurate, so PMs can build genuine conviction in what they should build.

In my current role as CPO, we built a system where instead of a PM spending 15 to 20 hours manually pulling together insights from multiple tools, an AI system synthesizes signals from product analytics, support queues, sales calls, and customer interviews into a structured brief: a scored output where opportunities are ranked by signal strength (frequency, account breadth, source convergence, revenue weight) and strategic alignment (financial levers, roadmap coherence, product priority).

The PM reviews it in 15 minutes on Monday morning and adds their own judgment: which patterns are real, which are noise, what the evidence is missing. Their job shifts from finding the signal to judging it. That is a different role. And it is the intervention that changes what gets built.

The time savings are real: 15 to 20 hours of manual review compressed to 15 minutes. But the more important outcome is accuracy. PM judgment is applied to the right layer of the problem. They are not sifting through raw data to find patterns. The patterns come to them, ranked, with real evidence from customers and prospects. PMs use that evidence to make decisions, add strategic context, and take their highest-conviction opportunities forward with their teams.

What unlocks when you fix the intelligence layer

With this intelligence layer in place, PMs can go from synthesized signal to a complete opportunity brief in an hour. The brief comes pre-populated with customer evidence, supporting data, and structured problem framing. What used to take weeks of research takes an afternoon.

If the conviction is there, a PM can go from brief to prototype the same day: a working, on-brand, interactive prototype a customer can react to in the context of their real workflow, tested with real customers the same week.

From a validated prototype, the PM can bring it to their team to discuss how to derisk the next step — whether that is a limited alpha release, a technical POC, or a research preview to a small set of accounts.

We have been running this loop for a few weeks now. Signal to validated prototype in under a week. What used to take months of manual discovery, brief-writing, and cross-functional coordination now runs in days.

How to get started

If you want to try this yourself, you do not need a large team or a complex infrastructure project. You need two files, an AI assistant that supports file context (Claude Code works well here), and about 30 minutes of setup.

The first file is your strategic context. It is a plain text document that captures your current strategic priorities, your product area ranking, your committed roadmap (so the AI knows what to exclude from the opportunity list), your financial levers (what actually moves ARR: retention, expansion, new logo), your competitive landscape, and the key metrics you are tracking against targets. One file. Every skill in the system reads it. You update it quarterly.

Most product leaders fill this in faster than they expect — because you already know this content. You are not inventing it. You are capturing what is already in your head and making it legible to the AI. The part that requires the most deliberate thought is the priority order: product areas ranked 1 through N, and financial levers ranked in explicit order. That explicitness is what gives the alignment scoring its teeth.

The second file is the signal synthesis skill: a structured prompt that reads your strategic context and knows how to evaluate incoming signals. You point it at data you already have — customer feedback exports, support ticket CSVs, CRM account data — and it maps each theme it finds to a 2x2 grid scored by signal strength and strategic alignment. Signal strength covers frequency, account breadth, source convergence, and ARR weight. Strategic alignment covers financial lever fit, product priority, and roadmap coherence.

Your first run will take about an hour from start to finish if you have two or three data sources ready to drop in. The output is a ranked opportunity table: 5 to 8 themes, each with real customer evidence, a score, and a quadrant label. You review it in 15 minutes and add your judgment — which patterns are real, which look like noise, what the data is missing. That is the first loop. From there, the PM who selects an opportunity can go to a full brief the same day, and to a prototype the same week.

Where most teams start instead

The most common starting point for AI adoption in product orgs is the output layer: how do we help PMs write better? Faster PRDs, better-structured documents, cleaner meeting summaries. The productivity improvement is immediate and visible.

This is a legitimate improvement for individual PM productivity. It is the wrong starting point for transforming how a product organization makes decisions.

Output-layer improvements without intelligence-layer improvements produce faster, more confident wrong answers. Better-written briefs for the wrong problems, shipped in half the time.

The transformation that matters happens at the decision layer: what signal is the team acting on, how was it synthesized, and how much should they trust it. When that layer is rebuilt around AI synthesis rather than manual review, the outputs change because the inputs changed.

What comes next

This is the first post in a ten-part series on building an AI-native product organization. Each post documents a specific piece of the system I am currently building in my role as CPO: the signal synthesis brief, the opportunity canvas evolution, how PMs shifted from spec-writers to prototype builders, how we run research previews instead of traditional launches, and how we measure whether the system is actually working.

Some of it will be directly applicable to your organization. Some will need adaptation for your stack, your team's current maturity, and your customers' expectations. The core sequence is this: fix the intelligence layer first, so you have the right inputs and the right conviction before you build.

Post 2 covers the signal synthesis system and the prioritization framework that replaced our weekly debate meetings.