5 questions to ask your product analytics vendor

Run a bake-off for product analytics, and something uncomfortable happens around the third demo. Everyone says yes to everything.

Yes, we capture everything. Yes, we have AI. Yes, mobile is covered. Yes, implementation is straightforward. The comparison grid fills in solid green across every column, and you are no closer to a decision than when you started.

The products do differ. Feature questions have just stopped exposing how, because almost any vendor can answer one honestly. The differences live one layer down, in what the architecture requires of you after the yes. A tracking spec somebody has to write. A release somebody has to wait for. A sample rate nobody volunteered.

So the questions below are architecture questions, phrased the way a buyer would actually ask them in a room.

We put them with two of the people who sit on the vendor side of those rooms. Nestor and Jennifer are solutions engineers here. They spend their weeks in evaluations, migrations, and the specific silence that follows a prospect discovering what their current tool was never capturing. They work for Fullstory, so read their answers with that in mind. The questions hold up no matter who you point them at, and you should point them at us.

1. If I need data to answer a question I didn't plan for, what do I have to do first?

Start here, because this question quietly decides how the next year goes. Nearly every meaningful decision you'll make starts with a question nobody thought to ask in advance. Conversion drops on a Tuesday. A competitor ships something. A board member wants to know why a cohort behaves differently. What happens next is the whole evaluation in miniature.

"When a buyer asks this, they're really testing whether the tool is proactive or reactive. In legacy platforms, answering an unplanned question almost always triggers a multi-week engineering ticket. You draft a tracking spec, convince developers to instrument custom event tags in the codebase, wait for the next app release, and then wait weeks for data to trickle in. You end up making critical product decisions on old, incomplete, or delayed information."

- Nestor, Solutions Engineer

Jennifer calls the pattern tag and wait, and puts a finer point on why it never fully resolves: "You can only analyze questions you've had the foresight to tag in advance. If I don't know today exactly what I'll want to look at next month, I'm not going to be able to look at it next month."

What a good answer sounds like

The vendor should be able to say you define the interaction now and look backward.

Checkout conversion slips this morning, you describe that step in the interface this afternoon, and you're reading the last six months of behavior against it. No code, no release, no waiting for new data to accumulate. That's the difference between capturing everything and then organizing it, versus deciding in advance what's worth capturing.

What to listen for

Nestor's tell, verbatim: "If the answer is 'it's just another quick track call,' that's code for: you won't have this answer for at least a month." Quick describes the developer's effort. It does not describe your wait, which runs from ticket to backlog to release to enough data to trust.

2. Is the behavioral data complete everywhere I operate, including mobile, or am I getting a sampled view?

Every vendor claims complete capture, which makes the claim worth almost nothing on its own. The compromises surface later, in the places that are hardest to instrument, which are also, inconveniently, the places your product increasingly lives.

"Modern apps use dynamic web frameworks and native mobile components, and partial data creates false conclusions about where users drop off. A lot of teams on legacy tools have no idea it's happening. They believe they're seeing every session, when they're seeing a sample."

"At Fullstory we capture when a mobile device goes offline. Shadow DOM. Single-page applications. Canvas and web components. The mobile frameworks that have historically been very hard for tools in this category, like Flutter and Jetpack Compose."

- Jennifer Hold, Lead Customer Solutions Engineer

Nestor frames what's actually at stake when the answer is a shrug: "If you're making million-dollar roadmap decisions on a sampled view or an incomplete mobile picture, you're making educated guesses and calling them evidence."

What a good answer sounds like

Specific and unprompted.

A vendor that has genuinely solved this will name the hard cases before you finish asking, because those are the ones their engineers are proud of. You want the capture to be automatic and deterministic across every surface, without draining battery, degrading app performance, or asking your team to hand-instrument anything. Privacy belongs in this answer too, not as a separate topic: capturing everything and controlling what's stored, masked, and exposed are the same conversation, and a vendor who can only do one of them has told you something.

What to listen for

"We support mobile," with no framework named. Ask which ones, and ask what happens to a session on a flaky connection. Then ask for the sample rate in writing. If it's a setting rather than a number, ask what it defaults to and who last changed it.

3. Is your AI reasoning over all of it, or only the events we remembered to tag?

This is the fastest-moving claim in the category and the easiest one to stage. Every vendor has shipped an assistant. Far fewer have changed what sits underneath it, and the assistant is only ever as good as what it can see.

"AI is only as smart as the data set you feed it. If your platform tracks the 20 or 30 events your team instrumented last quarter, your assistant is blind to most of what people actually do in your product. The real test is whether it can surface unknown unknowns: an untagged layout shift, a broken CSS selector, a dead click nobody ever mapped, without a human telling it to go look. Putting a language model on top of a sparse, manually tagged event table just gives you a nicer way to query the same gaps."

- Nestor, Solutions Engineer

Jennifer states the constraint plainly: "Model output is bounded by model input. Partial data in, partial answers out." She points to agentic session review as the version of this that goes past querying: an agent watching sessions and reporting what it found, rather than waiting to be asked about an event somebody defined.

What a good answer sounds like

The AI reasons over complete behavioral data, including everything nobody thought to tag, and it volunteers things nobody thought to ask about. That's the only version that helps, because the failures that cost you money are by definition the ones you weren't watching for.

What to listen for

Run this test live in the demo: ask their AI to tell you something about a behavior nobody on their team tagged in advance. If every answer traces back to a defined event, you haven't bought intelligence. You've bought a chat interface on a spreadsheet.

4. Who else in my business can use this, and can the data leave your platform?

Analytics tends to get evaluated by the team that will own the license and used by roughly that same team. It's worth deciding early whether that's the outcome you want, because the behavioral data that answers a product question is the same data that would have closed a support ticket faster or told a data scientist which accounts are drifting.

"Analytics shouldn't be a spectator sport locked inside a product manager's dashboard. Ask how the behavioral data serves a support agent trying to reproduce a rare bug, an engineer chasing network errors, or a data science team building churn models in Snowflake. Traditional tools force a compromise: aggregate charts with no individual context, or individual sessions that don't scale across millions of events. You want to move from a funnel drop-off to the specific session behind it in one click, and you want structured behavioral data streaming out of the platform into the rest of your stack rather than sitting in a walled garden."

- Nestor, Solutions Engineer

Jennifer has watched what happens when that constraint lifts: behavioral data into Snowflake and BigQuery, into the support tool, into Salesforce as customer context. "The tech stack cracks right open once you can send the data somewhere else."

What a good answer sounds like

One platform that connects the aggregate number to the individual experience behind it, plus a real pipeline out: a stream your warehouse, your CRM, and your support desk can consume on their own schedule.

What to listen for

Two things. First, whether the pricing model makes it economically irrational for anyone outside the core team to ever log in, which is a distribution decision disguised as a line item. Second, what "export" means concretely. Ask for the destination list and the format, and ask whether it's the raw feed or a summary of it.

5. How much engineering time stands between a new question and its answer?

The total cost of ownership question, and the one most likely to be answered with a number that has nothing to do with the invoice.

"Engineers should be building and fixing. They shouldn't be spending their time tagging data. And it isn't just the cost of that work. Every single time somebody has a question, a ticket gets filed, it has to climb out of the backlog, it has to be prioritized against everything product and engineering already committed to, and then it has to be built and shipped. Your team has to find engineering bandwidth before a question can be answered at all."

- Jennifer Hold, Lead Customer Solutions Engineer

Nestor puts it in procurement terms: "If every feature launch, redesign, or business question needs a developer to write a tracking spec and change application code, you're paying twice. Once for the license, and then continuously in engineering tax." The architectural fix, he says, is decoupling collection from organization. "Engineering implements the SDK once. Product, design, and growth self-serve from there, without pulling developers off the work that actually ships revenue."

What a good answer sounds like

Instrument once at install.

After that, defining what you want to measure is configuration rather than code, and the people asking the questions are the people who can answer them.

What to listen for

Ask a question about the past instead of the future: what did taxonomy maintenance look like for their last three customers a year after go-live? Implementation effort is always quoted at week one. The engineering tax is an ongoing subscription, and it isn't on the order form.

Five questions, one underlying test

Read them together, and they're the same question asked five ways. Does the platform capture what actually happened, or only what somebody thought to ask for in advance?

That distinction used to be about convenience. It decided whether you waited two weeks for an answer or got it that afternoon, which mattered, but you could live with it. It's now the thing your AI runs on. A model doesn't get smarter than the data underneath it, and a system that only ever looks where it was pointed will keep missing whatever nobody pointed at. That gap stops being a data problem and becomes every decision made on top of it.

So ask the five questions. Make the vendor answer the second half of each one, the part after the yes.

Ask them about us, too. We would rather answer these than be graded on a feature grid.