Why Quantitative Data by Itself Will Always Leave You Lost
It's been a week since you began that experiment—the one testing a bold new page layout. So you load up your web analytics and check your metrics. Page duration has ticked up! ... But conversions have dropped. What is going on? And how many experiments will it take to figure it out?
If only you had more precise, better, more complete data, then maybe you could find your answers ...
As a digital product manager, you constantly turn to data to help gauge product performance and identify areas for improvement. However, finding meaning in this steady stream of information—mining the insights that will truly pave the way for better digital experiences for your customers—is far easier said than done.
What if the struggle lies not with our abilities to interpret traditional analytics, but with traditional analytics itself? What if these metrics, by virtue of their very nature as an abstraction of reality, are doomed to fail us? And if this is the case, where should we turn for guidance instead?
Keep your maps close by for this journey ... but maybe not as close as you'd think.
The Map Is Not the Territory
In an excellent piece on the Farnam Street blog, Shane Parrish describes the concept of “the map is not the territory.” He summarizes the concept from a revolutionary 1931 presentation by mathematician Alfred Korzybski, who introduced the idea while elaborating on the relationship of both language and mathematics to physical reality.
In Parrish’s words, “the description of the thing is not the thing itself.” This is true for both language and mathematics. The word "elephant," of course, doesn't actually weigh three tons and have a trunk. (Did you have to read that twice?) The number “9” is an abstract concept and not a physical object. A painting of a pipe is not a pipe. And a map of your local city is not, in fact, the city.
Maps, models, descriptions, theories—and yes, quantitative analytics—fall into the category of abstractions as well.
If you're thinking, “We need abstraction to navigate through this complex world, don’t we? What’s the problem?” You're right.
We need maps and models and all the rest. Only, problems arise when we fail to acknowledge their limitations or rely too heavily on them. We have to recognize the ways they can lead us astray.
Reality is Messy. Do You Have a "Ground Truth?"
How do we ensure our various “maps” don't take us in the wrong direction? First, we have to remember that no maps or models are able to convey reality with 100% accuracy. “They are an abstraction, and abstraction means that information is lost to save space,” Parrish writes. “In our march to simplify reality with useful models, we confuse the models with reality. For many people, the model creates its own reality. It is as if the spreadsheet comes to life. We forget that reality is a lot messier.”
In our march to simplify reality with useful models, we confuse the models with reality. For many people, the model creates its own reality. It is as if the spreadsheet comes to life. We forget that reality is a lot messier.
— Shane Parrish
The more time you spend working with abstractions to make sense of reality, the easier it is to fool yourself. Nassim Taleb, the author of Anti-Fragile and many other books on risk and complexity, is known for calling out how the theoretical is often confused for the actual, especially by those who work heavily with abstractions:
For real people, if something works in theory, but not in practice, it doesn't work.
For academics, if something works in practice, but not in theory, it doesn't exist.
— Nassim Taleb
Taleb takes issue with the ease with which we can allow theory to prevail over practice. This is backwards because what works in practice is what actually happens. It is "ground truth" and cannot be dismissed.
Maintaining a connection to what's concrete—the territory—prevents us from making critical errors of judgment due to an over-reliance on maps.
So for digital products, how do we apply this concept?
Get Started in the Right Direction...
We need maps, but we have to remember not to mistake them for reality. So how do we use them wisely?
For teams tasked with improving product metrics, go ahead and start with a map. Load your product analytics dashboard. Study engagement metrics like session duration, bounce rates, and conversion rates. These measurements will alert you to problems and possible opportunities.
Just remember that no matter how effective your quantitative analytics may be at measuring how your product is performing, they aren't enough. You still need a way to answer the question of why it's performing that way. You need a way to pay attention to what's happening on the ground—a way to see the territory.
...Then Mind Your Surroundings by Observing the Territory
For digital products, you need a way to see the raw digital experience and know the qualitative, individual experience of the end user. This is your territory. It is the data that supplies the story behind the numbers and brings the metrics to life.
How do you research and observe what your customers actually experience when using your product? How do you measure and understand your customers’ needs and desires? We count three ways:
1. As simple as it sounds, you can make a habit of talking and listening to your customers.
a. Regularly read customer feedback.
b. Answer a certain number of support tickets.
c. Listen to friends and family who use your product.
d. Eat your own dog food. (That is, use your own product!)
2. Observe customer behavior through traditional usability testing.
3. Finally, use a digital experience analytics platform—one with session replay.
It’s this third method that will help you gain critical qualitative insights as quickly and efficiently as possible.
Immerse Yourself in Ground Truth: Their Real Experience
A digital experience analytics platform allows you to go from the map directly to the territory. For example, you can search for a specific issue with your digital customer experience—a software bug, a poorly performing landing page, a frustrating onboarding experience, and the list goes on and on—to watch the issue unfold from the perspective of users who encountered it. You see what happened and the actions they took—or attempted.
For example, the digital agency Fullstack wanted to investigate the reasons that bounce rates for one of their client websites were so high. Google Analytics gave them the numbers but not the reasons why. The Fullstack team used these analytics to focus their attention. Then, they turned to FullStory to "look at the territory" and replay these underperforming user sessions. The Fullstack team watched as users scrolled through their client's content, paying special attention to where they lost interest. By seeing where the content failed to meet user expectations, Fullstack knew what to overhaul in order to improve bounce rates. Fullstack has also used this quant+qual workflow to optimize conversion rates for PPC campaigns.
Connecting quantitative insights (the map) to qualitative research (the territory) can bring traditional web analytics and other quantitative numbers to vivid life. It's a one-two punch that grounds abstractions and interpretations in a relatable source of truth—the real digital experiences of users. Instead of having to come up with best guesses to explain key metrics, you get to observe and learn from the actual, individual customer experiences that comprise them.
When it comes to measuring digital customer experiences, this is as close to the territory as we can get.
Safe Travels and Many Happy Returns
We appreciate a good map as much as anyone. Maps are incredibly helpful for providing that high-level view you need to prioritize and focus attention. The key insight here is that as helpful as maps can be at providing direction or in simplifying complex information, they are not enough.
And when it comes to the “maps” of traditional analytics, it’s important to remember that these abstractions don’t reflect the actual experiences of what your customers go through. Don’t blindly follow these maps right off a cliff!
Instead, remember to connect your quantitative data to your qualitative insights. Observe the territory of your customer experience through listening to customers, running usability tests, or activating a digital experience analytics platform.
Immerse yourself in the ground truth of the real customer experience and ensure the map is never mistaken for the territory.
