thredUP is a little different from your average e-commerce marketplace. As the world’s largest online thrift and consignment store, thredUP’s entire inventory is temporary — every single item that’s in stock has a total quantity of one.
Backed by more than $131 million in funding, thredUP was founded in 2009 and has weathered storms both in the economy and in its vertical to become one of the standout startups in the e-commerce space.
A large part of that success has been thredUP’s strong engineering culture, led by VP Engineering Dan DeMeyere, and their obsession with improving customer experience.
Whether it’s fixing bugs, doing user research, or gauging the success of a product launch, DeMeyere and his team try to understand what users are actually experiencing — not just how many sessions are active or how much “engagement” they’re seeing.
We talked to Dan about how Fullstory and Rollbar combine to help thredUP better understand their users.
No more mysteries in user experience.
DeMeyere’s team at thredUP use a combination of Rollbar and Fullstory to understand and solve problems. Rollbar, an exception management tool, allows thredUP to monitor particular types of in-app events, analyze the causes of bugs, and fix them. Fullstory’s session replay provides much-needed context.
The thredUP team first discovered Fullstory’s capabilities when their marketing team began using a site-wide A/B testing system. One of their first tests involved promotions. The product team wanted to show discount codes selectively to only a certain segment of customers.
They set up the A/B test; all appeared to be running smoothly. But they soon noticed that the people who weren’t being shown discounts were still applying them once they got to their cart — and they had no idea why.
There were no obvious bugs in the code. They configured Rollbar to alert an engineer anytime a customer in the no-discount segment applied a discount code to their cart. For a week, thredUP engineers sat and watched as customer after customer somehow slipped the parameters of their test.
The breakthrough came when they turned to Fullstory and watched what was happening on-screen for the affected users.
“It was very obscure, but there was this very old system that we had for the way we do our carts that would do an up-sell to somebody based on a certain referral code,” DeMeyere says, “We weren’t finding it in our code, but we were able to rewind and see the customer paste in that value in the promotion field.”
“Once we figured that out [through watching session replays we] were able to move quickly and fix it, and that’s when the power [of Fullstory] became really apparent to us,” he adds. “We might have been pulling our hair out for weeks had we not thought to use Fullstory.”
Session replay is instant, bias-free user research.
Fullstory session replay helps thredUP understand how the changes they make to their site impact users.
When the thredUP team set out to build a new version of their product, they knew it would be a total overhaul of the shopping experience on their site. There would be a complete architecture rewrite on the back-end, and a complete teardown of the UI/UX on the front. Everything was going to change.
The product team needed to see if people were using the new site and all its new features the way they were hoping they would. They had certain hypotheses, and they just needed to test them.
If everything was so simple.
Testing user behavior through traditional research had inescapable problems:
Conventional user research often involves (intentionally or inadvertently) asking leading questions: “If you were going to do x … how would you do that right now?”
Research studies often aren’t very indicative of how users actually behave — especially thredUP’s heavy user base of busy moms who don’t have the time to poke around the UI to discover the features they require.
Even still, thredUP had plenty of data at their disposal — sessions, visits, engagement around clicks, impressions for modal windows. They even had the resources to crunch the data. Yet it wasn’t enough. They struggled to explain as some users deviated from expected behavior.
That’s when the light bulb to use Fullstory went off. Here’s DeMeyere:
Everything kind of changed. The problem we were trying to solve, that Fullstory did a much better job of solving than some of the other services that we were using, was trying to understand what our customers are actually doing.
Rollbar and Fullstory combine for faster, better bug diagnostics.
Fullstory started as a tool for thredUP’s product team to do user research and understand the impact of new features, but the engineering team quickly adopted it for another purpose — better bug fixing.
But it didn’t happen instantly. There was concern among the engineers that recording the actions of users on every page across the thredUP website was going to have an impact on performance. When pages load slowly, shoppers leave, so getting assurance on performance was a Big Deal.
The good news was that Fullstory doesn’t delay the loading of pages — with the gzipped version (16kb) edge-cached on Google’s CDN, Fullstory’s recording script (fs.js) is usually fetched in under 20ms.
Whatever concerns about performance evaporated once thredUP’s engineers got into a few session replays and were able to see people actually navigating their site. Soon, the engineering team came to find it an invaluable tool for understanding bugs and replicating them.
Before Fullstory, thredUP fixed bugs like this:
Document everything that’s been tested,
Collect more data,
Look for patterns,
Are exceptions all around one browser?
Are exceptions all around one acquisition channel?
Repeat until bug’s unique fingerprint is found.
It is almost impossible to fix a bug if you can’t replicate it, but if you can replicate it, you can form a hypothesis as to why it’s happening, test different kinds of input, collect a bunch of data around each test, and start to get closer to understanding what’s wrong.
This process can take a very long time. Many times it never happens. Not reproducible.
thredUP found a solution to this problem was to plug user information from Rollbar into Fullstory and then watch as their customer themselves produced the bugs! That resulted in saving time and frustration, faster bug fixes, and most importantly, a better customer experience. Here’s DeMeyere:
Any time we can make engineers move faster and be more productive with tooling that doesn’t cost us an insane amount of money, then we’re probably gonna opt for that.
The Fullstory perspective.
Data-driven decision making is often a point of pride within the tech companies. And there’s nothing wrong with data! The problems come when companies find themselves relying too much on aggregated data and abstracted models.
That can mean seeing metrics rather than the actual people who drive those metrics. It can mean building products that are designed for the average instead of the individual. Soon we’re spending less time understanding customer experience — less time paying attention to how they use our products. And sometimes your bug reporting structure evolves into a byzantine bureaucracy.
It doesn’t need to be that way.
thredUP found it’s possible to move fast and be obsessed with building a great experience for your customers — by making sure that even as a multi-million dollar business, the capability is still there to look through our users’ eyes and see what’s actually going on.
Thanks Dan DeMeyere at thredUP for sharing how thredUP is using Fullstory!
