Product metrics are important because they help create that alignment between all these different organizations at your company. Without product metrics, it's hard for everybody to speak on the same page.
—Tommy Noonan, Product Manager at Fullstory
Tommy Noonan, Product Manager at Fullstory, joined Hanna Woodburn, Product Marketing Manager, on the May 31, 2019, LinkedIn Live series to discuss the role product management metrics and KPIs in growing a thriving business.
Tommy and Hanna discussed topics including:
Key North Star Metrics you should be measuring for product growth.
Best tools to gather and understand product metrics.
How to better serve customers with data analysis.
You can keep up with our LinkedIn content by following us there or on Twitter.
Watch the Replay
Read the Transcript
Hanna Woodburn: Hi, everybody. It's Hanna with Fullstory here with another session of Fullstory LinkedIn Live. I'm so excited to be here with Tommy Noonan, a product manager here at Fullstory.
Tommy Noonan: Yeah. Thanks, Hanna. Happy to be here.
Hanna Woodburn: Thanks for joining us this Friday afternoon. We're coming at you from our Fullstory headquarters here in Atlanta, Georgia. Today we're going to be talking about product metrics. Just a reminder, this is a live session, so please feel free to leave any comments below. If you have questions, we'll answer those later on in this session.
Hanna Woodburn: Also, we're going to be talking about metrics today, and if this is a topic that you're really excited about, we're having a webinar actually later on in June with Amplitude. We'll be talking about north star metrics and how do you create north star metrics, what does that mean for your business. There should be a link in the comments if you're interested in joining that webinar. Please do sign up.
Hanna Woodburn: Tommy, just to kick us off, tell us a little bit about yourself. How did you end up at Fullstory?
Tommy Noonan: Yeah, for sure. I actually started my career at Accenture as a systems integration consultant. I did a lot of things like building custom solutions for more larger enterprise clients. Then I actually came to Fullstory from business school. I interned at Google as a product management intern on their mobile search team right before switching to here.
Hanna Woodburn: Cool. How did you fall into product management?
Tommy Noonan: Yeah. That is a great question. I feel like when I got to grad school, I had never heard of a product manager before, but I knew I really wanted to work with data to make decisions, be in front of customers, and affect the development or the strategy of the product. Everybody was like, "Yeah. It really sounds like a B2B product manager." I was like, "Yes. That's what I want to do. That's what I want to get into."
Hanna Woodburn: That's awesome. You're obviously very interested in data and how we can make better decisions using data. How do you think about product metrics?
Tommy Noonan: Yeah, for sure. To me, product metrics are all about decision making. They should be able to enable your organization to make better decisions faster. I kind of think about that along two axes. First is strategically. It's, "Are you tracking the right metrics?" I think about that as I want my metrics to tell me are customers being able to, I guess, complete the use cases that they're coming to our product to do. Operationally, that's all about, "Are the metrics easy for everyone to interpret, to read? Is everybody on the same page? Do they know where to find them?", but more of ... actually once you have the metrics, what do you do with them?
Hanna Woodburn: Yeah, because you have to do something with that data, otherwise it's not operationalized.
Tommy Noonan: I hope people do stuff with it, at least.
Hanna Woodburn: Yeah. What's the difference between a product metric and a KPI?
Tommy Noonan: Yeah. I actually think of these as fairly similar. At least, I use the two terms pretty interchangeably.
Hanna Woodburn: Yeah. Do you think that PMs are the best person to be the owner of product metrics?
Tommy Noonan: I actually do think PMs should be the owner, but it needs to be a collaborative process. What I mean by that is I'm definitely a firm believer that if everybody owns something, no one owns something. Kind of a tragedy of the commons type thing. Like I said, it needs to be collaborative. There are some things that maybe an engineer or a designer might understand a little bit better than a product manager. Product managers should probably work with their customer success teams and their support teams to get a better understanding of what their actual customers are saying when they start to develop metrics that they're going to actually make decisions off of.
Hanna Woodburn: That makes sense. We talked last week about cross-functional collaboration, and it seems like that's just a recurring theme that I hear when I talk with PMs, both here at Fullstory and at other companies, is no matter your role as a PM, whether you're focused on data or launching new features or whatever, it's really you're connecting to other people across the business to understand how do we move forward.
Tommy Noonan: Yeah, and that's a great point. Product metrics are important because they help create that alignment between all these different organizations at your company. Without product metrics, it's hard for everybody to speak on the same page. It's hard for people to prioritize things if everybody is kind of focused on different metrics or different ways to think about things. I feel like the mental resources that we all have are kind of the scarcest resource that we have at companies like this, and so being able to prioritize things quickly and effectively is super important.
Hanna Woodburn: Yeah. What makes a good product metric versus a bad one? Is there a bad one?
Tommy Noonan: I think there are probably certainly bad ones, but it also depends on what you're trying to track.
Hanna Woodburn: Sure.
Tommy Noonan: Churn, to me, is actually an example of potentially a great product metric. What better way for someone to tell you that they're not getting use out of their product for them to actually stop using you and churn? But it can also be a bad product metric because if I'm looking at churn, it's kind of too late. I would like to know that customers or users aren't completing their use cases well before they actually make the decision to quit.
Hanna Woodburn: Yeah, so you need some of those leading indicators.
Tommy Noonan: Exactly.
Hanna Woodburn: Yeah, definitely. We're a quick growing company. Let's talk a little bit about growth. How do you think product metrics change throughout growth?
Tommy Noonan: Yeah. Product metrics definitely can and should change as the company grows, and I definitely want to reiterate that it all goes back to the customer and how your product is evolving to meet your customer's needs. I think an example of that would be like at Fullstory, we probably originally found product market fit more in the SaaS support use cases. How do we best serve those customers? They were coming into Fullstory every day to analyze different customer trends, watch sessions of people encountering bugs, etc.
Tommy Noonan: That was something that we said daily active users is very important for us at that time. As our product has evolved a little bit, and as we start tackling new use cases, such as conversion rate optimization, product manager use cases, we decided that daily active user isn't a necessity for these users, so maybe weekly active is the right way to start thinking about this.
Tommy Noonan: There's definitely never going to be a light that comes on that says, "It's time to change your product metrics." Nothing is black and white, but I think the perpetual, constantly reevaluating who my customers are, how are they most getting value out of my product, is the best way to really decide what metrics should I be tracking.
Hanna Woodburn: Yeah. That makes a lot of sense. For those of you who are joining us, I'm talking with Tommy Noonan. He's a product manager here at Fullstory. We're talking about metrics. If you have any thoughts on product metrics, either ones that you think are really great or other ideas about how you measure engagement with your app, please feel free to post those in the comments below.
Hanna Woodburn: We talked a little bit about how as your company evolves, your customer base usually evolves, your market fit evolves, and your metrics need to evolve as well. How do you keep your metrics fresh?
Tommy Noonan: Yeah, for sure ... It's kind of like what are the customer interactions. You need to be very in tune with what your customers are doing and what are the most important features and what are the core problems they're trying to solve.
Hanna Woodburn: Yeah.
Tommy Noonan: That's certainly key. We've actually developed a north star metric, which I know we're having our webinar on in a couple weeks. It's a way for us to say, "There's a lot of different metrics out there. There's probably a lot of different use cases people are trying to do, but this is the thing that we hold, I guess, above the rest of them." An example of that at Fullstory is we actually track weekly active explorers, so users that come into our app once a week and use search or segment, as our metric that, as of right now, we're tracking as the core metric for Fullstory product team.
Hanna Woodburn: Yeah. I think it's really clarifying to have a north star metric because then you're able to say, "Okay. We have all of this data, we have all these metrics, and this is the one that we really care about tracking towards, the one we really want to prioritize."
Tommy Noonan: Definitely. That goes all back into the alignment of different teams, but it also speaks to what happens when metrics go up or down. How do we interpret the change that we're seeing? In my opinion, interpreting and reading that change is probably more important than the metrics themselves. When you have a north star metric and you actually take the time to socialize it and weave it throughout the fabric of your product team, it really can be enlightening in terms of, "Oh. A metric that we have is going down or up, but it doesn't change our north star metric. That might be something that we prioritize. This is not the most burning thing on my plate right now. I can look at that at the end of the week. It's not a drop everything that I'm doing right now."
Hanna Woodburn: Yeah, yeah. That's a good rubric. When you think about metrics and socializing them, how do you go about doing that?
Tommy Noonan: Yeah, for sure. Data is certainly the great equalizer, to me. I think of myself as a data-driven person, so I probably rely on that maybe even more than I should, but it always is really helpful when you're going to people that you know have to be bought in on a north star metric, whether it's product marketers, engineering managers, what have you, to say, "There's a reason why I'm led to this specific metric." Maybe it's something that reflects what you think of as the core strength of your product. Maybe it's something that you think of as the users that use this metric tell us that they're getting the most value out of our product. Being able to back up why you're choosing one thing or another is, to me, extremely important when it comes to actually convincing everybody to come around it.
Hanna Woodburn: Yeah, definitely. You talked a little bit about responding when your metric either goes down or up. How do you know when you should respond versus, "Oh, this is just normal fluctuations based on things that are happening"?
Tommy Noonan: Yeah. This is definitely a million dollar question, and I'm sure it keeps a lot of people up, myself included, at night. One of the ways I tackle that is with the north star metric. It at least gives me a little bit of clarity in terms of, "Okay. This is going down. It's important, but it's not necessarily the most burning thing on my plate." Also, we have the idea of like KPI or metrics trees. Those are probably best represented by how are our metrics related to each other, or how does one thing affect another.
Hanna Woodburn: Yeah.
Tommy Noonan: Taking our weekly active explorer, for instance, I think of it as a weekly active explorer is a combination of how many monthly active users we have, what percentage of those monthly active users are coming into our app on a weekly basis, and then lastly, what percentage of those weekly active users are actually exploring. That way, when I see my north star metric go up and down, I can kind of immediately pinpoint, "Oh, monthly active users is down. It's Christmastime, but the people who are actually here, they're searching, they're coming into the app frequently." That's something where I say, "Okay. No worries." It just gives you a head start and a little bit of clarity in terms of what to look for.
Hanna Woodburn: Yeah. It seems like you still have to contextualize the data elsewhere though, understanding what other factors might be at play.
Tommy Noonan: Yeah, absolutely. Kind of thinking about this in the positive light of what happens when your north star metric goes up, this is something that we look at really closely when it comes to things like feature launches or big, drastic changes in a product. An example for this for Fullstory is a few months ago, we launched a feature called Custom Events. We noticed that our customers that were using the feature were actually performing better on the north star metric than our customers who weren't.
Tommy Noonan: That in and of itself was like, "Great. We released something that we wanted to change our north star metric, and it did." What was really enlightening for us was actually looking at the KPI tree and figuring out why the weekly active explorer numbers were going up. That was when we found super quickly, because we already have, I guess, like a framework for analyzing these things, "Oh, these were actually never before seen users that were coming into our customers' apps and using Fullstory now because they wanted to use this new functionality." That was something that was like, I guess, a great product metric win for us, being able to predict what would happen and then actually see it happen in our metrics.
Hanna Woodburn: Yeah. I love that. If anyone who's viewing has any product metric wins you'd like to share, you can do that below in the comments. I'm curious. If someone is watching this and they're wanting to get started maybe creating a north star metric or becoming more data-driven, what advice do you have for them?
Tommy Noonan: Yeah. That's a great question. Let me think about where to begin. For me, I think it's ... When I think about when I first came to Fullstory, it was making sure that we had the data needed to analyze the things I wanted to analyze or start to like string all these things together, eventually bubbling up into a north star metric. A lot of times at early stage companies, you might not necessarily have an entire BI or an analytics team that's sole focus is to work on an analytics stack that people can query and draw inferences from.
Hanna Woodburn: Mm-hmm (affirmative).
Tommy Noonan: That was something that was very big for me. I would say it's understanding what data you have and then how to get the most value out of it because even though getting the data's important, I think you'll find pretty quickly that there's a plethora of things you can analyze. It's really about asking the right questions and figuring out what you want to track. I think that all goes back to the customer, trying to understand who are the people that you're trying to serve and what do I need to know to best serve them.
Hanna Woodburn: Yeah. That makes a lot of sense. You sort of start with that perspective of understanding your customer and then using that to say, "Okay. What are the questions that I care about? And then how can I answer those questions with data?"
Tommy Noonan: Yes, exactly.
Hanna Woodburn: Okay. Cool. That makes a lot of sense. You mentioned tech stack. What is in your tech stack that you love for understanding your users and metrics?
Tommy Noonan: Yeah, for sure. I don't think this is going to surprise anyone, but I am a very big Fullstory user, both the actual UI and I'm a heavy power user of our data export feature, which is kind of like the raw data underlying everything that you might analyze in the app. I also have a SQL background from Accenture, so I love using BigQuery. That's kind of the, I guess, analytics warehouse that we use to take things from not just product engagement data, but things like Salesforce, customer support tickets, customer segmentation data, kind of everything I can meld together in that type of environment. Then we also have Looker that sits on top of that. It's been super helpful, honestly not as much from the data visualization side, but more of just like how is our business logic to convert all of this disparate data into actual insights. Looker kind of houses that for us.
Hanna Woodburn: Yeah. That's awesome. I think from my perspective, I know that I've gotten to work with you on some data projects and saying, "When are we launching features? How are people engaging with this? What's happening?", but it's been really exciting to see you roll out north star metrics here at Fullstory. I think it's created a lot of clarity around what are people doing on our app and what do we want to truly be tracking towards. I think as an observer of that, I can recognize the value of using data to make decisions.
Tommy Noonan: Yeah. That's definitely great to hear. The north star metric has definitely been super enlightening for me. I almost saw it as like product management tech debt. As opposed to being kind of reactionary every single time we heard of something or something was going on, we now are at least monitoring what we feel is the right thing and responding appropriately.
Hanna Woodburn: Yeah, definitely. That's awesome. We're going to turn to the audience now. If you have any questions, please feel free to drop them in the comment section. We did have a comment. I think this is great. It said, "The age old question. No one grew up wanting to be a product manager", which that certainly seemed to be your experience.
Tommy Noonan: Yeah. I would definitely say so.
Hanna Woodburn: Do you have any advice for people who maybe are considering a career in product management?
Tommy Noonan: Yeah, for sure. I definitely took one route to product management. I went to consulting and then business school and transferred in here, but I almost think that one of the better ways to do it is just to go ahead and actually get in the companies that have product management positions, even if you're not going to be a product manager. Just show them that you are, I guess, the right person for it. There's definitely no right way to get into the field.
Hanna Woodburn: Yeah, definitely. We talked a little bit about someone who's just starting off measuring engagement or product metrics. What metrics would you specifically suggest a small team start off with?
Tommy Noonan: Yeah. That's a great question.
Hanna Woodburn: Is it daily active users? Monthly active users? It just depends maybe?
Tommy Noonan: Yeah. I definitely recommend ... Those are probably the core three I would start with, monthly, weekly, and daily. Then from there it's like knowing what type of business you are. Are you transactional-based, that maybe people just come to create a transaction and aren't really coming on a daily basis to complete tasks? That might have something different. You might be tracking transactions instead of just users that are coming.
Hanna Woodburn: Yeah, definitely. What do you think the challenge is for people who are trying to create product metrics? Are there specific challenges you faced that others might be facing?
Tommy Noonan: Getting everybody to align on metrics is tough. You're not going to get alignment unless you prove that you're willing to listen to everyone who has something to say about it. Listening to people from different perspectives, I think, is super important. I think data is helpful, and it can help build conviction in yourself before you want to present an idea out there that you know is probably going to get poked and prodded from every single way, but listening to people, building that trust, and showing them your thought process for building up to a north star metric was super helpful for me.
Hanna Woodburn: Yeah, definitely. That makes a lot of sense. Great. Thank you so much for being here with us today. I really appreciate your time and insights.
Tommy Noonan: Yeah, absolutely. This was fun.
Hanna Woodburn: Yeah. Awesome. Thank you for watching. If you've been interested in product metrics, you want to learn more, we're having a webinar with Amplitude at the end of June, so please join us for that. You'll be talking about north star metrics.
Tommy Noonan: Yep.
Hanna Woodburn: And we'll be talking with a PM from Amplitude about north star metrics too. If you enjoyed this content, please follow our LinkedIn page. We'll be back here next week with another LinkedIn Live. Thanks everybody.