Data-Guided Product Optimization: 4 Steps to "Always Move the Needle"
Find in Your Data Opportunities to Continuously Growth Your Results
👋 Hi! Nacho here. Welcome to Product Direction’s newsletter. As we approach the end of the year, I’m working with companies and leaders crafting their 2024 strategies, roadmaps and OKRs. If you want to know more about the workshops and coaching we do, contact me!
A lot of “product literature” covers topics related to creating new products: how to define a vision, how to do discovery, how to slice it for small iterative delivery, etc.
Yet a big part of our work is continuously improving and optimizing our existing product. Enhancing the interface to make users more productive, reducing friction points, optimizing messaging, etc.
While product teams will spend a lot of time on this activity, they tend to do it arbitrarily rather than following strong patterns to find optimization opportunities continuously.
So let’s unpack this topic in 4 areas:
Modeling your Product Metrics
KPI Trees and Dimensions: Slice and Dice data
How do I know what can be optimized?
Capturing Opportunities as Hypothesis
1. Modeling your Product Metrics
The first step to optimizing your product is appropriately understanding how it works and grows.
We usually have some initial understanding, considering that we can recognize underlying generic models in our products, like Funnels, SaaS User Models, Social loops, etc.
In some products, like e-commerce, this can be (at least at first sight) very straightforward, using the funnel framework to define the different steps and conversion rates in between.
Yet most products have evolved in complexity, even e-commerces, and started thinking about longer purchase journeys with multiple sessions and more complex customer lifetime value with loyalty and purchases over time.
How do you cope with this complexity?
Don’t try to copy-paste a framework. Build your own model.
Continuing the example, our e-commerce team had a few important loops in their product they wanted to reflect on the model.
They had a robust recurring traffic source, through retargeting and campaigns to continue purchasing past viewed items.
They included cancelations since it was a potential problem and conflicting KPI they needed to keep in line when experimenting with the funnel.
They had a Loyalty loop for users who purchased.
Why was it necessary to reflect this in their model? Because it helped them visualize how their product really worked, the growth loops and levers that interacted with each other, and to find opportunities for different parts of the experience.
Visualizing the model helps us align our worldview and the key growth levers for our product.
2. KPI Trees and Dimensions: Slice and Dice data
You may be wondering, is this enough?
The short answer is no. The model can help you with the overview, but to find detailed KPIs to optimize the product performance, you will need to slice and dice data.
2.1. Using KPI trees to find more granular metrics
As its name suggests, KPI trees are tree-shaped structures, with a high-level metric at the top, which expands its branches in the sub-metrics that add up to the root node.
For example, in e-commerce, we may have Revenue as top-level, which expands into Visitors, Conversion Rate, and Average Sell Rate.
We can further grow the tree by expanding the Conversion Rate into the sub-conversions of the funnel (Landing to Search, Search to Detail, etc.).
And we may go even further, by expanding the Conversion-Search-to-Detail, into different elements that affect it:
Search Response Time
Number of results per query
Click rate top 3 positions
By identifying the lower-level levers that affect our KPIs, we gain more chances of finding optimization opportunities. For example, it may be hard to see how to improve Search-to-Detail conversion, but if we find that the average response time is 20 seconds, then we have a clear area to optimize.
Furthermore, the tree can signal metrics we don’t have in the model and should be paid attention to. In our example, the e-commerce company didn’t consider in their model Average Sell Rate, but it was a critical optimization element to boost revenues.
2.2. Further refining with Dimensions
Thanks for reading ProductDirection! Subscribe for free to receive new posts and support my work.
Slicing our top-level metrics into more granular ones is also not enough. We also need to see data sliced “vertically” with different dimensions. Teenagers may behave differently than adults, mobile users will have different usage patterns than web users, and so on.
Typical dimensions are:
Traffic source: Organic, Email, Referral, Paid, etc.
Device / Platform: Web (browser), Mobile App (Android, iOS). Screen size or version.
Demographic (B2C): Geography, age, gender, socioeconomic.
Firmographic (B2B): Company age, size, serviceable markets, etc.
User segment: New vs. Recurrent (or Cohorts), Behavior (i.e., Buyers), or activity-based (i.e., heavy users).
Product Segments: Categories, verticals, type of service
You can define your variables depending on what is relevant to your product. Analyzing the previously identified KPIs through the lenses of these dimensions is critical to finding good optimization opportunities.
3. How do I know what can be optimized? (a.k.a. Using Dashboards Properly!)
In the first two steps, we reviewed how to measure our product.
Now, it’s time to find opportunities. Unsurprisingly, what we need to do is to compare data in different ways to understand the best chances to move the needle.
Dashboards are our go-to tool for gathering relevant information and finding valuable differences.
Dashboards not only highlight the most relevant metrics but also show us performance over time or different dimensions, in visually compelling ways.
3.1. What should we compare?
There are several ways to compare KPIs to identify where we may have opportunities for improvement. Let’s review 5 typical use cases:
Time-based: compare with previous periods, like Year over Year (YoY) or Week over Week (WoW). This is one of the most typical things built into dashboards and is critical to understanding how our product is doing. “Do we have a spike in users vs. last year? Yes, the new marketing campaign launched.” “Mobile conversion rate dropped? Yes, the latest app release introduced a bug”.
It’s worth noticing that this is usually more reactive. We see the status and react if it is not what we expected.
Vs. Target: similarly, we can compare our performance against our targets. If we plan to achieve certain revenue growth by the end of the quarter, what is our current result and projection? Are we on course, or should we try different tactics or focus areas?
Dimension-based: one of the most valuable ways to identify product optimization opportunities is to compare across different dimension values. “Why are mobile users behaving much worse than web users? We may need to improve the small screen experience.” “Why is this geography doing much better than others? What can we replicate? What is resonating with these users?”
As you can see, this is more proactive and more about fine-tuning or customizing the product experience to serve different users and use cases.
Cohort: used frequently for retention and subscription models, cohorts group users with common characteristics or behaviors. The most common one is for retention, grouping users per sign-up date. We want to see if users who sign up today and use the current version of the product are performing better than the ones who did it last week or month.
Competitors: finally, it is extremely useful to compare against competitors. If you are below your rival in conversion rate, you would have a strong indication that your product can be optimized, since others are serving users better than you do.
Unfortunately, this information is not generally available, but we have some proxies. Industry averages may be available. For example, in travel, Phocuswright publishes online traffic and conversion rates reports, segmented by vertical and geography. All industries have some type of benchmarks available.
Comparing across these options will reveal opportunities for optimization.
3.2. What if I don’t have a dashboard
In my coaching work, I face many teams who love these ideas but complain that they have no data analysts or dashboards available to put them into practice. I do not accept this excuse.
It’s a PM's (or, more precisely, a trio's) responsibility to have this information available. At the end of the day, a dashboard is a more complex visualization of a data table. So fortunately, we have all the necessary tools and skills to implement it.
The easiest MVP for a dashboard is listing all your key metrics, and start tracking them week over week on a table:
The effort required to build this table is setting up a few SQL queries (it can take your engineer lead a few hours max), and then each Monday morning taking 15’ to change the date parameters, run the query, and capture the result.
When you start seeing the value, you can extend it, adding results with different dimensions. The effort is simply tweaking the queries to have a few different groupings.
You have no excuse.
With a few hours of work, you can assemble everything you need to dig into your data and find ways to optimize your product’s performance.
4. Capturing Opportunities as Hypothesis
Now that we have (1) modeled our product growth levers, (2) sliced and diced the most critical aspects of our data, and (3) analyzed and compared values, we will have spots that we believe can be acted upon to improve our metrics.
We will typically create a task, user story, or similar “ticket” in any workflow management tool. For example: “Improve the landing page copy”.
This fails to capture why we are doing it and what we expect from it.
What we need to do instead is capturing it as a hypothesis:
If we change Z, Y will increase/decrease by X(%)
Example: “If we change the landing page copy, the number of signups will increase by 5%.”
This framing describes what we are trying to accomplish, and will allow us to test if we were right or wrong, to learn and improve in future iterations.
We believe that <this change>, Will <increase/decrease Y by X%> (for <target ser>),
Because <reason to believe>
“We believe that making the call to action more visible will increase signups by 10% for first-time visitors because current usability tests show is not well understood.”
A more complete version includes “why” we believe this change will have certain impact, improving communication and transparency about our reasons to believe and prioritize opportunities.
4.1 Combining Quantitative with Qualitative: the What and the Why
As you can see, this article is all about quantitative analysis of existing products.
But quantitative only explains what is happening. To complement the picture of “why” its happening, we need to complement it with qualitative analysis.
As in the last hypothesis example, we can see that the landing page is not converting as expected, but with a few usability tests, we can see that the call to action is not well understood.
Instead of randomly trying to change parts of the landing page, this combination of insights can help us succeed with the optimizations we implement.
4.2 Using Data to Create Hypothesis for All Opportunities
We have focused this article on finding opportunities in the data. But, as you know, opportunities would come from multiple sources: qualitative discovery activities, client requests, competitor moves, stakeholders input, etc.
Having laid the foundation for our data analysis, we can now create hypothesis for any incoming option, and have more clarity on what we are expecting to achieve with it.
This 4 steps cover foundational work that all product teams must do to become outcome-oriented, and gain both understanding and confidence on how they can deliver more value to their customers in ways that improve business results.
That is why I frame it as one of the first “transformational milestones” for product organizations in my maturity model.
If you have already implemented part of it, your goal now is to enter a continuous improvement cycle. We can always improve what and how we measure, compare and define hypothesis, increasing the sophistication to achieve even higher rewards.
I hope that was useful and practical!
Disclaimer: For those of you who want hands-on help putting it to practice, I cover this topics on one of my most popular in-company workshops: Data-Guided Product Optimization. If you want to learn more, contact me!