How to: Report your experiment’s findings

Part 4 of a four part series on how to identify, sell, run and report on experiments

Nov 12, 2023

The experiment is complete… What now?

Once the experiment is finished, assuming everything about the design of the experiment and the tracking of the results was set up correctly, there are three key actions to take:

Analyze your results
Report your results
Identify next steps

Analyze your results:

Analysis is often a surprisingly challenging part of the experimentation process. While you’re looking to extract truth from data, you’ll find that it is often possible to use data to create many seemingly believable narratives. This makes it challenging to land on solid truth.

During analysis, you’re looking to create a clear separation between the core hypothesis validation that the experiment was designed around, from the other interesting factors that might have surfaced during the experiment. Focusing on your core hypothesis is a good way to stay grounded in your search for truth.

If the experiment was set up effectively this should be simple as you’ll have clear data indicators of if the hypothesis was proven or disproven. If your experiment was more open-ended, then you may not have a great deal of clarity but the experiment might have helped point you in a more promising direction.

The following are a few questions that can help navigate the analysis process:

Core Hypothesis Questions:
1. Was the experiment conclusive? Can I make a pronouncement on my initial hypothesis?
  1. Always go back to your initial hypothesis and analyze the data you obtained in context of that hypothesis (specifically if it proves/disproves it).
  2. If applicable, check for statistical significance to determine if there’s a clear result for A/B or multivariate tests.
2. What conclusions can I make about the experiment’s results that I would bet my life on?
  1. Push yourself to an extreme standard as this will help clarify the things you full-heartedly believe in, from the things that you sort of or mostly believe in.
  2. If you report assumption-riddled insights, you may lead your team down the dangerous path of developing solutions that won’t work as the underlying premise for those solutions is untrue. Ensure that you’re clearly separating the proven insights from the ones that haven’t been validated.
3. What things would perform exactly the same if a 3rd party were to re-run the experiment?
  1. This is another way to evaluate the results you’re seeing, this time through the lens of repeatability.
  2. Results that you strongly believe could be replicated, or that others could replicate in the future, are results that you can count on. If you have something that will perform consistently you might have the foundation for a new feature, product, or process.
4. How did the key metrics I was tracking perform? What could their performance mean in relation to the core hypothesis?
  1. Did the metrics perform according to your initial expectations or were you surprised? Is there a discernible pattern or are some metrics moving in contradictory ways?
  2. Look at the different possibilities your metrics are illustrating but remember the bar of truth you’re looking for. It’s ok for an experiment to end with new questions and some ambiguity as long as you’re acknowledging that that’s the case. Most experiments result in follow-up research being required.
Other questions worth asking:
1. Are there any qualitative aspects worth highlighting?
2. What variables were most surprising/unsurprising?
3. What would be the most interesting things to follow up on?
4. Were there clear outliers in the experiment?

Once you’ve answered the above questions, you should be capable of putting together a report on the experiment and determining what comes next.

Before covering reporting, there are a few things to keep in mind during analysis:

Inconclusive results

If your experiment was inconclusive, the first place to start is with the experiment design:

Did you build the right sample?
Did you set up an experience that effectively tested your hypothesis?
Were there variables in the experiment that you might not have been aware of initially that would have significantly affected the results?
Have you run the experiment long enough?

In some cases, you might have solid experiment design and still get no results. Generally, this means that there’s no significant change in behavior. If this is the case, that is a valuable finding in itself and should be reported as such.

Report your results:

Being capable of effectively translating your findings to a broader audience is one of the single most impactful things you can do as a product manager.
If you’re experimenting effectively, the information and insights that you’re uncovering are some of the most well-proven and reliable insights accessible to the business. As a result, you can have a strong effect on how the business operates and thinks.
To do this, however, you need to be very effective at reporting information. In many cases, you also need to spend a great degree of care and time putting together compelling resources that others can leverage to understand your work.

The key information items of a report:

At the end of the day, in most organizations, the stakeholders interacting with your experiment only care about the results and that the results are reliable (understanding process and methods is secondary for most stakeholders).

You’ll also find that getting your results understood is something that only happens through proactive action. The more friction there is to learn about your findings the less likely anyone is to learn them. The more ambiguity, the more likely each stakeholder is to come up with their own interpretation.

In terms of reporting, the key aspects to cover are:

The hypothesis
The experiment conditions
The results
The insights from the results

In practice, a report will sound something like this:

“We had the following hypothesis”
“To test that hypothesis we set up this experiment”
“The following things happened during the course of the experiment”
“Based on these results we have strong proof that the following things are true… We also believe that the hypothesis has been proven/disproven...“

To ensure your results are not lost, focus on clean, crisp, impossible-to-miss reporting. Ask yourself “What are the key things that everyone should echo after reviewing?” Focus on delivering those key things and minimize possible distractions.

Format and delivery:

Without proper distribution and packaging your findings will get lost in the shuffle. This is an essential aspect of reporting, the following are two (of many) ways to do this:

A written doc or email covering the information and data from the experiment is a low-effort way to formalize the findings.
- I’ll generally use this medium if the results are worth reporting on but the underlying thesis is not fully consolidated.
- I try to send these to any relevant stakeholders for the experiment (i.e. anyone who would care to/should read the report).
- To make this readable, look to highlight key portions such as the hypothesis, results and insights. Try to add a “too long, didn’t read” section at the top of your email covering the most essential insights to ensure at least something is retained.
A slide deck is a strong medium if the results are high leverage or if you’re at the culmination of a lot of research and work. Yes, slide decks have a bad rep but if you’re not in a reading-friendly organization (in most cases you won’t be) they can be a very effective tool to get people to quickly understand and absorb your work’s findings.
- The deck should cover (as mentioned above) the hypothesis, the experiment, the results, and the insights.
- Each section is 1-2 slides, contains the highest importance information, and is visually tailored (writing, assets used in the experiment, videos of user behavior, data visualization, etc) to best convey the idea behind the slide.
- You can always add an annex at the end with more detailed information on the experiment if necessary.
- Worth noting, the goal of a slide deck is not to run through a presentation, but instead to create an easily consumable report that can also serve as the basis for further discussion.

Regardless of the medium or distribution method for your insights, always ensure two things happen:

You have made your findings as digestible as humanly possible
1. The bar to strive for is: If everyone in your organization read your report and were asked to state the findings every person would say the exact same thing.
You have allowed people to digest findings async but have also established time to review them synchronously with key stakeholders
1. The async review lets stakeholders digest findings, the sync review often becomes a rich collaboration and discussion space around your insights.
2. Both are extremely valuable and lead to a successful internalization of the findings you’ve presented.

Identifying next steps

The next step for an experiment should revolve around one question: is it worth continuing down this line of thinking?

The answer to that question depends on the perceived value to be gained, the level of insight you have on the matter, and the context of the surrounding business. With that in mind, there are 3 possible paths after an experiment is analyzed and reported:

Experiment or research further
Pivot
Productize

Experiment or research further

This is a valuable path to pursue if you have:

Clear areas of further investigation that are promising and likely to lead to insights that will bring value to the business.
The buy-in from the organization to continue your research path.
No clear productization path for what you’ve learned or limited value in productizing at this time.

If you are to continue experimenting or researching then you should consider the following risks to your research path:

Loss of organizational buy-in
- Keep in mind when evaluating the option to do further research that there is often internal pressure. It’s easy to lose momentum and buy-in in some cultures the longer the research takes.
- Consider if the organization may be losing patience or feeling the pressure to move towards action. Selling is key to preventing this but sometimes so is acting (by productizing or taking other steps to unlock some of the value you’ve been experimenting around).
Limited future value
- While it isn’t necessary to have a complete understanding of how the research you’re doing will unlock value, it should at least be in an area of the business that is rich in opportunity.
- This means that there should be a reasonable expectation that better understanding the problems you’re looking at will unlock value in the future.
- To an extent, you should also consider the future viability of productizing around your research path. Specifically, there should be an aligned expectation (between you and the organization) that productizing around your research area is not only a likely outcome, but a shared priority.
Narrow focus
- Another thing to keep in mind when deciding to do more research is if you can start taking your findings and piecing them together into a larger theory.
- Shifting your mindset from a single hypothesis to a larger theory can help you find a broader, more impactful area of focus for subsequent work.
- Ask yourself questions like “What is the broader pattern that may be happening here?”, “If I distilled what I’m learning down to its simplest element, what would that be?” or “What would be the bigger truth here that could unlock a lot of value?” to get yourself into this mindset.

Pivot

This is will happen if:

The areas of further investigation are not promising or likely to lead to insights that will bring value to the business.
There is limited/no buy-in from the organization to continue your research path.
There is no clear productization path for what you’ve learned or limited value in productizing.

There are some key things to keep in mind when pivoting away from a research path:

Document and catalogue your findings
- In the future you may come back to your research as the context of the business changes.
- Throughly document your work and make it easy to find/search for. Future you will appreciate this.
Make sure stakeholders are aligned
- Many times stakeholders will have been invested in your research and will be surprised by a pivot.
- Ensure that you’ve built the shared understanding and buy-in to pivot effectively. Explain why the pivot is happening and clarify where the work will go from here.

If you consistently report and sell the value of your research you might be able to prevent pivots, block pivots or gain an extra bit of room to finish out your research. This is why formalizing findings and building alignment is key to longevity in your research focus.

If you are forced into a pivot, remember that this door may be open for you again and that your work has not been wasted.

Productize

Productization is the best outcome as this means your findings are valuable enough to be made part of the product or processes of the company. Productizing simply means taking the mechanics of the experiment and doing the necessary work to enable those mechanics to function or be used at scale.