How to: Identify a good experiment opportunity
Part 1 of a four part series on how to identify, sell, run and report on experiments
Experiments can establish the foundation of new processes, features, and ways of thinking as well as save companies from making bad decisions. This series intends to provide a detailed guide on experimentation. It will likely be most helpful to people who are just starting to introduce experiments into their workflow.
How to: Identify a good experiment opportunity ← You’re Here
Why experimentation matters
It’s easy to confuse facts with beliefs because both of them feel the same and sound the same. However, while facts are by definition always true, beliefs are only sometimes true.
The ability to discern fact from belief is then a key skill to uncovering the world’s truth and as a result, ensure your product matches that truth.
Experimentation is one of the key tools that allows you to explore and map that distinction. It is, in essence, a tool for fact-finding. It is also, in the world of product development, generally a lower cost, lower risk, and lower commitment option than its alternatives.
A general framework for what experiments are good/bad for:
It’s important to establish that experiments aren’t the perfect tool for every job. Experiments are very well suited for some things but poorly suited for others.
The following is a list of the different things experiments are well and poorly suited for:
Well Suited:
Simulating how a feature may perform if built.
Iterating on a pre-existing solution.
Mapping customer behavior for a given scenario.
Exploring the productization of an insight (along with testing that insight’s validity).
Validating if a certain product or process can be removed.
Poorly Suited:
Solving a business problem long term.
Ultimately you’ll need a product/process to be created to fully solve most problems.
Understanding users, markets, behaviors, and journeys (in some cases).
Generally, you’re better off conducting user research than using experimentation to validate certain kinds of insights.
Additionally, if you haven’t looked at data and/or talked to customers then you might be skipping a step by running the experiment.
Simulating scenarios that are very long-term, intangible, or high risk to the company:
Certain scenarios are not well suited for experimentation, if there’s considerable risk to your company, if the insight requires a lot of time to validate or if you’re trying to test big intangible decisions like partnering/acquiring/merging with another company then experimentation is likely a bad tool.
If your experiment idea falls under the “bad” list then ask yourself: 1. What is the goal for the experiment 2. Is an experiment the best way to get there?
Additionally, I want to stress that it’s worth experimenting when the insight is worth knowing in the first place. If the insight is trivial or can otherwise be learned in easier ways (like data analysis, customer interviews, usability testing, etc) then experimentation may not be worth doing in the first place. This seems obvious but it can be hard in practice to make this deliberation
A strong hypothesis
This is arguably the murkiest part of experimentation because different people can have different opinions on what constitutes a strong hypothesis. I like to think that a strong hypothesis is a hypothesis that:
Is logical and at least a few others would think your premise has validity.
Is based on first-hand experience or second-hand reporting from customers of the conditions, customers, and environments at play.
There’s a reason confirming/denying the hypothesis is valuable, knowing this thing will give you the necessary information to make an important decision.
Types of Hypothesis:
This is not an extensive list of the types of hypotheses but some of the more common patterns that come to mind:
X happens because of Y
When exposed to multiple options, customers will pick X
The outcome is better if x is changed
What if X existed/no longer existed?
X happens because of Y
Example: People buy the Caesar salad because it comes with croutons.
In this context, the experiment is the method by which you validate that x does happen because of y and to what degree y can impact x.
Notice that I used a simple relation of cause and effect, this is because the majority of the time multiple relationships of cause and effect are very hard to test in an experiment.
If you’re trying to prove multiple relationships most of the time you’re better off designing multiple experiments and performing those independently as this will give you the best odds of getting valuable data from your efforts.
When exposed to multiple options, customers pick x:
Example: If provided with multiple kinds of salads to pick from people would pick Caesar salad.
In this scenario we’re essentially seeing what happens when we expose customers to multiple competing variables and we’re measuring which variables customers gravitate towards.
This is a tricky type of hypothesis because the results are very experiment design dependent. It’s easy to tip the scales towards one variable or another. However, in my experience, it’s a very helpful question to ask that if validated allows the business to properly weigh the importance of different variables in their product.
It’s also important to note that most times the experiment won’t give you the answers on why customers pick x, it’ll just tell you that they will. You’ll have to conduct research to truly understand the why.
The outcome is better if x is changed:
Example: We’d sell more Caesar salads if they came with smoked salmon.
This is a classic optimization hypothesis here you’re just testing that a change to the product would lead to improved results.
Again, valuable question to ask that is very dependent on the execution but when approached correctly can lead to real improvements in the product experience.
The size of X is also variable. Many times people think of optimization as only being able to change a single variable to avoid inconclusive results. In some cases, if your performance is poor you’re better off experimenting with a completely different version of your experience to try to establish a new floor of performance that’s higher than your previous experience. While you won’t know exactly why the new version is better, in some cases, you’re better off optimizing something that performs well than something that performs poorly.
What if x existed / no longer existed:
Examples: What if we sold Cobb salad? / What if we stopped selling Caesar salad?
If you’re thinking about building a new feature then an experiment can be a very valuable way to determine if you should follow through with your plan.
In this scenario, the experiment is then attempting to measure/understand how customers would interact with that feature if it was built by creating a simulated version of that feature.
On the other hand, experiments are also really useful to prove that you can sunset features that you believe are no longer viable.
In this scenario, you’re in most cases hiding or removing the feature from some/all customers and measuring the results (seeing if feedback comes in, seeing if there’s a change in engagement, etc). This de-risks sunsetting features significantly.
Following your gut (or someone else’s)
I tend to find that good experiments are driven by very strong feelings or very negative feelings.
If I have very strong feelings about something it tends to be worth digging further. This isn’t because I’m some sort of savant but because strong feelings tend to be indicators of somethingness. What that somethingness is or if it’s actually there is unknown, the experiment will bring that out.
That somethingness many times is a set of patterns, behaviors, trends, or mechanics that we’re unable to quantify or explain at the time but that can eventually form the basis of something in the business.
Strong feeling driven experimentation can often yield positive outcomes in one way or another:
When you’re validated in your feelings you can add more ammo to whatever case you’re trying to make.
When the results are negative you’re then faced with confronting the reality of the outcome and learning from it. Many times these poor outcomes lead to important changes in perspective and direction.
While positive strong feelings about something are valuable, negative strong feelings are just as valuable. Annoyance, frustration, impatience, insecurity, fear are all great indicators of somethingness. People can be very right but they can also be very wrong about things, flipping your mindset to take advantage of both your beliefs and anti-beliefs is an easy lever to increase the amount of opportunities you’re identifying.
Leveraging other people’s gut feelings
All of this goes for people you trust, I find that in any team there are a few people who tend to have a general prescience about things. Those people’s feelings can also act as a compass for the direction to focus on. It’s important also to factor in the different exposure areas for different people on your team, a designer, a developer, a customer success specialist, a sales person with the prescience trait will often have feelings about very different areas of the business or the experience. Actively seeking to hear their opinions and thoughts about the product and business can often yield valuable opportunities.
It’s also important to highlight that things that many people consistently believe have a fun tendency to be wrong (often very wrong). This is because shared beliefs are often built on a tradition of knowledge and that tradition of knowledge is often based on opinion. These are very good experiment areas because they are often located at core areas of knowledge of the business and identifying misalignments in those areas can help the business find the things they should actually believe.
I have found that central ideas to mine and other people’s decision making were in some cases inaccurate or generally incomplete. They made sense conceptually and felt completely rational and beyond reproach and as a result we assumed they were self-evident. When you get a big enough idea like that wrong, you can spend months or even years going down less valuable paths. To find these I ask myself questions like: “What would really suck if we were totally wrong on”. This often yields the highest value areas to validate.
Story time:
For a period of about 2 years I was convinced that there was a pricing problem with a product I was managing and I had a strong belief that our pricing was a key blocker in our growth.
I debated and lobbied for an experiment to test this assumption and eventually was granted the opportunity. We found a clear way to test multiple different pricing options (all lower in different ways) in a cohesive experience to the customer. This was a several month experiment measuring short term conversion rate and long term retention.
When the results finally materialized the original pricing I had been so strongly against was by far the best option far outperforming everything else both in conversion rate and retention.
After a good amount of analysis trying to disprove this reality I settled on a new reality - pricing wasn’t the problem. This insight was fundamental for me and others to move forward in new directions which did prove fruitful.
Had we changed pricing from the start we would have likely taken significant losses and learnt that lesson the hard way. Had I not done the experiment I’d probably still be arguing for it today convinced pricing was the blocker for the growth I was seeking.