How to: Get buy-in for your experiment
Part 2 of a four part series on how to identify, sell, run and report on experiments
How to: Get buy-in for your experiment ← You’re Here
Why some organizations are averse to experimentation:
You might have had the luck to work at an organization that doesn’t see experimentation as a valuable activity (you might be working there now). This write-up intends to provide some tools for breaking through that aversion and getting your experiments off the ground.
If you work at an experimentation-averse organization, the reasoning against experimentation sounds a bit like this:
“It’s too hard and too time consuming.”
“We already know the answer, why do we need to validate something we already know?”
“Experimenting is going to take away time from more important things that we need to focus on.”
“We need to move quickly and experimenting is going to slow us down.”
“This is going to create a poor user experience for our users.”
”We don’t have time for this.”
While the reasoning may sound different from situation to situation, it often translates to the same idea:
The organization believes that its idea-to-value flow is already effectively yielding results. As a result, the organization doesn’t see a reason to introduce more mechanics to this flow.
While in some cases product managers and other individuals can “boil the ocean” with their research and experimentation, in most cases experimentation aversion stems from a lack of exposure to effective experimentation and a sense of security towards the company’s past way of working.
To get past this aversion, experimentation needs to be demystified and better understood.
Opening the door to experimentation:
In my experience, there’s always a gatekeeper for experimentation. They can be someone who’s directly managing you and is accountable for your output and time or they can be someone who is the owner of a key tool you need to leverage in your experiment.
In order to get experimentation on the map you need to convince this person that your effort is worthwhile while simultaneously aligning on a reasonable cost investment for your experiment. The following are a few things to keep in mind when having these conversations:
Make sure your reasoning for experimentation is rock solid:
The key thing going into any conversation is to have clear-cut reasoning for the experiment. This means a few things:
You have a deep understanding of the issue you’re looking to experiment on but you’re blocked from fully understanding the issue due to missing key insights.
You’ve attempted to gain those insights via other suitable means and hit a wall and the experiment has a high chance of unlocking those key insights.
Having those insights will improve your chances of future success or mitigate significant risk.
You have a clear idea of how the experiment would work that is reasonable in terms of cost to setup and duration.
If the above items are covered then you’re starting your conversation from a position of strength.
Make sure to explain your reasoning for conducting an experiment:
In many cases, there’s an asymmetry of information between you and the gatekeeper when it comes to your work. While you’re spending considerable time in the weeds they might be highly focused on the “bigger picture” or overall not invested in your area of work.
It’s then important to establish a shared foundation of knowledge so that the gatekeeper understands why you ended up deciding an experiment was necessary.
Once that shared knowledge exists, it’s then time to explain why an experiment would be worthwhile (what risk you’re mitigating for, what upside you might verify, or overall what value you’re expecting from an experiment).
Ideally, by doing this, you’ll have built enough buy-in to experiment, but some situations require further handling and convincing. In those cases, you need to be more thoughtful about the gatekeeper's concerns and philosophical disagreements with your reasoning.
Minimize cost/loss-of-focus concerns by addressing them head-on:
Many organizations opt out of doing experimentation because they believe that the cost of setup for an experiment is too high. This leads to people opting to execute the project version of the experiment instead.
There is an obvious paradox here, the project version of an experiment is, almost always, significantly more expensive to execute. What organizations really mean here is that running an experiment has costs outside of the costs they generally pay and requires effort outside of the effort they generally exercise.
Here you have to align with the gatekeeper on what an acceptable investment looks like (in addition to the potential value and reasoning) since this will be an investment outside of the ones they’re used to making. If the gatekeeper believes you won’t over-invest they will often be open to the experiment (even if it’s just to appease you).
Think about this as a resource allocation problem for the gatekeeper, and work with them to identify the acceptable time and capacity allotment for the experiment. If the gatekeeper understands you’re being mindful it’ll make them much more open to trying things out.
Don’t make it a battle of beliefs. Instead, focus on the upside/downside of the experiment:
Many times the reasoning for not running an experiment ends up being about the gatekeeper’s belief around a key variable or idea the experiment is looking to validate. If the belief is not based on fact, and you can’t get the gatekeeper to realize that, then you might need to pivot away from beliefs and focus instead on the upside/downside of the experiment.
Here are a few pathways for discussion that can be more productive:
In cases where you’re experimenting to validate a feature. Focus on how much can be learned via the experiment and how expensive and risky it would be to only learn as a result of a release. The experiment may help you minimize scope, hone in on the most valuable aspects of the feature, and prevent you from having to do follow-up work.
In cases where the experiment is only seeking to increase your knowledge base, identify the possibilities that could come out of the insight and use those possibilities to help the gatekeeper open their mind to the experiment. Most people love discovering new things especially if those new things can help create more value, key in on that value as fuel for your experiment.
In cases where there’s risk, focus on helping the gatekeeper create a clear idea of the bad things that can happen if you move forward without experimenting. Make this risk very tangible to them so that they can imagine the world where the risk is realized. From here you can use experimentation as the risk mitigation tool that will help them prevent that risk from realizing.
Get other members of the team to lobby for the experiment:
If other stakeholders of the team also believe in the experiment ask them to voice this to the gatekeeper. Having multiple voices echoing the same ideas can have a compelling effect on the gatekeeper’s thinking about the experiment and its value.
Keep in mind that you’re not the sole owner of the experiment, the gatekeeper and the team working on the experiment are also owners of that work. If everyone rows together with conviction you’ll find that a lot of the friction in this process will start to dissipate over time.
Pick your battles:
In some cases, it might not be worth spending your equity on pushing experimentation. This is especially the case when the learning is low-upside and you’re still on shaky ground with experimentation as a concept within your org.
On the flip side, if you have something you really believe in, something that’s very polarizing in the organization, or something that feels very risky to the team then those tend to be good areas to invest in since you’re likelier to uncover insights and as a result validate the value of experimentation.
Being thoughtful about when to use experimentation can sometimes be the most valuable thing to counter aversion. Experiments with pay-offs are ultimately the best path towards combating aversion as they can be used as proof that experimentation is valuable.
Asking for forgiveness instead of permission:
It needs to be mentioned that there are pathways to experimentation where permission isn’t necessarily granted. Sometimes organizations work themselves into a dysfunctional workflow that prevents them from unlocking further value and become unable to improve on that workflow. In a few rare cases, learning the truth about something that is holding the company back is worth crossing a line. That being said experimentation with buy-in is always the best version of this work and the goal should be to make experimentation an aligned approach.
There are some key things here to keep in mind / avoid when swimming in these murky waters:
Don’t run an experiment for something you’ve been explicitly denied permission for (changing a small variable on your original plan does not constitute a totally different experiment either).
You’re just going to piss people off and it’s kind of an asshole move. The upside is seldom justified and the downside is massive.
Don’t do it as a practice since you’ll just get permanently shut down.
The undercover approach should be a rare occurrence, a card you can play when you really need it. If you keep doing it, again, you’re going to piss people off and come off as an asshole.
Don’t do it if there’s a real negative downside.
If you’re affecting customers negatively, if you’re putting another team in a bad place, if there are real negative consequences to your experiment then it’s probably not the right path.
Don’t talk about it constantly, don’t formalize it.
If you make a big fuss over the thing… You’re probably going to get the thing shut down.
Don’t overinvest.
What’s worse than doing something you shouldn’t be doing? Doing it a lot. Think about this experiment as low-stakes research that you’re doing on the side, not as your big project.
Know what you’re doing:
If you don’t understand your business, customers, teams, and strategy then don’t do it, you’re very likely to mess things up for everyone. Do the work before crossing the experimentation line.
Again, a bought-in group is always better than a group where everyone is going their own way. While experimenting without consent can yield results, it’ll almost never yield buy-in. Make people stakeholders and share the wins, this is most often the path toward an experimentation-focused culture.
Lexicon for convincing people to experiment:
I often find that a concept is only useful if I have the words to express it. With that in mind, I wanted to share some of my favorite lines when talking about experimentation and what they intend to achieve. Hopefully, this helps you build your own language and script for these kinds of conversations:
“While we have a good idea that this is true, an experiment will help us learn more about how to build it effectively saving us from having to do more iterations after launch.”
Why it works: You’re moving the conversation away from the assumption and instead focusing on adding value (by better understanding how to solve the problem and doing it in a cheaper way than releasing).
“I know that you’re not as high on this as I am and that to you this feels like a bit of a waste of time. Is there an amount of time I could invest into this that you feel would still be ok?”
Why it works: You’re disarming the gatekeeper by addressing a big concern head-on (by doing this giving them confidence you’re going to be thoughtful about this initiative). You’re then collaborating with them to find a suitable middle ground and asking them to problem-solve with you vs approve/deny you.
“If we’re wrong about this, why do you think that would be the case? What’s the worst scenario for this? What could we have missed? Would we be open to testing that to be sure?”
Why it works: You’re pivoting the conversation to focus on the downside instead of assumptions and using the feelings/thoughts tied to the downside to help fuel the desire to experiment.
“I really believe that I can find some valuable learnings here and minimize some potential risks… I’d like to spend some time on it unless you’re opposed to it?
Why it works: You’re making your belief and passion explicit and essentially asking the gatekeeper for an opportunity to pursue this passion. They may be inclined to allow you because they know you’ll be more bought in as a result of having been allowed to do this.
“The thing that makes me want to do this isn’t necessarily just getting to this answer, but it’s instead what this answer unlocks. Think about what we could do if we validate X… We could do this thing, that thing, maybe even this other thing…”
Why it works: In cases where the key insight isn’t as interesting to the gatekeeper but it is to you, focusing on what it unlocks helps the gatekeeper understand why you’re excited. Making the possibilities more tangible can help the gatekeeper see your experiment in a totally new way and as a result, be more bought in.
Parting thoughts on getting buy-in:
I’ve found most people are excited about uncovering new insights, being proven right/wrong, and preventing mistakes from happening. The experimentation process provides that potential for excitement and it’s partly your job to foster enthusiasm around that process.
Cultivate excitement with those that are experiment averse and you’ll sometimes find that they’ll become advocates of experimentation over time. With enough successful experimentation, the culture of the organization might start leaning towards experimentation.
While the road to get buy-in might be hard, it wouldn’t have been possible without the gatekeeper’s approval. Remember to always validate the investment the gatekeeper has made in enabling you to experiment as this will help them feel ownership of experimentation and can help turn them into a long-term believer.
Finally, remember that most people are trying to do their best for the customer/product/company/team. Think about gatekeepers as partners and you’ll find ways to grow that partnership. Think about gatekeepers as blockers and you’ll often find yourself running into their walls.