This article focuses on finding the right scope for your projects. It won’t go into detail on how to make trade-offs or the mechanics of refining scope. Instead, it will dive into what makes for good scope and how to evaluate if you’re correctly weighing the variables.

Scope = Value + Timeline

Scope defines the volume and depth of solutions that are to be built to achieve an outcome. Scope is not something you want to define in full upfront; instead, it is a range of acceptable solutions that is continuously refined until you’ve reached a deliverable.

At its core, scope is defined by two key questions:

• What specific value are we trying to create?

• How long/much is the business willing to invest into extracting this value?

These two questions anchor the entire process of determining scope. The Goldilocks zone for scope is the place where your solution extracted an outcome while staying within the business’ tolerated investment level. When we correctly answer these two questions we end up with projects that are likely to succeed. Over-scoped or under-scoped projects are often projects that fail to effectively answer one or both questions.

Value

Starting with value, value is the benefit to be gained for both the customer and the business as a result of pursuing the project. In other words, it’s the reason we’re doing the project.

One of the most important insights I've gained about value is that it’s easy to mistake function for value. The failure to understand this distinction can doom your project before it begins. Let me illustrate with a series of examples:

• Example 1

  • Function: "I want to put water in a cup so I can drink it"

  • Value: "I want to feel hydrated"

• Example 2

  • Function: "We need a messaging system with channels and direct messages"

  • Value: "Teams need to communicate effectively (whether that’s in a 1-to-1, 1-to-many or many-to-many setting) so that they can collaborate effectively"

• Example 3

  • Function: "We need to display 20 different metrics with filtering options"

  • Value: "Analysts need to quickly gain insight into issues so that they can make data-driven decisions quickly"

This distinction might seem obvious, but it's crucial. When we focus on function, we lock ourselves into a specific solution. When we focus on value, we open ourselves to innovative approaches we might have otherwise missed.

Because scope is a range of acceptable options, we look for value to be precise in terms of benefit but ambiguous in terms of implementation, as this allows teams the freedom to explore solutions until they find the right one.

The first key to finding your project’s scope is ensuring that you are rock solid on the value you’re creating and that you’ve decoupled the function from the value as much as you can.

Timeline

Timeline is how long the business is willing to fund the value extraction. Every business has a patience threshold - even when they say, "Take as long as you need." they actually mean, “Take as long as you need within reason.” I try to convert this appetite level into a range of time explicitly. For example:

• "We need it ASAP" = Days-3 weeks

• "Take the time to do it right" = 1-2 months

• "Whatever it takes" = 4 months maximum

While most people hate having timeline awareness, I’ve found that these constraints are net-helpful because they remove options and force you and the team to really focus on what is required to hit the goal and what is not.

Note: A timeline is not the same as a deadline; timelines are guidance, and deadlines are restrictions; thinking of timelines is a better mechanism for teams to process scoping decisions as deadlines overpower value; you would rather the two be seen as linked, but where necessary, value always beats timeline.

Value is the target, timeline is the appetite

While value provides you with a clear target, a timeline tells you how long you have to hit it. A team with clear value targets will often hit these targets but take far too long to do so, a team with timeline constraints will hit a timeline but miss its target, a team with both will find its way to a successful feature.

It is essential that a team understands the answer to these questions and is bought in. I’ve found that conflict during projects is often tied to a misalignment around these questions or a lack of explicit answers.

Finding scope in practice

You will find a more in-depth article about the process of refining scope here, but at a high level, this is how I go about this process:

I always start by translating my project goals into simple, high-level user stories that focus on value rather than implementation. I then work with others/leadership to understand the acceptable investment level and develop my sense of what that looks like in terms of timeframe and how flexible it is. This gives me a clear picture of what value we’re trying to create and how long we’ll have to get there.

I will then take that context and bring it to the team, explaining my reasoning and beliefs in the process. This sets up the foundation for team exploration of a potential solution, which usually follows a general path of:

1. Open ideation around identifying possible solution paths

2. Evaluation of options against value and time constraints

3. Picking the strongest starting point solution that delivers the core value

4. Exploring that starting pointing and developing it into a lo-fi solution that validates if the path is worth continuing

5. As we gain confidence that the path works, prioritizing the list of enhancements needed to achieve a release.

Fundamentally, the process of refining scope is focused on exploring paths and quickly and effectively identifying the best one. By doing this, you permit yourself and the team to make scoping decisions along the way as you gain more information as opposed to all upfront (which often leads you to bloating releases and then having to cut scope at the last mile). This incremental path allows your scope to gradually get narrower and narrower until you’re left with the ultimate vision of what your solution should be.

Find scope along the way, nail down scope at the end

It is my belief that scope should be finalized as late as possible while being defined as gradually as possible. In an ideal world you’re getting to 100% final scope at about 80-90% completion of the project. This allows you and the team to continue being flexible and making decisions about what matters late into the project.

Now this doesn’t mean you have defined scope and are cutting at the end, it means that you have options and are choosing which ones make sense within the release and which ones don’t. To achieve this outcome, here are some practical tips I've found valuable:

1. Always start smaller than you think you need to.

   1. The smaller the starting point, the easier it is to move and make decisions, as there will be less tying you down. Always start in the order of most important and add along the way. It’s always easier to add scope than it is to remove it.

2. Add scope incrementally as you confirm progress.

   1. Oftentimes, teams make decisions before needing to - this leads to bloat. Allow yourself the pain of needing something, as that pain is the clearest guide you’ll have for scope. Focus on making incremental decisions that expand the depth and volume of the solution as these decisions become necessities or forks in the road.

3. Never delete, just backlog and prioritize for follow-up releases

   1. Teams will often fight tooth and nail for scope because they fear this is the only chance they’ll have to build it. Create a culture where each release is not an end but instead is a stepping stone, this will allow the team to make scope decisions more comfortably. Question every "must-have" feature against your value definition and keep a list of “nice-to-have” features for future releases.

4. Make scope decisions (and their stakes) shared across the team

   1. The more the team feels responsible and empowered to achieve value within the timeline, the better the decision-making will be, and the easier it will be to find the right solution for your problem. As much as you can avoid being the sole individual responsible for scoping decisions.

Parting thoughts:

Managing scope is one of those things in product management that will never stop being a challenge, as each project has its own world of scoping decisions. While a framework can help you more consistently navigate this opportunity, be ready for each challenge to have its own unique quirks and necessities.

Over time, finding the Goldilocks zone for scope becomes more natural. The key is reminding yourself to always stay focused on value while maintaining the flexibility to adjust as you learn.

Early in my career, I inherited a team that had been burned by previous product initiatives. Features would mysteriously vanish from roadmaps, requirements would shift without explanation, and promises would evaporate under pressure from leadership. The result? A deeply skeptical group that viewed product management as an unnecessary layer of bureaucracy.

Sound familiar? Many PMs face similar challenges. The key to overcoming this skepticism lies not in defending the role's existence but in demonstrating its value through consistent, principled action. Here's how to do it.

The 7 Principles of Building Product Management Authority

1. Build Trust Through Predictability

The foundation of product management authority isn't about being liked - it's about being trusted. I learned this lesson the hard way, having spent my first years oscillating between trying to be everyone's friend and clinical detachment. Neither worked. What did work was establishing a consistent, predictable approach to:

• Decision-making: Using clear frameworks that teams could understand and anticipate allowed teams to predict where I’d stand and adjust accordingly.

• Communication: Maintaining transparency about what's changing and why and updating teams regularly helped teams focus on their work.

• Commitment management: Following through on promises, no matter how small, even when the cost to me was high, gave teams faith in my word and its weight.

I found that when teams can predict your behavior and methods, they become easier to work with. This predictability removes ambiguity and builds trust, which evolves into respect. Teams quickly lose respect for managers whose actions seem arbitrary - they'll always prefer someone predictable over someone unpredictable, even if they disagree more often with the predictable one.

2. Master the Art of Collaboration

Product management isn't about being the smartest person in the room - it's about orchestrating collective intelligence. The faster you develop your ability to get people to work together effectively, the faster the team will understand the value of product management. Here's what effective collaborative leadership looks like in practice:

• There are forums for input about projects and decisions and those forums feel appropriate and heard

• Discussions feel free-flowing and shared across many team members as opposed to dictated by the PM

• Team members understand and trust specific individuals to navigate challenges within their domain and communicate their processes with the team

• There is a strong sense of accountability across the team without fear of repercussion or blame

Every team member should have input on product value; the key is finding the structure and moments to make that collaboration happen. An effective PM will quickly show their value through their ability to make teams work better together.

3. Demonstrate Unwavering Resilience

For teams to believe in product management, they need to see that our principles hold up under pressure. If product management betrays its principles at the first encounter with friction, then it will fail to inspire trust in the team. Withstanding pressure generally means:

• Maintaining consistent processes even when facing setbacks

• Using failures as learning opportunities rather than excuses

• Pushing through organizational resistance with principled persistence

• Showing that setbacks strengthen rather than weaken our approach

When teams see that you remain steady and focused on improvement regardless of circumstances, they begin to trust that they can adapt to your leadership style and methodology.

When teams believe their PM will not waver and will keep pushing to improve things, they will accept that reality and adapt to it. By being unrelenting, you’re sending a message to your team that they can safely adjust to you and that they can trust that you will be consistent regardless of what comes your way.

4. Champion the Long-term Vision

One of product management's most valuable contributions is maintaining focus on the bigger picture. In a detail-oriented environment, having an individual who can broaden the perspective is often very helpful in improving team performance. This is something technical teams often realize and seek out. Specifically, owning and articulating your product’s vision impacts the team in different ways:

• It provides context for decisions and trade-offs

• Teams are motivated by understanding their work's impact

• It creates a shared sense of direction

• It helps teams understand the "why" behind decisions

• It empowers teams to filter and debate decisions through a shared lense

Most individuals have a strong inner desire to be part of something bigger than themselves. When teams believe their mission is worthy, they will find ways to elevate themselves to meet their goals.

At its best, product management is about getting people to see the potential in something and believing they can meet it together. By providing this value to your team, you will indirectly illustrate to the team that product management brings an important intangible variable to the work, and they will value it more as a a result.

5. Develop Craft Knowledge

Nothing undermines PM credibility faster than strong opinions paired with shallow understanding. While having opinions is core to the role, speaking confidently without understanding fundamentals undermines credibility.

You're often working with people whose entire career focuses on their craft - they need to see you value and respect it to accept your input as valid. Here there are several ways to get better:

• Invest time in understanding technical fundamentals through courses, books, projects

• Learn continuously from team members' expertise by proactively asking questions, especially when you feel those questions are dumb

• Seek to participate in projects that challenge your technical knowledge in new ways

The key here is that you accept that the subject matter you work with is a world in itself and that you have a commitment to being knowledgeable about that world and respecting it for what it is. Your world can be software, hardware, or anything else. The key is that you commit yourself to becoming someone who understands the inner workings of that world and is constantly trying to build a better understanding of how they can be better at the creating process.

6. Deliver Measurable Results

Product management ultimately depends on results. To believe in it, teams need to see tangible, differentiated value from your approach. When teams win, ideas become easy to implement and buy into; when teams struggle to produce value, teams start isolating and fighting against their structure.

There are many, many ways to get better at delivering results (many of my past articles address this problem directly). However, there are some easy strategies you can pursue to highlight how practices are tied to results:

• Implement value forecasting and project retrospectives to identify how/if practices are leading to more consistent value delivery

• Show your team how proper project scoring leads to better prioritization and decision-making through examples and work sessions

• Connect product management practices to concrete results in tangible ways through stories and scenarios

Remember: you're a business person first, equipped with tools and perspective to drive value creation through collaborative leadership. Your actions and, most importantly, your results will determine whether others see product management as essential or expendable. By making product management practices well understood and tying those practices to tangible results, you can help build a shared belief in the value of your role.

7. Become a Product Evangelist

Effective product management requires more than process expertise - it demands genuine belief in your product's potential impact. Being a hardcore fan of your product helps teams become bigger fans of you. This is because authentic passion for your product:

• Reveals to the team that you care about what is being built and that you will care for the work they do

• Inspires teams to push beyond immediate obstacles and, in doing so, builds a connection with the team

• Creates emotional investment in outcomes, which often fuels creativity and innovation

• Builds resilience during challenging periods and creates a “we’re in this together” mindset

The industry has many mercenary PMs - those motivated primarily by compensation and advancement. Truly effective product leadership requires being an evangelist who can inspire others even when sacrifices are needed. Your genuine belief in your product's impact and approach to building it can spark similar dedication in others.

Authenticity matters here; if teams believe in your commitment and passion, they will be infected by it and carry it on. Passion is a fuel source for creativity and collaboration, and that fuel will open the team up to product management practices as they will relate these to a desire to make better things.

Parting thoughts:

Building belief in product management isn't a one-time achievement. It's built and rebuilt through daily actions, decisions, and interactions. Teams may be initially skeptical of product management, but consistent application of these principles can transform that skepticism into trust and eventually into belief.

I've found that organizations don't really change their mind about product management through theoretical arguments. They change their mind through experiencing effective product management in action. Each time you demonstrate trustworthiness, show deep craft knowledge, and maintain long-term vision while delivering near-term results, you're not just doing your job - you're showing what product management at its best can be.

The future of product management isn't determined by industry trends or thought leaders, It's determined by how we practice it every day. Focus on embodying these principles consistently, and the value of product management will become self-evident to your organization.

Every product team faces an endless flood of questions, problems, and possibilities competing for attention. Yet, in my experience, successful releases ultimately come down to answering just 1-3 critical questions. Get these right, and everything else falls into place. Get them wrong, and no amount of perfect execution can save you.

This article will show you how to identify these critical questions, align your team around them, answer them decisively, and use those answers to drive successful and faster execution. We'll cover:

• What critical questions are and why they matter

• The three types of critical questions you'll encounter (value, solution, technical)

• How to find critical questions as a team

• A practical framework for answering these questions quickly

• How to know when you've found your answers

• Moving from answers to execution

• Getting organizational buy-in for this approach

What is a critical question:

A critical question is the make-or-break challenge that determines a feature release's success or failure. While releases are complex, their outcomes usually hinge on answering just 1-3 fundamental questions correctly.

This insight has reshaped how I approach releases. Instead of trying to solve everything at once, I work with the team to build a release strategy around answering these critical questions quickly and decisively. By doing so, I’ve found you can derisk the entire release early and consistently drive value. Once you have these answers, they become the foundation for all other decisions, accelerating development and increasing your chances of success.

Critical questions typically fall into three primary categories:

1. Value extraction: Is it worth it?

This is the most common type of critical question. It asks whether your project's core premise is valid and if you can deliver meaningful value to users through it. The answer might be 'no'—and that's valuable to learn early as it allows you to either pivot toward real value or stop the project before heavy investment.

Projects often fail not from poor execution, but because they conflict with user realities, expectations, or needs. Solving this question allows you to prove a release’s worth without the cost of a full cycle behind it.

Example variations of the question:

• Will users consider this worth their time?

• Is the core assumption the solution is built on a true assumption?

• Will the value be immediately obvious to users?

• Can we get this feature sold to the right people?

• Can we get people to experience valuable results every time they use it?

• Can we maintain enough quality content to power the experience?

• Will ongoing support costs outweigh the value delivered?

• Is now the right time to push this feature through?

2. Solution shape: How will it work?

This type of question comes up when your solution's form is so crucial that getting it wrong would prevent users from experiencing any value. This becomes critical in projects where workflow integration or user friction can make or break success.

Example variations of the question:

• Will this fit seamlessly into existing user workflows?

• Can users complete key tasks consistently and efficiently?

• Can we minimize friction in critical paths?

• Can we make the experience engaging and feel rewarding at key steps?

• Will this coexist with related features and systems?

• Do we have the right framework for how this experience should work?

3. Technical approach: Can we build it?

Here, we’re trying to determine whether a technological decision completely defines the solution and makes it possible or impossible. This question often appears when the solution that is desired is at the cutting edge or when there is a significant constraint that may make the solution’s cost, reliability, or capability untenable.

Example variations of the question:

• Can we make this feature really fast and performant?

• Can we scale up this feature to the demands of our user base effectively?

• Is the technology we’re using to build this feature mature enough?

• Can we support this feature with our existing team’s knowledge and skill sets?

• Are we going to regret this approach in 6-12 months?

Finding critical questions as a team

Now that we’ve covered the common types of critical questions we can run through how to identify them within a release and go about solving them.

Identifying critical questions isn't a solo sport—it requires your entire team's perspective and insight. Before diving in, ensure your team has a shared understanding of five key areas:

• The Opportunity: What specific challenge are we addressing?

• The Context: Why is this worth solving, and why are we uniquely positioned to solve it?

• The Value: What concrete benefits will our customers experience?

• The Usage Scenarios: In what real-world situations will this solution create value?

• The Business Constraints: What practical limitations around cost, integration, and scale must we consider?

Really take the time to validate and refine these knowledge areas with the team as nailing this background will improve your odds of identifying the critical question on the first try.

Crafting the question:

Once your team has built this shared context, you can begin crafting your critical question. While you can have up to three, aim for one powerful question that captures your core challenge. Your critical question should be:

• Concise: Expressed in one or two clear sentences

• Specific: Focused on a single-core challenge

• Actionable: Something you can validate through experimentation

Have a session with your team framed around identifying what matters bring with you your thoughts on what the question(s) is and why you feel that way. Then, ask the team to express their own thoughts. If you don’t have alignment, continue to push through and debate ideas until you have it. Do not compromise halfway, as this often ends up being ineffective; instead of splitting the difference, empower people to go and make something to illustrate their point, or if that’s not possible, pick an initial approach and keep open the promise of reviewing the need to pursue the alternate direction if needed.

There are often strong opinions and lengthy debates, which means there’s passion behind the project, which is always valuable. Don’t feel the need to suppress that energy. Instead, focus on channeling that passion in the direction of “let’s prove this out” and give the most passionate people the opportunity to make something (if feasible). If two team members have very strong competing ideas, have them prototype their approaches and let those approaches speak for themselves. You’re looking to cultivate “a best idea wins” environment, not a “best argument wins” environment. If you have team members who are more silent or don’t enjoy participating in public debates, try to have private conversions or have them write their thoughts down first and bring their ideas to the table. Focus on getting everyone involved as this will give you the most angles to evaluate.

Also, don’t worry about getting the question perfect immediately, and remind the team that evolution will happen. Trust your team's instincts—you're better off starting with a good enough question than getting stuck in endless debate. You can always refine your question as you learn more. As long as the team feels you’re generally in the ballpark, you can probably get started.

Answering the critical questions

Once you've identified your critical questions, it's time to find answers. But here's a crucial mindset shift from most project approaches: you are not pursuing perfection or conducting exhaustive research. Your goal is to answer these questions as quickly and efficiently as possible.

Teams often fall into a trap at this stage. They get caught in endless cycles of debate and persuasion, spending weeks or months discussing theoretical solutions. Meetings pile up. Documents grow longer. Everyone has an opinion to share. Before you know it, you've spent more time talking about the solution than actually exploring it.

But here's the uncomfortable truth: you can't argue your way to a great product. No amount of passionate debate or perfectly crafted presentations will tell you if you're on the right track. The only way to genuinely answer your critical questions is to put something real in front of stakeholders (and often users) and have the solution prove its worth. This means creating tangible artifacts early—prototypes, mockups, proofs-of-concept, or simplified solutions that test your core assumptions. These don't need to be perfect; they just need to be real enough to generate authentic reactions and insights.

Create, Play, Discuss, Evaluate (Repeat)

The heart of answering critical questions lies in rapid cycles of experimentation and learning. At its core, this cycle has four key elements:

1. Create something tangible, even if it's rough. Shoot to build a proof-of-concept or work through a very basic implementation of the thing. The key is that it's something people can react to, not just discuss theoretically.

2. Play with what you've created. Get it in front of people—team members, users, stakeholders—and let them interact with it naturally. Give them alone time with it in the order of days. This hands-on interaction often reveals insights that no amount of explanation could uncover.

3. Discuss what you've learned, but keep it focused. What surprised people? What excited them? Where did they struggle? Is this looking viable? These real-world observations are worth more than hours of theoretical debate.

4. Evaluate honestly whether you're getting closer to answering your critical question. Is your confidence growing or wavering? Are you seeing patterns in the feedback? Does the team's energy and excitement build with each iteration?

5. Take all the new knowledge gained and apply it to create a better version of what you had before. Keep going until you have something the team really believes in.

What makes this cycle powerful is how it transforms abstract questions into concrete learning. Instead of endlessly debating possibilities, you're constantly creating and testing hypotheses. The solution becomes a living, evolving thing that the entire team can see, touch, and improve.

This approach also tends to energize teams. There's something inherently motivating about seeing your ideas take shape and evolve based on real feedback. Each cycle builds momentum, replacing the fatigue of endless discussion with the excitement of discovery.

Remember: be ruthless about throwing away anything that doesn't directly help answer your critical question. There will be plenty of time later to polish features and add capabilities. Right now, your only goal is to find that answer as efficiently as possible. Every piece of effort should be laser-focused on building confidence in your direction.

Knowing when you found the answer: The 70-80% Confidence Rule

After several cycles of creation and learning, how do you know when you've sufficiently answered your critical question? In my experience, you're looking for 70-80% confidence in your answer. But this isn't about gut feelings or arbitrary thresholds- it's about recognizing specific signals that indicate you're ready to move forward.

Signs of 70-80% Confidence:

• Your proof-of-concepts validate core assumptions about the solution.

• Team members consistently tell the same story about how the solution will work and why.

• Stakeholders show genuine enthusiasm for what they’re interacting with.

• User feedback consistently validates your core assumptions and approach.

• You can point to specific, realistic scenarios where your solution delivers clear value.

• Everyone feels confident that the questions you came into this process with are answered.

Warning Signs You're Not There Yet:

• Team members describe fundamentally different solutions when asked about the approach.

• Key team members express lingering doubts about the current direction.

• Fundamental debates about the approach keep resurfacing.

• User feedback remains mixed or unclear about the core value.

• The team struggles to articulate how users will experience value.

• Your team isn’t aligned about the shape of the questions being answered or about how well you answered them.

Once you reach the 70% confidence threshold, project execution often becomes almost intuitive. The team shares a strong, unified understanding of where you're going and why. Decisions become clearer because you have a solid foundation against which to evaluate them. You can move faster because you're no longer questioning the fundamentals.

Getting to 70-80% confidence doesn't mean having everything figured out. You'll still have open questions and details to resolve. Trust that if you've followed the process and seen these positive signals, you've built the foundation you need for successful execution and move on through the rest of the project.

From Answer to Execution: Riding the Momentum

When you start seeing promising signals, it's time to shift gears. The momentum you've built to answer your critical question is valuable. Use it. Continue exposing work and gathering feedback, but now with a broader lens. Your incremental builds can tackle bigger chunks of the solution, always staying grounded in the answers you’ve arrived at.

Execution brings its own set of complex problems to solve—but here's the difference: you're now solving them from a position of strength. Your team shares a deep understanding of the core value being delivered and a clear vision of where you're headed.

Keep these principles in mind as you move forward:

• Stay iterative: Break down the remaining work into smaller, focused challenges.

• Maintain momentum: Keep the team moving and building, and use exposure to the product to generate this momentum.

• Stay grounded: Regularly reconnect with your core answers and validate what you’re doing against them.

• Remain flexible: Be ready to adapt as you learn more, possibly even revisiting your original answers if needed.

• Keep learning: Continue gathering feedback and seek to increase the scale and depth of exposure for your work-in-progress solution.

Is this approach easy to execute? Not at all. Constant discipline is required to maintain focus and make tough choices. You'll face pressure to chase interesting but peripheral problems. You'll be tempted to revisit settled questions. You'll want to optimize details before their time.

But if you can maintain your focus on what truly matters—if you can keep cutting through the noise and stay grounded in your core answers—you'll find yourself building products that not only work but also truly resonate with users.

Case Study: Building Social Proof Through Search

The following case study shows how this framework played out in a real project. It illustrates how focusing on critical questions helped us build a valuable search feature efficiently.

The Context:

The Opportunity: We have a huge network of users but people who don't know us are often skeptical of the size of that network, and, as a result, they're skeptical of the value of being part of that network.

The Context: This problem is worth solving as it will improve the odds that users sign up and join the network. We're uniquely positioned to solve this problem as we have the strongest network and very clean/complete user data for members of our network.

The Value: Users with this concern will feel that they're getting honest, transparent answers as they'll be able to find people they know and validate that the network is worth joining.

The Usage Scenarios: When users come to our website from Google searches or any other entry-point, they're able to undertake an in-product search for someone they know.

The Business Constraints: We need to support search behavior at our current traffic volume and beyond. We need to ensure that we're delivering valuable results each time and minimizing the frequency of poor/few results. We need to update our underlying data frequently to ensure searches show updated results.

Finding the Question:

From here, the team had a long discussion where different stakeholders pursued investigations in their domain. Design spent time thinking about how this experience integrates with our site, engineering looked at different options for building the search and prioritizing results, and product spent time mapping the most common search patterns likely to be reproduced on the site.

The team then identified one core question:

• Can we deliver accurate, valuable search results at high rates?

This was the critical question that defined the release’s odds of success. For the feature to work, search results needed to be consistently valuable.

Answering the Question:

The team found that the most effective way to answer the question was to:

• Build a dummy page with an unpolished-looking search function

• Tie that search function to our production DB so people can search within the actual user DB that real users would use

• Focus primarily on the search prioritization logic and test that logic for specific search patterns we identified

The team spent about five cycles just iterating on search prioritization logic and ended up with something that very effectively solved the real search cases we believed users would undertake. We also opened up the search prototype to other people at the company, and these people tested it and started using it to solve their own separate use cases.

Eventually, the team decided the search logic was strong. It wasn't perfect, but pretty effective and strong enough to advance. From here, the team moved to the execution phase, breaking down the remaining release into a sequence of items (building the final UI, building remaining backend functionality, and integrating it onto different pages).

This case study shows how focusing on one critical question - can we deliver valuable search results consistently - helped us validate our core assumption before investing in the full feature build. By using a stripped-down prototype, we could iterate quickly on what mattered most while avoiding getting stuck on less important details early on.

Getting organizational buy-in for this approach

Teams often resist changing their workflows, even when current processes aren't working well. At first glance, the critical questions approach might seem inefficient since you're not planning the full solution from day one. Here's how I've successfully sold this methodology and gotten it adopted:

Start with pain points:
Most teams have vivid memories of building the wrong thing—projects that consumed months of effort only to miss the mark with users. Use these examples to illustrate the cost of moving forward without answering fundamental questions first. Make it clear that focusing on critical questions early dramatically reduces the risk of these expensive mistakes.

Emphasize better decision-making:
The critical questions approach gives teams tangible evidence about whether a solution works before major investment. This concrete understanding leads to faster, more confident decisions about scope and direction compared to trying to plan everything upfront.

Prove it through practice:
Rather than pushing for widespread adoption, start with a single project. Use this as your proof of concept, regularly sharing progress and wins with stakeholders. Let the results speak for themselves—teams that see this approach in action quickly recognize its value.

While this methodology might not feel necessary for teams that are already shipping quickly and effectively, it's particularly powerful for organizations stuck in slow, waterfall-style processes. It can provide a practical path to more efficient, focused product development while reducing the risk of building the wrong thing.

On the other hand, a balanced roadmap pursues multiple different strategies (often concurrently) in an attempt to improve its odds of succeeding. This article isn’t about the content of the roadmap but instead focuses on its form. Specifically, it focuses on the sizes of projects to be pursued and the quantity of each type.

This article makes the case for a more balanced roadmap-building strategy as well as how to pull it off.

Balancing the roadmap: rocks, pebbles and sand

You might have heard of the empty jar concept before, I’ll extend it here to fit the context of a product roadmap. Think of your roadmap as an empty jar that you have to fill. To fill it you have rocks, pebbles, and sand.

• Rocks: These are your large projects, usually new features, revamping old ones, or tackling large tech debt projects. Usually around 1-3 months of work.

• Pebbles: These are your midsized projects, usually small features, feature improvements, performance enhancements, and other mid-sized iterations. Usually around 2-3 weeks of work.

• Sand: These are your small projects, usually smaller feature improvements, adjustments, and other small iterations. Usually around 1-1.5 weeks of work.

I draw the line at rocks as anything larger is very prone to failing. A project longer than 3 months to me is a mountain, and you might as well break it down into rocks, pebbles, and sand. What this model implies is also that everything, at the end of the day, is the same matter, just broken down to a size that maximizes your ability to extract value and get that thing done.

Given these elements, what you want to do is give yourself the opportunity to take big swings but derisk those projects with smaller projects and iterations. The big swing gets you the home run, but by pursuing some smaller projects, you’re still liable to get a single or double per plate appearance even when you miss a big swing.

Rocks

The roadmap can only really start with the anchor pieces and these anchor pieces are your rocks. If you start by filling the jar with sand and pebbles, you’ll quickly run out of room for the rocks. You’ll then find yourself emptying the jar to get the rocks in. If you put the rocks into the jar first, you can fill the remaining empty space with pebbles and sand.

Commit to your large projects first but make room for the rest. If you’re in a rock-obsessed culture, start by trying to create a commitment of only 50-65% of your time spent on rocks with the rest spread across pebbles and sand. The roadmap will need all three kinds of work to achieve balance just as the jar will need all three elements to be full.

Don’t fall into the rock trap:

There is often a lot of temptation to only try to fit rocks into the jar but you’ll find that the jar simply can’t fit too many rocks and that you’ll feel that a lot was left on the table.

In most people, there is a natural bias towards large-scale projects. There are a few reasons behind this:

• Large-scale projects are generally easier to visualize, describe, and arrive at,

• They don’t force you to make tradeoffs (when the scope is large enough, it’s always easy to just add another thing to make the project make more sense),

• It’s easy to picture the outcome of a large-scale project and even easier to imagine the upside being transformative to the business.

Unfortunately, large projects cost a lot and take a long time to achieve (which reduces your ability to pursue other projects), given the scale, they require making lots of assumptions, which creates lots of room for error.

Large projects are worth pursuing but spending 12 months on 3-4 projects for most teams is a terrible idea as you’re really hoping those 3-4 things hit big and are giving yourself very little room for iteration. When you don’t hit big the downside can be catastrophic.

Pebbles and Sand:

You want to have a lot of room for sand and pebbles for a simple reason: The more shots you take, the higher the projected point total you’ll have. This doesn’t mean doing a bunch of crappy work, just that when all things are the same, you’re better off doing more than less. The easiest way to do this is to do more mid-to-small sized things.

This will require a big commitment from your organization: The entire system depends on you having room for small and mid-sized work. I’ve found this buy-in to be the single biggest challenge to roadmap building as most people, even when they say they want iteration, naturally only want large-scale projects. However, the sand and pebbles are what will give your product the ability to evolve quickly and react to opportunities and feedback and, as a result, are essential.

To make this even harder there is a really important commitment you’ll have to secure: when it comes to sand and pebbles, don’t lock those projects in ahead of time. You just want to commit to your rocks and get commitment to an open road of other things that will be picked from the highest-ranking items in the sand and pebble queues when the time comes. This allows you to be reactive and iterate based on what’s happening. By not planning ahead of time you allow yourself the flexibility to respond to today’s problems, not yesteryear’s ideas.

Sustainable jar-filling practices

Focus on the goal, not the implementation:

When someone reads the words that describe or identify your project remember that in those words is coded a set of ideas and that that set of ideas can be different from person to person. This can lead to disconnects about what the project is, what it’s trying to achieve, and its overall value. This disconnect hurts your ability to get projects done.

Try to define and identify your projects as much as you can by the goal, problem, or use case that it is setting out to address (vs by a code name or feature/implementation details). Ultimately, the team, through working on the project, may arrive at a completely different solution than what was imagined at the beginning of the project. By avoiding a focus on implementation, you give people the ability to hone in on the why of a project, not the how. This also ensures that anyone who comes upon that project has a higher likelihood of understanding the value it is looking to extract.

Create lots and lots of optionality:

The best way to improve your roadmap’s likelihood of success is to give yourself a lot of optionality with your investments. Optionality (in the roadmap-building context) is the ability to control investments gradually and to decide to quickly double down or cut back when things aren’t working.

To put this idea into the context of rocks, pebbles, and sand: Sand and pebbles give you a lot of optionality because the investment period is shorter, whereas rocks lock you into investments for longer.

You’ll find that it is much easier to do multiple pebble and sand projects back-to-back within a certain area than it is to do a rock and follow that up with sand and pebbles. This is because teams lose enthusiasm and momentum the longer they work on something. The joy of shipping is the one thing that can refuel teams in their pursuit of a long-term solution to a problem. As a result, roadmaps that pursue a lot of smaller-scaled iterations have the ability to stay committed to a problem for longer than roadmaps that are looking to solve the same problem with one large release.

More is more:

It’s better to do a single project at a time with quality than two with no quality. However, it’s often better to do 2 projects, each at 80% of their potential, than 1 project at 100% of its potential. Ultimately, you want to increase the volume of shipping until you hit a wall with your team’s ability to still deliver with quality. Sacrifice perfection (note perfection, NOT quality, as quality is essential) in lieu of velocity and volume. You can always improve upon things gradually but the door often closes if the investment proves to be too long.

The key to success with this strategy is to be able to tackle multiple items in parallel; the more concurrent initiatives your team can pursue in parallel, the easier it is to slot smaller-sized work. I try to have at minimum two independent projects in progress at all times for any given team; ultimately, this depends on the team, the team size, and your ability to support parallel work with quality.

Break things down:

You have to start by decomposing your roadmap before projects start. I’ve often found that most rocks are just collections of pebbles, and most pebbles can be made into sand. By tying a potential project to a size that best reflects the business’ appetite to solve that problem, you are starting projects with a healthy constraint. This honesty allows teams to accept the limitations of a project and work within those limitations. As a result, any change in the shape of a project is additive.

The more you shift things down a tier, the better your odds get as well. Often times downshifting is really just removing the parts of a project that have risk or are less proven and turning those into their own smaller stake endeavors.

Give yourself and the team the true constraints for a project and you’ll often find that this will improve the likelihood that that project gets picked up as well as the effectiveness with which a team can solve that particular problem.

The mental and emotional value of breaking things down:

Roadmaps composed of only large projects are naturally prone to lower vibrations. Large projects tend to be late, large projects tend to require cutting, large projects are left often unfulfilled in order to meet a release date. A large-project-oriented roadmap is a roadmap built through subtraction. You are constantly subtracting your way to a solution. Teams come up with complete solutions that are then cut, at best, early in the process and, at worst, close to a release. This is both inefficient as well as draining for teams. Editing is a key part of the process, and all editing is about sacrificing, but healthy editing feels like getting more for less, whereas unhealthy editing feels like being conned out of something.

In an environment that is more properly balanced, you can begin shifting towards an addition-based roadmap. Here, you’re often finding ways to start small, and over the course of a project, as a result of enthusiasm, discovery, or any other insight, you end up adding to your work. Addition is a highly enjoyable experience like receiving a gift; you did not expect it, but you’re glad it came your way. Editing in this context feels like trading for something better and it’s the most enjoyable kind of editing.

Adjust the bar to entry to match the investment:

Teams often apply consistent rules to every kind of project they pursue. These can be incredibly rigorous validation processes to gut-driven gamble-based approaches.

The problem with this is not in the process itself but in the fact that a one-size-fits-all mindset creates an implicit bias towards projects of a certain kind. For example, In a highly rigorous scoring culture, only rock projects are pursued because sand and pebble projects are often too fuzzy (or worse their perceived value is too low to warrant all the effort it takes to score).

To avoid this, I adjust the bar to entry for projects based on size:

• Sand projects only require philosophical alignment and some key stakeholders liking the idea.

• Pebble projects can also get through with just philosophical alignment and stakeholders liking the idea. However, we can ask for scoring for a pebble if there is contention.

• Rock projects require scoring unless they play a key strategic role for the business or have a really high degree of belief from key stakeholders.

Ultimately, the bar you create is up to you, but the important idea is that you adjust it deliberately to incentivize the kind of balance you’re seeking.

The challenge with information is that it (1) requires regular collection, (2) requires proper cleaning to be useful, and (3) both collecting and cleaning are time-consuming.

This article goes into how to acquire information on a regular basis and how you can attempt to do so efficiently.

“The universe is only as large as our perception of it. When we cultivate our awareness, we are expanding the universe. This expands the scope, not just of the material at our disposal to create from, but of the life we get to live.”

— Rick Rubin, The Creative Act: A Way of Being

The feedback factory

In most businesses, there is an abundance of customer contact. The reality is that, in many cases, the daily contact your business has with customers covers a very broad and very deep slice of the customer experience and perception of your product. It includes:

• Product data that is created by users using the product,

• Support interactions between your team and customers,

• Interactions your sales team has with customers,

• Reviews, NPS surveys, social media interactions,

• In-person events.

In addition to this contact, there are also specific efforts you can undertake to gather additional information, including customer interviews, surveys, analysis, usability tests, experiments, and gradual feature roll-outs, among other things.

The main challenge is that many times this rich stream of feedback never makes it to your desk, and as a result, you’re left with incomplete information and blindspots about your product and business.

The question is, then, how do you sustainably set your team up to gather information at scale, process that information, and have it feed your decision-making? The answer is in building a feedback factory.

Building the feedback factory

A feedback factory is a model for systematizing the routine acquisition and processing of feedback at scale. To build the feedback factory you have to assemble the three core systems and ingrain these systems into your culture. These three systems are:

1. Collection

2. Cataloguing

3. Socialization

Collection

A key idea behind building a feedback factory is to try to expand the ways you can capture valuable customer input. By tapping into all sources of contact with customers you’ll be able to collect more feedback than you thought was possible.

Collection includes any acquisition of information about your customer, business, market, or products, and the storage of this information. There are 4 core mechanics that make up a great collection system:

The feedback portal

You want to create a space where both team members and customers can log any feedback or insights. The feedback collection portal setup is simple: you want one or more forms that team members or customers can fill out that links to a database where you store all logged feedback. You then want to permeate the usage of this form to everyone at your company and use it whenever there is applicable customer contact, whether it be used on calls, emails, ticket reviews, or even integrated directly into your products.

Ultimately, by expanding this approach to every customer contact across your business, you’ll end up building a small window into those interactions, and through that window will come meaningful nuggets from which you can build projects. This will become the most abundant form of feedback that you have stored. It’ll be composed of short messages about your product experience but collected at high levels of volume allowing you to effectively quantify and qualify the intensity of problems and opportunities.

Contact with customer-facing team leads

Connecting with team leads, I’ve found, is a great way to get high-level feedback, but it is also how you build a shared commitment to feedback collection. This is because team leads are the ones who can keep individuals outside of your team committed to capturing and logging feedback.

You’ll want a weekly or bi-weekly sync dedicated to their domain area where you can learn from their customer contact and understanding of the business line. I’ve found this recurring cadence strengthens the bond between teams and gives you the ability to quickly course correct if feedback collection is lacking.

Research integration into the software development lifecycle

Once you move onto a world where you’re trying to solve a specific problem you need to collect information in more targeted ways. The key here is integrating gathering feedback and information into the software development process.

Specifically, it means building the time and appetite to do validation/discovery activities at the beginning, middle, and end of projects. Not all projects need research, and when they do, they might only need it at certain stages. However, without the ability to dig in during a project, you’ll often find that you’re missing the necessary information to make correct scoping and problem-solving decisions.

Practically speaking, this means determining what kind of validation (if any) your approach to a project needs, what information you’re missing that could enable better decisions, and deploying efforts designed around capturing this information.

This concept is an article on its own, but some examples of this are interviewing customers before a project starts, doing usability sessions, and running early access programs (to name a few). These exercises can be critical to your decision making process and make up a key part of collecting feedback.

Reviewing data on a regular cadence

It’s easy to think of feedback as verbal customer communication, however, any customer reaction is feedback. I believe data often reflects customer sentiment, preference, concerns, and confusion about the product. Data is a form of non-verbal feedback and is just as important as feedback delivered verbally by customers.

You want to track key metrics from your product experience on an ongoing basis so that you can develop a more cohesive understanding of the area you manage. I look at certain datasets daily (often every few hours), certain data sets weekly, and certain data sets monthly. By building these regular checking windows you’ll end up building a sophisticated understanding of the area you manage and will be spurred to undertake investigations that can generate valuable projects.

Helping others understand what feedback is:

I’ve found that people often discount information that is valuable feedback because they don’t see it as feedback. It sounds obvious, but it’s worth remembering: how is someone supposed to collect feedback if they don’t fully know what feedback is?

I like to describe feedback in the following way:

• Feedback is any information pertaining to how customers experience your business. Fundamentally, any experience that a customer expresses about your product, services, value, and so forth is feedback. Confusion about your product is feedback. Anger, love, and anything in between is feedback.

As you can surmise, almost every single customer interaction contains feedback. This can be overwhelming for people in customer-facing roles, but the more they buy into this idea and the more motivated they are about that information being used, the better off your business will be in terms of its ability to make decisions.

Cataloging

Ok, so you have tons of information now. New problem, how do you make that information accessible weeks, months, or even years after it was originally captured? Well, the answer is painful but necessary - you need to catalog information and create and maintain libraries.

There are four keys to doing this:

Put as much as you can into the feedback portal.

The more you concentrate your knowledge base in a single place the less things are likely to get lost. The feedback portal is one such place and it has the distinct advantage that the information making it into the feedback portal is getting there via form.

Make sure that your forms are proactively capturing information that can be used to catalog insights. Specifically, ask form fillers to provide user type, feedback type, and product area along with the feedback itself. Doing so will save you hours of feedback re-processing.

Create re-usable knowledge hubs.

As you build more and more feedback materials (interviews, write-ups, analysis, etc) the challenge is how to make it intuitive for people to find this information and remember to store any new information in that same place. It is essential for a sustainable feedback factory that the information you’re collecting does not get scattered. There are a few key ways you can group these materials effectively.

1. Group by research activity type: Here, you’re grouping information by the type of feedback-gathering activity that was used instead of the content area. Think about a hub where you can find sections containing all of your interviews, all of your surveys, all of your usability sessions, etc.

2. Group by specific projects or features: Here you’re creating a hubs specific to projects or features. This allows anyone who is looking for information about that specific feature or use case to easily find previous research efforts around that area.

3. Group by user types: Here’s you’re grouping materials and information by specific user types that you work with allowing anyone to search for information by that user type.

Ultimately the type of aggregation you do depends on your business, what information is most valuable, and how you often go about solving problems. That being said, the key is that you are grouping information in some way so that it is easier for others to find it and so that people always know where to store additional information.

Try to consolidate your knowledge to a small number of resources

The more spread out your information is the less likely you are to use it and the more likely you are to forget you have it. As much as you can, try to keep a small number of places where information lives as this will help create sustainable patterns around collecting, cataloging, and retrieving information.

Try to have 3-4 main structures where your feedback lives with key differentiation between them. In my experience, a model where you have the feedback portal (for short-form insight), knowledge hubs (for theme specific longer form insight), and metric trackers (with data on a specific domain area) has been an effective practice as I always know where to find insights based on the kind of information I’m looking for.

Routinely clean and reorganize insights

Maintenance is an essential component of a productive feedback-collection machine. You need to allocate time on a regular basis to undergo feedback cleaning, reorganization, cataloging, etc efforts. I often schedule deep cleans during slow times during the year and find that this practice yields cleaner spaces but often also yields new insights as I’m looking at feedback with a fresh perspective.

Socialization

The final piece to this system is the socialization of feedback, its use, and embedding it in your culture. This is the heart of the system, and without it, the rest of the feedback factory will fall apart.

Simply put, the feedback factory exists to help make better product decisions, if it is not clear that this is the case then the factory itself will cease production as individuals will stop being motivated to collect feedback.

Socialization is the routine broadcasting of the usage of collected feedback to power the roadmap. For me, this works in a few ways:

• I routinely make projects in the backlog of the roadmap based on feedback items that were logged and tie the project to those specific items. This allows others to see feedback be used to foment actual projects and see in a very practical way me making use of their feedback.

• I routinely make use of captured feedback to help drive the product development process and onboard team members onto a project. This sets up product development teams to expect and seek out feedback to make decisions and allows them to speak about their projects using the language of our customers.

• I routinely highlight the specific knowledge items that helped drive the execution of the roadmap project as a project comes to a release. We often follow up with customers and team members directly about how their feedback was used as we deliver a project. Team members and customers alike find this to be valuable and feel that this shows that their opinions are listened to.

• I routinely acknowledge the usefulness of feedback in large company or team-wide meetings. I specifically highlight the impact certain logged feedback had on our ability to make good decisions, avoid bad ones, and evaluate the full set of possibilities. I also deliberately call out specific individuals who have done a great job at gathering feedback as this (1) rewards them publicly and (2) creates an incentive for others to collect feedback.

By consistently making use of feedback you’ll find it clearer to define what you need in terms of future feedback as well as motivate those capturing it. The more explicit the connection of feedback to decision-making, the higher the buy-in level for participants in the feedback process.

Parting thoughts:

Any roadmap-building process is based on information. The less information you have the weaker your decision set is. The more information you have, the more options are available to you, and as a result, the more likely you are to arrive at the right option.

To ensure you have information, you need a company-wide buy-in to the importance of collecting feedback. To pull this off, you’ll need really good working relationships with other teams, and they’ll need to trust you in your ability to execute and make use of the information they’re collecting. You’ll also need to build a system where you can collect and catalog feedback at scale and keep people motivated to support that system.

In a functioning organization the whole business is bringing you the information you need to make good decisions, leaving you with a better foundation from which to make these decisions and putting the onus on your ability to evaluate and prioritize. This structure has been essential to my roadmap for many years, and hopefully, it will be to yours too.

I firmly believe that when used correctly, standups help projects stay on track and help identify issues long before they escalate. I’ve spent a long time trying to figure out with others how to make standups valuable and purposeful. I believe the current formula I use works pretty well, so I want to share it with you.

This article covers what I believe is the foundation of a productive standup.

The key to a great standup is touchpoint-spotting.

Standups should focus on one thing and one thing only: Identify any necessary touchpoints in the workflow to drive higher release quality.

By focusing on identifying anything that needs to be discussed to improve the likelihood of a successful release (whether that discussion happens during a standup, in a separate meeting, or asynchronously after a review), the standup becomes a routine checkpoint that enables high-leverage conversations to happen.

What are those touchpoints? Here’s a list of frequent touchpoints I’ve identified during my standups:

• Identifying blockers,

• Requesting feedback,

• Asking others to review progress,

• Demo stuff,

• Identify the need for and schedule pairing,

• Build alignment on something,

• Requesting context on something,

• Align on a deployment/release approach,

• Align on a testing approach,

• Align on priorities,

• Talk over a timeline and the likelihood of meeting it/the need to adjust it,

• Go over testing results.

• Asking for help,

• Anything else where contact helps improve the likelihood of a successful release.

As you can tell from the above Standups are NOT recitations of what you did. if we don’t have any of the above touchpoints to talk through, then the standup ends then and there. Standups are about necessary touchpoints and necessary touchpoints only, if there’s no necessary touchpoint then there’s no need for the standup.

Standup Mechanics:

Now, there are a lot of touchpoints listed above, but the reality is that covering those touchpoints in standup is pretty simple. I follow a straightforward model built on a loop between touchpoint-spotting and next-step identification.

Touchpoint-spotting mechanics:

• I’ll start standup by asking the team if anyone has anything they want to connect on; if they do, we start there (unless I have something I believe is critical). If there’s nothing, I’ll start.

• When I start, I’ll often pull up our work board so that everyone can focus on a shared visual document of our current work. I’ll work backward from the work that’s closest to releasing to the work that’s furthest from releasing.

  • I’ve found this ordering significantly improves a team's release rate. It creates urgency around getting stuff out and establishes an implicit mental priority for work (first what’s close, then what’s next).

  • Again, if there’s something critical, however, I’ll prioritize that first, or if there’s a meaningful discussion to be had, I'll prioritize that, too.

• As I cover the board, others and I call out any work items about which we have questions, concerns, or information that needs sharing with others. Most of the time, this leads to deeper conversations.

• We don’t go over everything we’re working on, instead, we deliberately focus on the things that people actually feel the need to talk about. This makes the standup efficient and valuable. Again, if there’s nothing that qualifies then we end the standup early.

Next step identification mechanics:

• Often, as we identify these touchpoints, long discussions ensue. The key to a time-effective standup is identifying the things that should discussed fully during the standup from the things that need additional time outside of it.

• When starting these discussions, it’s okay to let things breathe a bit if people are making progress and the progress is valuable and relevant to everyone. Also, letting topics breathe gives you the ability to identify whether we should fully discuss something or if it’s something to be covered later. This decision is mostly subjective, and it’s your job to figure out consistently what should be closed during that time slot and what should be moved.

• If you realize you’re running out of time or if there’s no clear end in sight, politely let people know that this item might just need a follow-up as we’re running out of time to cover everything.

Note: Keep your average to 15 minutes:

You can sometimes run over, but you don’t want standups to become planning meetings, as people will start losing their appetite and endurance for them. You want them to primarily serve as ways to identify touchpoints, resolve the ones that can be quickly addressed in the standup, and schedule and align on any touchpoints that need their own space.

Standups should take an average of 15 minutes. If they’re consistently under or over 15 minutes, the team either isn’t surfacing enough items, or you’re not effectively identifying and scheduling follow-ups to items that surfaced and are instead just handling them within the meeting. Some days will be light, and some days will be heavy, but in the aggregate, your team should be spending about 15 minutes a day on touchpoint identification and assessment.

Closing Notes:

Everyone hates meetings, and rightly so; meetings can often devolve into a weird kind of theatre where everyone pretends that what’s being talked about matters while everyone knows it doesn’t.

Standups are not performative, they are not an exercise in bureaucracy, and they are not needed if they’re not needed. If there is nothing to talk about, then at best, standups are just a way to see everyone each morning. If there’s a reason a touchpoint can help make things flow smoothly then it’s a place where you can have that conversation.

Standups, like other process tools, can be effective when used correctly and detrimental when not. To make a standup valuable, focus on creating a meeting built around identifying needed touchpoints. By doing this, you may find that the standup can aid flow, create a forum for team members where they can be vocal about projects, and, most importantly, have a tool that can be used to identify if projects are starting to get off track.

• Interactions with stakeholders become more productive as stakeholders are consistently well-informed about the state of things.

• Meeting times and meeting volume are cut down since you need to do less context-building during meetings and instead can focus more on solutions.

• Stakeholders start addressing issues much earlier than before and voicing their opinions more consistently.

• You’ll gain a better understanding of your product and the ebbs and flows of your projects, which will allow you to improve your practices.

This article covers the current model I use (which was workshopped by the PM team at SportsRecruits and through the feedback of many teammates), why I’ve found it effective, and how I’ve tried to keep it valuable week after week.

Weather Reports:

Weather reports are simple; they’re an update on how things are going in your domain, clearly highlighting which areas have clear skies and which areas have stormy weather.

The report is concise and uses clear imagery (through weather emojis) to help others quickly screen and prioritize information.

By doing this, weather reports allow teams to quickly gather key information about your roadmap, ask questions, dig deeper, and, most importantly, act quickly.

Weather Report Dos:

1. Focus on the key things: Highlight the top 5 things on your roadmap that you want stakeholders to be aware of and potentially involved in.

2. Be concise: The more words, the harder it is to find what matters. Brevity is essential for the report to be read by others.

3. This is where discussions start: Think of the report as the starting point for discussions, work sessions, and other connection points and not as the endpoint.

4. Emojis for focus: Use weather emojis at the beginning of each bullet to highlight how you’re feeling and make your report easy to skim:

   1. ☀️ ”Sunny” This is going great!

   2. 🌤️ ⛅ 🌥️ ”Partly Cloudy” There are aspects of this that are going well, and there are aspects that I’m unsure about because they need definition or have the potential to become issues.

   3. ☁️ ”Cloudy” There are a lot of aspects that I’m unsure about.

   4. 🌧️ ”Rainy” There are issues here. We should be able to resolve them, but I want to point them out.

   5. ⛈️ ”Storming” There are some serious issues here. The team is going to need some help to resolve these issues.

   6. 🌪️ ”Tornado” Oh shit! We need to figure this out.

5. Keep a cadence: Publish these weekly so that everyone builds a routine of checking these for information.

6. Make it interactive: Include videos, links, and demos to make the team's progress tangible and ideally interactive for anyone wanting to learn more.

7. Ask for feedback: Make sure to ask for feedback on the value of your reports and what others would like to see in them. This will help you refine your writing over time.
   

Weather Report Don’ts:

1. Oversell or overpolish your update: You should report on the ground truth, especially if it’s challenging, so that others can step in and help. If you hide a problem, it’s on you if it gets worse.

2. Skip weeks regularly: The more consistent you are, the likelier issues or opportunities will be spotted. It’s okay to skip it sometimes if you’re totally overwhelmed, but the report should come first in most weeks, given its impact on others.

3. Overwrite. The more information in a report, the more that information turns to noise. There’s a sweet spot between too much and not enough. You’re trying to find that sweet spot every week.

4. Overwork and don’t do it if you and others are not experiencing value. Ultimately, every workflow that fails fails because it requires too much from people or because there’s no belief that it is valuable. Pursue reporting with the mindset of creating value but doing so efficiently.

Example weather report:

🌧️ Outage related to transcoding bug

• What happened:

  • The transcoding queue didn't correctly prioritize jobs for about 3 hours, from 11:30 to 14:30 EST.

  • This led to around 526 video files getting stuck in the queue without finishing, affecting 256 users.

  • This issue was a result of the recent change to the prioritization approach for transcoding.

• What we’ve done since and what’s next:

  • We have since processed these jobs and implemented a failsafe for future failures.

  • However, this failsafe will not work in the long run, so we’re going to schedule work to better address the issue and its underlying causes.

• Additional Links:

  • For a technical breakdown of what happened and what the failsafe is see this link: {{link}}
    

🚀 New data cleanup tools

• The new data cleanup tools are officially out this week! They will help reduce time spent on integration-related issues where user data is misaligned.

• Additional Links:

  • Here’s a guide on how it works: {{link}}

  • Here’s a demo of it working: {{link}}

🌤️ Updated user signup flow.

• I'm currently running inventory on the release to determine if there’s anything left, but, likely, all of the work is officially out of dev and in team testing/ready for QA.

• We’ve found some areas of code around the account creation endpoint that need to be refactored, but we’ve agreed to do this post-release.

• Additional Links:

  • Relevant tickets: {{link}}

  • See designs here: {{link}}

  • Try it out in this demo environment: {{link}}

🌞 New and improved favoriting feature

• This week, we had a design review on the latest progress and discussed the technical approach. The review specifically covered cleaning up the existing system, adding the ability to bulk favorite, and permeating favorites across other features.

• From here, we’ll start ironing out the first release and continue to iterate across the different areas of the project. As the initial release definition starts coming together, we should have a much more robust update next week.

• Additional Links:

  • Check out the technical approach doc: {{link}}

  • Check out the current designs here: {{link}}

Scoring is the process through which you can add some rigor to your roadmap building and, as a result, improve your odds of success. Overall, scoring brings a lot of positive dynamics to your team:

1. It forces you to define value: Scoring pushes you to describe value in tangible terms. This adds rigor to the way projects are selected and how they extract value. Over time, this habit will improve your ability to identify high-upside projects, and it might also spread to others across your organization.

2. It helps shape projects in ways that improve their odds of success: By scoring projects, you’ll more consistently identify why some projects are unlikely to succeed and mitigate those risks ahead of time.

3. It creates clarity and alignment about the value you’re trying to unlock: Scoring allows everyone to understand the value to be extracted by doing the project by clearly describing that value. By understanding this, others can better form opinions about how to best tackle the project and debate on whether that value is worth the investment in the first place.

With all this in mind, I want to propose a simple exercise you can use to score projects quickly.

Value = Reach * Conversion Rate * Confidence Level

A project’s value is almost always defined by the combination of two factors:

1. The addressable market of your solution (Reach)

2. The conversion rate of that market to your solution (Conversion Rate)

Ultimately, these two variables are kind of all you need to score any project. There are many frameworks out there (most of which work), however, I’ve found that with these two variable I have 90% of what I need to pick projects with a relative degree of accuracy.

The final variable is Confidence Level, which, simply put, mitigates opinion and ego. By putting reach, conversion rate, and confidence level together you have something that easily describes the value of any given project (as long as that project is defined by a metric of some kind).

Let’s dig into these variables a bit more closely:

Reach

• Reach describes the market/opportunity size of your solution. Reach is not tied to impact, instead, it looks to quantify the number of addressable entities for your project.

• Reach is measured in the number of people, events or time, impacted for a specific period. The time period is variable and generally depends on your company’s payback period for initiatives.

• Sometimes projects have reach outside of a single dimension, for example some projects reduce retention while also improving organizational efficiency. I often score projects against different dimensions to put forth the clearest argument for its value.

Conversion rate:

• Conversion rate measures the % of your reach that the solution will actually solve for. Conversion rate (or win rate) is essentially your gut feeling of how effective your solution will be.

• Discussions about conversion rates often lead to realizations about your ability, or lack thereof, to extract value.

  • This ends up being the heart of value discussions as reach tends to be more easily agreed upon. If everyone agrees on the reach, conversion rate is then a tangible way to talk through impact for a project and align expectations.

• Conversion rate is also a tangible metric which can be figured out (through experimentation, research, etc) which will often allow you to further advance the likelihood of a project even if it’s rejected upfront.

Confidence level:

• Confidence level is an easy way to ensure that opinions are weighted against good practices.

  • The confidence level is set based on if you’ve undergone specific activities that derisk the project and, as a result, are more confident that the project’s value is what you say it is.

• These are the options available for Confidence level:

  • Supported by Experiment / Existing Feature = 90%

  • Supported by Feedback / Product Data = 67%

  • Supported by Market Analysis = 45%

  • Supported by Vision Alignment = 23%

• As you’ve probably surmised from the above, the project’s value expectation suffers the more fuzzy its foundation of insight is. This creates an incentive to derisk projects by pursuing activities that naturally increase confidence in your problem and its execution.

• Confidence level can be an especially good tool for getting past situations where highest paid person‘s opinion wins. Projects that are only backed by opinion will naturally score worse and be harder to defend whereas projects that have been derisked in some way will score best.

Example Scoring:

Let’s put these variables into practice. Below are 3 different projects scored using the above variables:

1. Signup Optimization Project:

Let’s say you’re thinking about a project related to optimizing a key step within your signup process.

• Reach:

  • In this situation, reach is the total number of people who attempt to sign up but don’t complete it within a time period.

  • More specifically, let’s say it’s the number of people who bounce during a specific step that you’ve identified is problematic. Let’s say per year it’s 20,000 people.

    • Reach Used: 20,000 over 12 months.

• Conversion Rate:

  • In this situation, the conversion rate is the expected percentage of people who attempt to sign up but don’t complete it, but who would now complete it because that step’s friction is removed.

    • Let’s say that you believe that you can convert about 50% of the people you’ve identified in reach by addressing the issues with that step.

    • Conversion Rate Used: 50%.

• Confidence Level:

  • Let’s say you’ve done a funnel analysis of signup and drop-off points, you’ve localized the problem with data, and you have a sense of why it happens but haven’t really tested it or talked to customers).

  • Confidence Level Used: In this situation, you’re using the 67% confidence level as your score is supported by product data

• Final Score:

  • 20,000 people * 50% of those converting via signup * 67% confidence level = 6,700

  • If you pursued this project you’d generate 6,700 incremental leads signing up every year.

2. Value Expanding Feature Project:

Let’s say you’re building a feature that expands the value of your business by introducing a way to solve a problem you haven’t addressed.

• Reach: In this situation, reach is the total number of existing or new customers that could potentially become customers of this new solution over the course of 1 year.

  • Let’s say it’s 500 new customers at an average ARR value of $10,000. 500*10,000 = 5M.

  • Reach Used: $5 Million in ARR signed up within 1 year.

• Conversion Rate: In this situation, conversion rate is % of existing or new customers that will become customers of this new solution.

  • Let’s say you believe that within a year, you can convert 10% of those customers to this new product.

  • Conversion Rate Used: 10%

• Confidence Level:

  • Let’s say you’ve looked at the market and have a sense of competitors doing this and that it is desired but you’ve never connected the dots in terms of if your customers would actually care to have this).

  • Confidence Level Used: In this situation, you’re using the 45% confidence level as your score is only supported by market analysis

• Final Score:

  • 5,000,000 * 10% of those converting via signup * 45% confidence level = 225,000

  • If you built this new solution you’d be able to generate $225k in additional ARR in the first year of launching it.

3. Internal Efficiency Feature Project:

Let’s say you’re building a feature that reduces the duration of a certain internal operations process.

• Reach:

  • In this situation, reach is the total number of hours that that internal process currently takes to perform per quarter. Let’s say it’s 300 hours quarterly.

  • Reach Used: 300 hours per quarter.

• Conversion Rate:

  • In this situation, conversion rate is the expected % of hours that are removed as a result of the solution.

  • Let’s say you believe you can get 70% of those hours removed with this solution.

  • Conversion Rate Used: 70%

• Confidence Level:

  • Let’s say you’ve done the internal process yourself and hacked at a new way to do it that the solution will be based on).

  • Confidence Level Used: In this situation, you’re using the 90% confidence level as your score is only supported by market analysis.

• Final Score:

  • 300 * 70% of those converting via signup * 90% confidence level = 189

  • If you pursued this project you’d remove 189 hours of operations work per quarter.

Scoring Related Notes:

There are a lot of important ideas that pertain to scoring (too many to share in one post). The following are some of the most important learnings I felt were worth including in this article:

• Scoring doesn’t replace doing PM work.

  • Scoring can easily become a crutch for poor PM' ing, as it can give a false sense of rigor and sophistication. Ultimately, someone who has a clear vision, business sense, a deep understanding of customers, and a strong team can achieve an effective roadmap without ever scoring a project.

  • At different times I’ve encouraged my team to drop the rigor and trust their beliefs to mitigate for this risk. In many cases, if there’s a deep belief in the project built on a strong knowledge base (that was primarily developed through customer contact), then you should probably just do the project regardless of how it scores.

• Scoring isn’t the what determines which projects get done and which ones don’t.

  • I originally believed that the purpose of scoring was to decide which projects should get done and which ones shouldn’t.

  • I soon realized that when you pursue a highly regimented scoring approach all you’re left with is measurable work and unfortunately, a lot of really important work (brand sentiment, experience quality, system health, debt creation/mitigation to name a few) is immeasurable or hard to measure.

  • A project’s score within a framework is not the deciding element for if it should be done, it is a data point in your consideration for doing it and an entry point for everyone else to help validate your project’s value.

• Cost isn’t a part of this model because it does not impact value, it instead impacts return on investment.

  • The model does not go into timing or cost as those are properties that define return on investment. Conversation around those variables is important but when evaluating a project I prefer to start with a pure value assessment since doing investment management analysis for a project that has no upside, and as a result that you’ll never do, is a waste of time.

  • Once a project is scored you should be able to have thorough conversations about the worth and timing of projects and the necessary adjustments to extract value.

• Standardization can be counterproductive:

  • A lot of frameworks try to compare projects with totally different goals against one another by consolidating value into an abstract score (if no one knows what the number means then it’s not a useful number).

  • I’ve found this to be detrimental to my ability to identify value and pick the right projects. It’s really hard to compare apples to oranges; if you do, all you’re doing is following whatever natural bias you have towards either apples or oranges.

  • I prefer just talking about value and making decisions on what the business values more over trying to translate that value into some opaque metric and building a dependency on that metric to make decisions.

• Don’t score for the sake of scoring.

  • Not every project needs scoring and in fact scoring can be highly detrimental to your ability to get certain projects through even when they’re overwhelmingly the right decision.

  • Scoring is effective for large projects with clear tangible value. When you have strategic partnership, key customer pain points, small iterations, or other hard to measure opportunities you should likely revisit your need to score as the scoring framework might unfairly punish these fuzzier items.

  • For projects that don’t score well but that I’m still going to do, I won’t have an actual metric value for doing it because that wasn’t why the project made it to the roadmap. It made it to the roadmap because of my belief in its value and I’d rather express that value the best I can rather than make up a value number I don’t believe in.

• Re-Scoring is as valuable as scoring.

  • Once projects are done do your best to re-score them with the actual results. This can be an incredibly insightful process for you and your team as you’ll learn more about the nature of your product and roadmap and why things work/don’t work.

  • This type of review will help you the types of projects you go for and sharpen your instincts for which projects work and which ones don’t.

Firefighting comes down to 4 key skills:

1. Diagnosing

2. Mitigating

3. Communicating

4. Resolving

This article goes into firefighting and how to consistently diagnose, mitigate, communicate, and resolve issues.

Diagnosing

Diagnosing comes down to 2 key ideas: having the right people and environment in place to diagnose and following a structured path that has a high likelihood of resulting in a correct diagnosis.

Environment and People:

Diagnosis can be the most challenging step as the anxiety and stress from the issue can get in the way of critical thinking. As much as you can create an environment for thoughtful problem-solving this will significantly improve the odds of success and the speed at which you find a solution. Set yourself and others up for success by using calm tones when discussing the issue, remind and reassure everyone that you’ll arrive at a solution, and, when applicable, use humor to defuse tension and contextualize the issue.

Most importantly, ensure that the right stakeholders are involved with the right level of involvement. Ensure that you have people actively working on the issue who have the ability to solve the issue and that you’re in contact with teams that are dealing with the backlash of the issue. If you don’t know who should be working on the issue escalate the question to someone who can make that determination.

I’ve found that the single most valuable thing a PM brings to diagnosing is focus. It is hard to stay focused on doing the right things due to stress, incomplete information, desire to know more, and uncertainty. As a PM your job is to constantly align the team on identifying the best next step, pursuing that step, and reassessing the path. If the team is consistently doing this you’ll effectively diagnose and resolve any issues you face.

Path to Diagnosis:

I’ve found that the most reliable path to a diagnosis is to create a timeline of events that can effectively explain why the issue occurred and why we’re seeing the current impact. Timeline building is a great way to ensure that you don’t miss any key variables in the diagnosis and once you have that narrative defined it’ll be much easier to identify the solution. The narrative should answer a few questions:

1. What is happening and where is it happening?

2. How does it happen?

3. Who is affected now and who might be affected later?

4. Why did it start happening and why didn’t we notice it until now?
   

When going through this process you should be capable of proving your narrative is right by checking your assumptions against existing cases and finding consistency across every case you see. If every impacted user or instance is consistent with your assumptions then you likely have proven yourself right. Finally, you need to ask yourself if you were wrong why would it be? This extra step of challenging your assumption can be key to expanding your understanding of your problem and identifying missing pieces to your problem.

With this information, you can work with technical and operations teams to correctly define the solution for the issue and the follow-through.

Mitigating

Mitigating is about reducing the breadth and depth of impact of the fire, in some cases, you can skip this step altogether and just focus on solving the issue. Mitigation is valuable when fully solving the issue will take too long and where stopping the bleeding can help buy you time and focus. If you can’t get affected users back to normal ask yourself if you can prevent further users from being exposed as this often can make a huge difference in the total damage of the issue.

Mitigation should not be done by default and should instead be a step decided upon after analysis. I’ve found the following rules as a good way to decide if it’s worth investing:

• Do invest if it aligns with the ultimate solution.

• Do invest if it significantly improves the quality of life for those impacted or reduces further damage.

• Don’t invest if it significantly gets in the way of solving the issue.

• Don’t invest if there’s any risk the mitigation compounds the issue.

If you are pursuing mitigation ensure that your stakeholders understand the nature of this action and that they don’t interpret this step as actually solving the problem.

Communicating

Throughout the issue, you need to keep people up to speed on what’s happening so that they can tackle their end of the problem-solving. Great communication can often be a game-changer during fires. There are three key ideas to great communication:

1. Communicate what you know

2. Communicate what you’re doing

3. Communicate regularly

Note: When communicating you want to avoid misleading teams, especially if they’re going to bring information to customers, as misinformation can often make the problem worse. You should not make assumptions or guesses at the cause or impact of the issue if you do not understand it as it may create further confusion and stress, but you can provide not fully validated information if you properly caveat it and the likelihood that you’re right is >=80% as this can help teams navigate the issue.

Deliver your updates in structured messages calling out what’s happening, who’s impacted, what’s been done to address it, what’s next, and key things that need to be done (if applicable).

You want to communicate what you reliably know as well as what the team’s next steps are. Specifically, you want to make sure the following things are understood:

• The reach and depth of the issue.

• Who else may be impacted in the future.

• How you’re mitigating that impact.

• What the team is doing next to mitigate or resolve the issue fully (and ideally what the perceived timeline of that is).

Ultimately you’re trying to ensure that everyone is up to speed as close to the real-time as possible but you should always make resolving the issue your first priority. When it comes to how often to share updates shoot for every 30-60 minutes during the early stages of an issue as this will: 1. build trust across the group that you will keep everyone up-to-speed as needed (and as a result minimize distractions) and 2. empower people to properly follow-up with customers/stakeholders.

Resolving

When it comes to resolving the issue, the key thing a PM can do is provide the business and customer context so that the right solution is pursued. In any resolution decision-making process options will be presented and these options will vary in their user impact and costs. By understanding your business and customers you can help teams make the right decision for how to solve the issue.

When looking for the ideal solution remember that:

• The ideal solution to any issue will remove the problem entirely for the affected users.

  • It will often do this while not creating a secondary more significant problem although sometimes it may create a less significant one.

• Oftentimes, to fully resolve the issue a more significant solution, system refactoring, or revamping will be needed.

  • It’s important to delineate what the today part of the solution is and what the tomorrow part of the solution is so that everyone is aligned on how to proceed.

• When pursuing a solution, do NOT attempt to release an untested, or minimally tested, solution, especially under heavy fatigue.

  • The single worst thing you can do in a fire situation is attempt a solution you haven’t fully vetted as this can, and will, often make things worse. The wrong solution can easily turn a moderate problem into a huge problem and it can easily reset your problem solving furthering the damage and frustration.

As the technical solution is being implemented, work with ops teams on the customer follow-through for the solution. Oftentimes, the lengths you and the team go to to put back what was broken and make amends with customers can significantly improve the standing of your business and the relationship with your customers. Nail down the customer communication and any operational follow-through whether it’s cleanup, reimbursement, or committing to preventative work.

Post Fire

Once an issue is fully resolved invest in doing a retrospective about what things went wrong and how can you prevent the same thing from happening again. Identify in these sessions the work that needs to be done to minimize future issues as well as the protocol to be followed.

Whenever possible put monitoring or tracking in place for the issue area as this will help you identify and resolve issues faster in the future. Strong monitoring is the best way to prevent issues from escalating and every fire is an opportunity to improve how you’re tackling prevention.

As mentioned before, follow-up work will likely be necessary to prevent the issue from happening again. Use the momentum of the fire to tackle any system issues that may lead to further fires. Fires create momentum to improve the overall health and safety of your system, use that momentum to get the necessary projects in the roadmap.

Overall, firefighting is a stressful part of PM’ing but it can also be an immensely powerful tool for building reliability as a team and in your system. Take each fire as an opportunity to learn more about your system, your team, and yourself and you’ll find that these learnings will help you become a better PM.

Operations-based solutions tend to maximize outcomes within the constraints of today. A few examples:

• The fastest way to improved internal efficiency is not to build a new internal tool but instead to cut scope and do less.

• The fastest way to higher adoption is not to build a new onboarding feature but to individually connect with customers, understand their pain points, and onboard them.

• The fastest way to drive more conversions is not to build an extra paid feature but to nail down communications, sales processes, pricing, or even run promotions.

Software, on the other hand, tends to break current constraints and expands the business’ problem-solving range and depth:

• In the former scenario, a new internal efficiency tool could wipe an efficiency problem away completely by automating that workflow.

• A new onboarding feature could establish a new floor for adoption.

• A new paid feature could raise the conversion rates across the user base.

When businesses put all their eggs in the software basket they inevitably leave a lot of value on the table because every problem is weeks or months away from being solved and you can only really solve a few problems at a time. When businesses put all their eggs in the ops basket they can easily develop a short-sighted mindset present and create debt that’ll be hard to shake in the future.

It is true that operations solutions can break constraints and software solutions can optimize within constraints, signing a massive partnership, for example, is a transformative, ceiling-raising operations solution. However, this general mental model can help evaluate the right short-term and long-term approach to a problem.

Properties of each solution type:

Software-based solutions

• Require a high degree of investment in terms of development but also in terms of integrating the new feature successfully into the business (building workflows, communication, support, sales, etc).

• Carry long-term maintenance/support/optimization costs that are often overlooked.

• Because of this overall cost, they have to deliver substantial amounts of value to achieve positive ROI. The investment cost is also almost always paid upfront.

• Given the investment scope, businesses can also only afford to pursue a few solutions like this at a time.

• Iteration cycles are slower as each cycle requires additional development which makes it harder to obtain results and validate the solution.

• The total possible value for a solution of this kind is often very high since software can make almost anything possible.
  

Operations-based solutions

• Generally have much lower upfront costs.

• Are easier to manage, support, and discontinue long-term. Businesses can quickly abandon ops processes if these aren’t working.

• This low-cost, low-commitment nature makes them easier to sign off on and leverage to extract value quickly.

• Businesses can generally pursue a great number of ops solutions at any given time (although this capability creates coordination and cohesion challenges).

• Iteration cycles are faster since each cycle has a low investment cost which makes it easier to obtain results and validate the solution.

• The total possible value for a solution of this kind is limited by the constraints of the people, time, capabilities, and capital available to the business.
  

Given the difference in mechanics, each solution type is well-suited for certain problems but ill-suited for others. A few questions can be helpful to determine if ops or software is the right path for a problem:

• Could we reasonably solve this without software? Would that solution be faster? Could we unlock 60-80% of the value with that approach? Is this something that we’re comfortable not building into the product?

  • If all are yes then an ops solution could reasonably solve the problem in a comparable way to software making it a good option.

  • If only some are yes then ops might still be well suited but ops is likely an intermediary solution or a precursor to a long-term software solution.

  • If all are no then this is very likely a software-shaped problem needing a software-based solution.

Software-based ops (hybrid model)

A new hybrid approach has become more prevalent with the rise of low-code platforms and LLMs. In many cases, hybrid solutions are fundamentally shaped like ops solutions in terms of cost and cycle speed but have payoffs that are more software-shaped. This makes them very well suited for a great deal of problems.

In particular, I’ve found these to be well-shaped for areas such as:

• Internal efficiency (replacing pure software internal tooling)

• Experimentation (expanding the possible experiments you can run),

• Replacing high manual labor-based features (replacing things like customizable reports and assets, personalized email campaigns, and marketing page enablement).

How this concept affects PMs:

In many orgs, teams will become software-centric or ops-centric. I’ve found that it can often be the PM’s job to help teams delineate when one or the other is best suited. In creating this open-mindedness PMs and teams can become more effective and nimble in their problem-solving and collaboration.

Effective product management at a small company is often about maximizing the volume and quality of ops bets and software bets the business is making. Indexing too much on one or the other will often lead to disappointment. Software bets take too long to pay out and operations solutions can’t deliver the long-term upside the business needs to continue growing. By effectively leveraging both approaches businesses can effectively maximize opportunities short term while setting themselves up for high upside opportunities long term.

Many PMs will approach reviews with a creator mindset and quickly spit out many potential UX or UI updates, in some cases even feature ideas, without acknowledging the thinking and approach laid out by the designer. In contrast, some PMs do not go deep enough during design reviews leading to designs that can often lack the alignment and depth to be effective when built.

It is key to acknowledge that the designer is the driver of the design and it is through their understanding of the business case, customer, and context that the team will succeed in finding the right experience. Collaboration is necessary, of course, but the designer is ultimately running point on the design. The PM, however, is running point on ensuring the design drives value and delivers on the intended use case by ensuring a use case and value-focused mindset during the process. Through this combination of roles, the discovery and design process finds the necessary pressure, trust, and collaboration to deliver effective design solutions.

The PM’s role in a design review:

The PM’s job is to ensure that the conditions are in place for the designer to succeed. Simply put, the goal is to sharpen design thinking by challenging, clarifying, and contextualizing where applicable. The PM should bring a clear and in-depth understanding of the customer, use case, market, and business implications as well as an open mind and understanding of basic design concepts. As a reviewer, the PM is seeking, throughout the review, to:

1. Clearly understand the laid-out approach

2. Assess the design approach in its ability to solve the use case

3. Correct any misconceptions about the business context

4. Challenge the approach and identify risk areas/complexity that need to be mitigated

5. Ensure that the design approach is aligned with the business case, the philosophical direction of the product, and the end user.

6. Help refine the key follow-up questions/areas to be investigated by the designer.

By doing the above things you’ll become a more effective partner for the designer because you’ll be helping the designer find the path and enriching their mental model for the solution. The designer is the driver, only they can get you to the outcome the team needs, but that doesn’t mean they won’t get lost - this is where the PM can provide the most value, ensuring that we can find the way.

When do things enter the PM domain?

I think there are a few places where one can draw the line for where the PM can, and should, jump in:

• The investment cost line:

  • When a path’s cost exceeds the acceptable investment level for the problem it’s partly the PM’s job to help the team understand a threshold has been reached and the solution needs to be downscaled.

  • It’s important with this type of intervention, to try to understand why the designer ended up where they are. Are they seeing something you’re not, or is there a misalignment in how certain variables are weighed?

  • This is most effectively done with team input and understanding, which requires nailing the exact reasoning for why it would be over-investing.

• The business requirement line:

  • When a path is solving alternative business requirements or business requirements that don’t exist, or not solving the core business requirements at all, then a PM can step in and pull back.

  • The PM is responsible for the outcome being driven and (ideally, always) has the most use case knowledge. This makes them the most capable actor to re-direct focus.

  • Again, here it’s important to try to understand why the designer ended up where they are.

• The decision needs to be made line:

  • When an area of the project becomes blocked there’s no agreed-upon solution, and one can’t be arrived at through research/testing/validation, it’s the PM’s job to make the decision and take accountability for its outcomes.

  • In many projects, this won’t be required, but when it is required, it can be one of the more important things the PM can do as it ensures the project continues moving forward.

• The work is not good enough line:

  • If you’ve spent the time to fully understand the laid-out approach and at the end feel like it’s not good enough and is missing elements that have to be there for launch, it is in your domain to course correct and push the designer to dig deeper.

  • This can be the hardest intervention to make, but when it happens it’s the most important intervention because it ensures that you don’t put something out there that fails to deliver value.

How to approach a design review:

• Ask yourself if you’re at the beginning or finishing phase.

  • Discovery tends to be in either the beginning or the finishing phase. During the beginning phase, you’re trying to nail down the philosophy and requirements for your solution. The beginning is open and benefits from gathering information, talking to customers, experimenting, and having long philosophical conversations. It’s the going broad stage of the work.

  • During the finishing phase, you’re trying to move the work towards a buildable place. The finishing stage is focused on closing out open questions, focusing on the core value to be delivered, editing down to the essentials, and ensuring what we end up building nails the problem/philosophy/value we set out to achieve.

  • It’s important to always have a strong sense of if you’re beginning or finishing and adjust your approach accordingly. If you don’t know where you are in the design process, you might end up investing in processes that aren’t necessary and the process will lose momentum and focus.

• During the review, start by understanding the state of the design.

  • It’s very easy to get lost in a misunderstanding about the level of definition something has. Understand the progress level of the work so that you don’t spend time on the things that they haven’t thought much about. Focus on the things they spent the most time on in the review, or if they’re going the wrong way, focus on clarifying the focus/understanding the misunderstanding.

  • Ask the designer things like

    • “What things should I pay attention to and which ones aren’t ready?”,

    • “Is the work still very conceptual, or more evolved”,

    • “Should I pay any attention to the UI or just the experience?”

• Then dive deep into understanding the design. Hold your thoughts until the review is finished.

  • Spend as much time understanding the approach as possible before making any pronouncements. Try to understand why the designer made each choice and how each choice works together to create a mental model for the solution. This should be 1/2 of the review with the second half being a discussion.

  • During the review emphasize that you’re trying to understand the approach fully and that you’ll be asking many questions, that these will not be judgmental, just information gathering. For understanding, oftentimes intense questioning is necessary, this can make designers feel challenged. You want to mitigate this going in.

  • In terms of questioning, focus on understanding the designer’s decision-making process at great depth, look at each frame and each experience step, and ask the designer things like:

    • “Walk me through how you got to this implementation, what considerations did you have, and why this felt like the best decision?”

    • “What led you to place this step here specifically? How does this placement affect the rest of the experience?”

    • “What do you think are the most important factors to mitigate? Why those factors, and how do you think the approach mitigates those?”

    • “How do you see the intended user using the feature? Walk me through the flow for x use case ?”

    • “What do you think will work and what are you worried will not? What did you deliberate the most on with this approach?”

    • “What led you to this specific UI (dig into specific affordances, placement, components, and how they affect the user’s experience and interpretation of the experience?”

    • “How do you expect the user to feel throughout this experience?”

    • “Do you feel that the user will clearly understand at each stage why this is being asked of them or what their options are?”

    • “What are the principles that you feel are really important in this experience?”

• Once the design approach is fully understood, hone in on the key questions worth discussing.

  • Once you fully understand what’s being presented, focus on highlighting the areas that work and the areas you’re worried about. Discuss each of these to the depth applicable. Mostly focus on what’s missing or what’s going the wrong way over what everyone likes. Discussions around these areas are where the majority of the progress will be made.

• Look for problems at a few different levels:

  • Path: will users effectively get from point A to point B or is there danger they’ll get lost or stuck?

  • Comprehension: will users understand the affordances in place, is there enough context building, is there an overwhelming amount of information, do users understand the possible actions and their outcomes?

  • Requirements: are we solving for the requirements or are we off track, are there new requirements that may have surfaced, will this thing deliver the value we expect?

  • Complexity: is the solution being presented overly complex from any lens, is it overly complex to build, hard to maintain, hard to support, has redundant parts, has unnecessary parts, is it overkill, or should it be simpler?

  • Placement: Why is this feature in one part of the product and not the other? Why does this path take us here? Why does it exist at the 1st layer of interaction and not the second?

  • Sentiment: is there a specific feeling or notion that is key to the user’s success with this experience? Does the laid-out experience keep that in mind/achieve it?

  • Principles: are there essential principles to this experience that must permeate the experience (for example, the experience should always feel non-intimidating, or users should feel rewarded at each step)? Does the design factor these principles in?

• Finally, focus on the clear next steps to move the project forward.

  • Always end with clear next steps or areas of questioning, and align with the designer on what these are and why they’re valuable. Be very conscious about what the process needs, in particular:

    • Do we just need to keep exploring the use case further and go more in-depth into certain areas?

    • Do we need to validate certain notions so that we can safely move forward?

    • Do we need to be more focused on a very specific use case/customer journey and run the design through that use case/customer journey?

    • Do we have a lot of unknown unknowns, should we go talk to customers to find these?

    • Do we need technical input to ensure that what we’re coming up with is buildable?

  • Design reviews will very often end with new questions. It’s important to highlight the progress made in closing older questions and pinpoint the new areas of investigation. If both the PM and designer are aligned on these, you’ll often make quick progress on the problem to be solved.
    

Note: Over-invest in philosophical conversations when there isn’t a strong internal sense of what you’re trying to do, don’t over-invest in philosophical conversations when the truth can easily be arrived at with exposure to the customer. Remember that sometimes, you might need to ship the thing and see what happens with the scale.

It is tempting when facing this situation to enter a victim mindset and blame any problems that come up on the poor decisions of your predecessor or the organization at large. This guide has the goal of helping you find a more productive, healthier relationship with the work that was done before you so that you can ultimately be more effective in your role.

Empathize, empathize, empathize

Understanding your predecessor’s work can help you significantly fast-track your insight into the business, products, and organization. Your predecessor was not a moron. The chances are that your predecessor was quite bright and impactful but worked in a limiting environment. Respect their work and their capabilities.

Try to understand, In detail, the decision-making processes of your predecessor. Specifically, try to understand the following:

• How did they approach their work?

• Why did they approach their work in that way?

• What were they prone to doing/missing as a result?

• What did they excel at?

The moment you become dismissive of someone else’s work you’ll become myopic and fail to see the value and patterns they have created which can slow down your rate of progress.

Understand the products you’re inheriting, and fight the urge to burn it all down

Some PMs immediately move to “rebuild” mode. Rebuilding is often a massive mistake as it is resource-consuming, requires a lot of organizational equity, and, if done too early, will likely kill value inadvertently.

If you approach your inheritance as if you were starting at -100%, you’ll end up spending all your time getting to 0%. Think about your inheritance in terms of starting with an advantage that can be built upon. In most cases, fine-tuning, optimization, tweaks, and simplification can help you supercharge what you already have. Strive to be fair but opinionated as you develop your understanding of the work and ask yourself:

• What works well?

• What doesn’t work well?

• What is close to working well but needs a few changes?

• What do customers love?

• What features do customers use the most?

• What is the team most passionate about?

• What strikes you as the most compelling?

Give yourself the time to understand your products. Don’t knock down a structural pillar out of spite, and don’t take shots in the dark out of frustration.

Understand the decision-making environment at the time

It is essential to understand the decision-making environment to effectively understand why a feature exists in the way it does. If you don’t understand why a decision was made at the time, it’ll be much harder to make a new decision on the feature now.

To better understand the decision-making environment at the time, ask yourself questions like:

• What was the process like when these features were built?

• At the time, was there enough talent or domain expertise on the team to solve this problem?

• Were there internal or external pressures when these decisions were made?

• Has the business significantly changed around this feature?

• What would have happened with my process if deployed at that time in the organization?

If you don’t understand the company’s culture and its progression, you’ll eventually find it working against you and your goals. Most of the time, you’ll find that bad historical outcomes were a result of poor processes, low skill, or pressures from leadership, investors, or high-profile customers. Outside of those factors, you’ll find that some features might have just overstayed their welcome and that the business has changed since they were originally launched.

Work to define what your version of the role will be like

When inheriting a product and roadmap, it is key to establish your version of the job and build alignment with stakeholders.

In many cases, there will be give and take, it’s hard for people to dramatically shift expectations on a role, but you can work on this expactation over time. While this type of expectation can initially be set during interviewing, in many cases, expectation setting is an ongoing part of working within an organization.

Take the time as you build a working relationship with your manager and team to talk through what you believe is most essential in the role, what you expect of yourself and of others, and what will happen if you’re capable of pursuing the role in this way. Having these kinds of conversations can be essential for you to carve out the version of the role you believe in while understanding what areas you’ll need to address and support so that your team continues to operate effectively.

1) Yes, imposter syndrome is normal.

Yes, it will get better with time. You’ll be surprised how time will make even the most intimidating tasks feel second nature. That being said, use that feeling of fear, shame, and helplessness as fuel to get you to the next level.

Most of all, start getting comfortable with not knowing. Every year there will be many, many challenges that will be totally new and unknown to you. When encountering these, focus on the feelings of curiosity and fascination to build a better relationship with not knowing.

2) It’ll never be enough… and it’ll never end.

Fall in love with the journey or you’ll end up hating the work. It will kind of always be a grind and it will always challenge you.

In some environments, others will indirectly chip away at your love for the work. Protect your passion as much as you can. Find ways to make the work move you. Find the magic in the creative process and find the magic in the impact the work has on people to keep yourself motivated and invested.

3) Stop trying to control everything.

Everything will get 10x easier once you realize control is pointless. You can’t control others, even when you have authority, you’ll barely be able to control people.

Focus instead on being a source of clarity, focus, energy, empathy, and passion. Creating and harnessing momentum is much easier and effective than pushing and dragging others.

4) You’re not paid by the word.

Precision and clarity are critical to getting ideas across. The less you say, the more each word matters. How you say it can also make all the difference.

5) Disagreements are a natural part of the creative process.

Never lose sight that the creative process requires conflict. At the same time, ensure that conflict can happen in a safe way. Never make it personal, always take the time to understand the other person’s thought process, and challenge ideas while acknowledging that you don’t know what the best outcome is (only that you’re seeking it).

Don’t be afraid to get into a really passionate discussion about something. If things end poorly, take the time to reach out and acknowledge your appreciation for the other person’s work, passion, and desire to get it right. Always mend fences when fences need mending.

6) Always take the time to form an opinion.

At the end of the day, your perspective is one of the key things you can add to a team. Remember though, your opinion is only valuable if you’ve really put in the work. You also don’t have to be a contrarian or fit some idealized template, it’s perfectly fine to look at a situation and have nothing additional to say if the team has said all that needed to be said.

In most cases, if the team knows you’re reliable, and that you’re going the extra mile, they’ll respect what you have to say. Even in environments where you feel like your opinion is not wanted, you’ll find that if you continue to develop it, you may eventually find a way to break through. A good opinion in the right situation makes all the difference.

7) Strong opinions, loosely held.

You’re seeking truth in projects, not self-validation. You want to be incredibly resolute about what you believe in but, at the same time, be ready to throw that belief out the window if you’re presented with highly compelling evidence or arguments against it. Being stubborn when you’re wrong is useless. Being stubborn when you’re right can be essential to the success of a project.

It takes a lot of willpower to make a meaningful impact, but it also requires an open mind and a strong ability to reset.

8) Relationships matter a lot.

Building and maintaining trust with stakeholders is essential to any success you’ll have. Relationship management will be key throughout your journey whether it is to foster a strong collaborative process, persuade others to change their mind, get help when necessary, or battle tough times. Without others trusting you, you’ll find the job to be impossible.

9) You deal in the What and Why, only sometimes in the How.

If you’ve come from an individual contributor background this will be really hard to shake off. Once you do shake it off, you’ll find that you can become an amazing partner to your teammates. The What and the Why is the stuff you have time and skill set for. In most cases, you won’t have the time, or oftentimes, the skill and knowledge to deliver an effective How.

Over time you’ll find a need for the How and the joy it can bring. Find safe spaces for the How whether it is in your personal projects or side projects in your work. If you abandon the How you’ll find it harder and harder to relate to your team over time.

10) Never compromise on common sense.

You’ll be blown away by how quickly common sense goes out the window. You’ll also be amazed how quickly you’ll lose it yourself. Compromising gets things done but push to never compromise on common sense. Without common sense, a lot will break with your products.

11) Nurture the creative environment

Create an environment for creative decision-making by communicating clear use cases, goals, and priorities, and having an open mind. Spend a lot of time learning how to make yourself a better partner to designers, engineers, and other creators. I highly recommend reading about music producers such as Brian Eno and Rick Rubin, and how they partner with artists in the creative process.

12) Cherish the craft

Software is a craft and most people on your team are artisans. Think about what motivates every team member and why they’re in this line of work.

When thinking of software, a famous line from Moneyball comes to mind: “How can you not be romantic about baseball?”

It is an unbelievable thing we get to do every day. In the work, in the details, there is magic. Creating is an incredible privilege, creating for others (especially creating things you love) is even more so. Don’t lose sight of that privilege.

13) You literally can’t solve every problem

There are too many problems. New problems are made every day (if not every hour). The sooner you stop playing wack-a-mole the more effective you’ll be.

Constantly ask yourself “What is the highest value problem?”, and do your best to constantly pursue that problem. Additionally, seek to build systems and documentation that enable others to tackle problems that you care about solving.

14) If you believe in an idea, pursue and validate it

Change is a constant. If you think something is worth pursuing, validate it by yourself and with the team. It might end up getting prioritized in the roadmap. Your belief that something is worth doing is one of the strongest intrinsic guides you’ll have through this messy world. Nurture it, develop it, and lean into it.

15) Simplicity is a key

Simplicity without quality loss is the key to many of your problems. In any problem you’re seeking to solve, start by finding the essential. When you’ve arrived at the essential you’ll find, most times, that you’re much closer to the answer than you previously thought (if not there already).

The areas in which I’ve had the most impact were the areas where I was able to reduce lots of complexity to an easy to understand concept that accurately described the phenomenon or mechanic at hand. Most progress is built on a very well understood concept or sets of connected concepts. The more effectively you can understand and effectively describe complex ideas the more impactful and wide reaching your work will become.

16) Be Yourself

The best PM version of yourself will be founded on who you are as a person. You’ll never be your best by being someone else. If you focus on finding the best PM version of yourself you’ll be something no one else can be.

• Low value | high cost → AKA mistakes

• Low value | low cost → AKA why bother

• High value | high cost → AKA core roadmap items

• High value | low cost → AKA low-hanging fruit

As a result of being high value and low cost, low-hanging fruit is some of the most desired work for an organization. It is, however, hard to find low-hanging fruit if you aren’t regularly and deliberately looking for it.

This guide seeks to provide paths that can yield low-hanging fruit opportunities.

8 ways to find low-hanging fruit:

1) Find the friction

Why it works:

• Most problems in your product will not be reported by end users unless they are leading to the total failure (or significant breakdown) of a feature.

• As a result, there’s a very good chance that your product has friction in some key places that are invisible to you but highly disruptive to your users.

• If you find and unblock those pathways and you’ll see a higher fraction of users performing actions successfully. Do this for an important enough pathway and you’ve found a big win.
  

How to look for it:

Start by thinking of the most important actions that happen in your product, then try to identify any points of friction in the process of performing those actions.

Ask yourself “Would people reasonably understand the path to take?”, “Is the feature slow, is load time an issue?”, “Is it a big pain to get from point A to B with this feature?” and “Does this experience suck?” to find potential improvement areas.

Additionally, look at recordings, do a few feature walkthroughs yourself, look at feature data, talk to customers about it, run usability testing or feature walkthrough sessions, and/or map a funnel for the feature if possible as these can help uncover those points of friction.

2) Find a small win for the biggest possible audience

Why it works:

• Big wins are generally much harder to find than small wins.

• There are more small wins out there than big wins.

• It is, often times, easier to find a small win for a big population than a big win on a small population (or a big win on a big population). A small win on a big population will, often times, have the same effect as a big win on a small population.

• It is therefore an effective strategy to try to identify small wins on the broadest population possible as these will be easier to find and likely to yield big effects on your product and users.
  

How to look for it:

Look for areas of your product that experience a great deal of traffic or that substantial portions of your population rely on. Focus on identifying small potential ways to significantly improve those areas.

Ask yourself “What are some ways we can improve efficiency by 1-10%?”, “What’s the easiest and cheapest way to make a massive improvement without significantly changing the product we have today?”, or “What’s the clearest no-brainer improvement we can make?” to start identifying these areas.

3) Find the hidden bug in a core system

Why it works:

• The longer a system has existed and the more other systems leverage it, the more likely there are hidden issues with it or with the systems that rely on it.

• Similarly to looking for friction, finding and removing the hidden disruption in a high-leverage system will often create many positive outcomes.

• In many cases, teams will inadvertently build solutions around broken features. By solving critical bugs you can remove that unnecessary overhead and make your system more reliable in the process.
  

How to look for it:

Think about your critical systems and assume there’s something wrong with them. Subsequently, spend time trying to “break” the feature to see if you can identify issues.

Ask yourself “If this feature is broken, why would we not know?”, “What are the worst ways this feature could be broken?”, and “What part of this feature does no one really know how it works” to find potential leads.

Additionally, spend time with your team and other internal stakeholders of the feature and try to dig for any known issues or concern areas that might exist. Stakeholders can often guide you in fruitful directions as they’ll have a strong understanding of the feature as well as fears about how it could fail.

4) Find the biggest expenses

Why it works:

• The longer a 3rd party exists in your system the likelier it is that you’re overusing it without realizing it.

• Cost problems tend to compound over time as teams can grow to leverage the third party in inefficient ways or turbocharge expenditures accidentally.

• If your business doesn’t do a good job cleaning 3rd party bills or have effective expenditure monitoring, then you’re likely to find many wins around cost-cutting and making your business more profitable.
  

How to look for it:

1. Take all the expenses your business has, especially the expenses related to software and 3rd party tools (think about any feature that leverages a third-party system that charges at a fixed (or better yet variable) rate).

2. Map all those expenses in a spreadsheet along with the cost, billing cycles, any variables that you control, the usage data for that feature, and the qualitative importance of the feature.

3. Once you have that list in place, start with the highest-order expenses and work your way down trying to find which ones you can cut, simplify or downscale. Chances are that you’ll find expenditure bloat somewhere.

5) Find the most painful internal processes

Why it works:

• Non-technical teams will often get stuck in terrible processes because they lack the development support to fix the issue or because the company is stuck in an outdated mental model for a problem.

• These processes can create significant wasted effort as well as affect morale. Worst of all, they can lead to the business incorrectly assuming it can’t scale because it assumes the operational model is unchangeable (without significant investment).

• You can often find simple yet powerful solutions to these issues. These solutions often unlock significant growth potential for the business and free up teams to more effectively use their time.
  

How to look for it:

If possible start with a concrete measurement of the most time-sucking processes for internal teams. If these are not available, ask team leads and individuals about what they think these processes are and where they rank one against the other.

Ask yourself, “What’s a process that needs to get done but currently gets done very inefficiently?”, “Could that process be done with a different tool and as a result be made significantly more efficient?”, “What’s the 20% version of this process that would yield 80% of the result?” and “Could we scrap that process entirely, if not what would be the second best path?” to identify leads.

Dig into internal processes and look for the ones that would be easiest to solve. In these processes, you can often find many easy internal efficiency wins. For more information on how to tackle internal efficiency projects check out this article.

6) Find the leak

Why it works:

• A bad enough data problem will consistently disrupt the user experience, your team’s focus, and introduce risk to your system.

• In many cases addressing data inconsistency/utility issues can result in an efficiency boost to your product or can enable the team to drive a better experience down the line.

• In most cases, teams already know where the bad data issues lie (if not where they start) making these kinds of problems some of the easiest to identify and understand.
  

How to look for it:

Is there a data set in your system that’s totally broken? For example, do you have data that should be standardized but is not, data that should be in one format but is in a less useful format, or data that is only captured half the time?

Ask yourself “Where is the bad data becoming bad?”, “What are we doing today to address it?”, “What is the impact of this bad data both in terms of issues but also missing upside?”, and “What is the ideal solution to this bad data creation problem?” to better understand the leak and the potential value of fixing it.

7) Find a way to connect the dots

Why it works:

• It is often the case that two experiences should have been connected but your team never had the time to complete the connection.

• Solving these cases can sometimes unlock a 1+1 = 3 scenario as the synergies between experiences will amplify the outcome for the user or supercharge the overall experience.
  

How to look for it:

There are a couple of good ways to find these kinds of opportunities:

Look to better leverage raw materials.

• Think about your product’s raw material. What are the outputs that are being created by your systems? These can be things like videos, favorites, likes, comments, posts, documents, etc. Now think about how that raw material could be leveraged elsewhere in your product to achieve a positive effect.

Look to unite two separate experiences.

• Think about two key actions/workflows/experiences your users are performing that have overlaps in the use case, data, or mindset. Then think about what the benefits would be from connecting those two experiences in a more seamless way.

8) Find the feature that needs retiring

Why it works:

• Sunsetting features can create efficiency and morale boosts in ways that adding features cannot.

• Features long past their usage will slow the system down, detract from the team’s core focus, and require deep knowledge that is often isolated to certain individuals.

• As most organizations and teams have a hard time sunsetting past work, there is often a lot of opportunity to pursue this angle.
  

How to look for it:

Look for things in your system that create complexity and cost for your team but that have questionable or unknown value.

Ask yourself “What are our least used features?”, “What are the features we hear the least about?”, “What are the features that the team hates interacting with or supporting the most?”, “What do we keep getting bugs about that we hate working on?”, and “What happens if we do away with the feature altogether?” to find potential areas of your product that need to be cut out or simplified.

The experiment is complete… What now?

Once the experiment is finished, assuming everything about the design of the experiment and the tracking of the results was set up correctly, there are three key actions to take:

1. Analyze your results

2. Report your results

3. Identify next steps

Analyze your results:

Analysis is often a surprisingly challenging part of the experimentation process. While you’re looking to extract truth from data, you’ll find that it is often possible to use data to create many seemingly believable narratives. This makes it challenging to land on solid truth.

During analysis, you’re looking to create a clear separation between the core hypothesis validation that the experiment was designed around, from the other interesting factors that might have surfaced during the experiment. Focusing on your core hypothesis is a good way to stay grounded in your search for truth.

If the experiment was set up effectively this should be simple as you’ll have clear data indicators of if the hypothesis was proven or disproven. If your experiment was more open-ended, then you may not have a great deal of clarity but the experiment might have helped point you in a more promising direction.

The following are a few questions that can help navigate the analysis process:

1. Core Hypothesis Questions:

   1. Was the experiment conclusive? Can I make a pronouncement on my initial hypothesis?

      1. Always go back to your initial hypothesis and analyze the data you obtained in context of that hypothesis (specifically if it proves/disproves it).

      2. If applicable, check for statistical significance to determine if there’s a clear result for A/B or multivariate tests.

   2. What conclusions can I make about the experiment’s results that I would bet my life on?

      1. Push yourself to an extreme standard as this will help clarify the things you full-heartedly believe in, from the things that you sort of or mostly believe in.

      2. If you report assumption-riddled insights, you may lead your team down the dangerous path of developing solutions that won’t work as the underlying premise for those solutions is untrue. Ensure that you’re clearly separating the proven insights from the ones that haven’t been validated.

   3. What things would perform exactly the same if a 3rd party were to re-run the experiment?

      1. This is another way to evaluate the results you’re seeing, this time through the lens of repeatability.

      2. Results that you strongly believe could be replicated, or that others could replicate in the future, are results that you can count on. If you have something that will perform consistently you might have the foundation for a new feature, product, or process.

   4. How did the key metrics I was tracking perform? What could their performance mean in relation to the core hypothesis?

      1. Did the metrics perform according to your initial expectations or were you surprised? Is there a discernible pattern or are some metrics moving in contradictory ways?

      2. Look at the different possibilities your metrics are illustrating but remember the bar of truth you’re looking for. It’s ok for an experiment to end with new questions and some ambiguity as long as you’re acknowledging that that’s the case. Most experiments result in follow-up research being required.

2. Other questions worth asking:

   1. Are there any qualitative aspects worth highlighting?

   2. What variables were most surprising/unsurprising?

   3. What would be the most interesting things to follow up on?

   4. Were there clear outliers in the experiment?

Once you’ve answered the above questions, you should be capable of putting together a report on the experiment and determining what comes next.

Before covering reporting, there are a few things to keep in mind during analysis:

Inconclusive results

If your experiment was inconclusive, the first place to start is with the experiment design:

1. Did you build the right sample?

2. Did you set up an experience that effectively tested your hypothesis?

3. Were there variables in the experiment that you might not have been aware of initially that would have significantly affected the results?

4. Have you run the experiment long enough?

In some cases, you might have solid experiment design and still get no results. Generally, this means that there’s no significant change in behavior. If this is the case, that is a valuable finding in itself and should be reported as such.

Report your results:

• Being capable of effectively translating your findings to a broader audience is one of the single most impactful things you can do as a product manager.

• If you’re experimenting effectively, the information and insights that you’re uncovering are some of the most well-proven and reliable insights accessible to the business. As a result, you can have a strong effect on how the business operates and thinks.

• To do this, however, you need to be very effective at reporting information. In many cases, you also need to spend a great degree of care and time putting together compelling resources that others can leverage to understand your work.

The key information items of a report:

At the end of the day, in most organizations, the stakeholders interacting with your experiment only care about the results and that the results are reliable (understanding process and methods is secondary for most stakeholders).

You’ll also find that getting your results understood is something that only happens through proactive action. The more friction there is to learn about your findings the less likely anyone is to learn them. The more ambiguity, the more likely each stakeholder is to come up with their own interpretation.

In terms of reporting, the key aspects to cover are:

1. The hypothesis

2. The experiment conditions

3. The results

4. The insights from the results

In practice, a report will sound something like this:

1. “We had the following hypothesis”

2. “To test that hypothesis we set up this experiment”

3. “The following things happened during the course of the experiment”

4. “Based on these results we have strong proof that the following things are true… We also believe that the hypothesis has been proven/disproven...“

To ensure your results are not lost, focus on clean, crisp, impossible-to-miss reporting. Ask yourself “What are the key things that everyone should echo after reviewing?” Focus on delivering those key things and minimize possible distractions.

Format and delivery:

Without proper distribution and packaging your findings will get lost in the shuffle. This is an essential aspect of reporting, the following are two (of many) ways to do this:

• A written doc or email covering the information and data from the experiment is a low-effort way to formalize the findings.

  • I’ll generally use this medium if the results are worth reporting on but the underlying thesis is not fully consolidated.

  • I try to send these to any relevant stakeholders for the experiment (i.e. anyone who would care to/should read the report).

  • To make this readable, look to highlight key portions such as the hypothesis, results and insights. Try to add a “too long, didn’t read” section at the top of your email covering the most essential insights to ensure at least something is retained.

• A slide deck is a strong medium if the results are high leverage or if you’re at the culmination of a lot of research and work. Yes, slide decks have a bad rep but if you’re not in a reading-friendly organization (in most cases you won’t be) they can be a very effective tool to get people to quickly understand and absorb your work’s findings.

  • The deck should cover (as mentioned above) the hypothesis, the experiment, the results, and the insights.

  • Each section is 1-2 slides, contains the highest importance information, and is visually tailored (writing, assets used in the experiment, videos of user behavior, data visualization, etc) to best convey the idea behind the slide.

  • You can always add an annex at the end with more detailed information on the experiment if necessary.

  • Worth noting, the goal of a slide deck is not to run through a presentation, but instead to create an easily consumable report that can also serve as the basis for further discussion.

Regardless of the medium or distribution method for your insights, always ensure two things happen:

1. You have made your findings as digestible as humanly possible

   1. The bar to strive for is: If everyone in your organization read your report and were asked to state the findings every person would say the exact same thing.

2. You have allowed people to digest findings async but have also established time to review them synchronously with key stakeholders

   1. The async review lets stakeholders digest findings, the sync review often becomes a rich collaboration and discussion space around your insights.

   2. Both are extremely valuable and lead to a successful internalization of the findings you’ve presented.

Identifying next steps

The next step for an experiment should revolve around one question: is it worth continuing down this line of thinking?

The answer to that question depends on the perceived value to be gained, the level of insight you have on the matter, and the context of the surrounding business. With that in mind, there are 3 possible paths after an experiment is analyzed and reported:

• Experiment or research further

• Pivot

• Productize

Experiment or research further

This is a valuable path to pursue if you have:

1. Clear areas of further investigation that are promising and likely to lead to insights that will bring value to the business.

2. The buy-in from the organization to continue your research path.

3. No clear productization path for what you’ve learned or limited value in productizing at this time.

If you are to continue experimenting or researching then you should consider the following risks to your research path:

• Loss of organizational buy-in

  • Keep in mind when evaluating the option to do further research that there is often internal pressure. It’s easy to lose momentum and buy-in in some cultures the longer the research takes.

  • Consider if the organization may be losing patience or feeling the pressure to move towards action. Selling is key to preventing this but sometimes so is acting (by productizing or taking other steps to unlock some of the value you’ve been experimenting around).

• Limited future value

  • While it isn’t necessary to have a complete understanding of how the research you’re doing will unlock value, it should at least be in an area of the business that is rich in opportunity.

  • This means that there should be a reasonable expectation that better understanding the problems you’re looking at will unlock value in the future.

  • To an extent, you should also consider the future viability of productizing around your research path. Specifically, there should be an aligned expectation (between you and the organization) that productizing around your research area is not only a likely outcome, but a shared priority.

• Narrow focus

  • Another thing to keep in mind when deciding to do more research is if you can start taking your findings and piecing them together into a larger theory.

  • Shifting your mindset from a single hypothesis to a larger theory can help you find a broader, more impactful area of focus for subsequent work.

  • Ask yourself questions like “What is the broader pattern that may be happening here?”, “If I distilled what I’m learning down to its simplest element, what would that be?” or “What would be the bigger truth here that could unlock a lot of value?” to get yourself into this mindset.

Pivot

This is will happen if:

1. The areas of further investigation are not promising or likely to lead to insights that will bring value to the business.

2. There is limited/no buy-in from the organization to continue your research path.

3. There is no clear productization path for what you’ve learned or limited value in productizing.

There are some key things to keep in mind when pivoting away from a research path:

• Document and catalogue your findings

  • In the future you may come back to your research as the context of the business changes.

  • Throughly document your work and make it easy to find/search for. Future you will appreciate this.

• Make sure stakeholders are aligned

  • Many times stakeholders will have been invested in your research and will be surprised by a pivot.

  • Ensure that you’ve built the shared understanding and buy-in to pivot effectively. Explain why the pivot is happening and clarify where the work will go from here.

If you consistently report and sell the value of your research you might be able to prevent pivots, block pivots or gain an extra bit of room to finish out your research. This is why formalizing findings and building alignment is key to longevity in your research focus.

If you are forced into a pivot, remember that this door may be open for you again and that your work has not been wasted.

Productize

Productization is the best outcome as this means your findings are valuable enough to be made part of the product or processes of the company. Productizing simply means taking the mechanics of the experiment and doing the necessary work to enable those mechanics to function or be used at scale.

This is a valuable path to pursue if you have:

1. A clear productization path for what you’ve learned and the right timing to execute on these insights

Things to keep in mind when shifting to productization:

• If you are thinking of productizing, first ask yourself: “Is there more I need to understand about this before driving value/productizing?”

  • Ensure that you are getting enough validation to derisk productization, but, at the same time, don’t be afraid to push for productizing mechanics that are getting strong results.

  • The investment to be made for productization is significant, make sure that your data and process is solid and that the insights are reliable enough to spend money on.

  • If you are being asked to productize based on an experiment and you’re not confident in that path, bring your reasoning and your data to the stakeholder.

    • When there is a disconnect in productizing where one party wants to do it and the other does not, it tends to be because each party has a different interpretation of what the experiment showed.

    • You might not end up getting the stakeholder to buy-in to your position but the conversation will reveal the reason for the misalignment and you might be able to problem solve from there.

  • Productizing and continuing the research path is not mutually exclusive.

    • The more you’re capable of parallel pathing these two actions the more positive your organization will see your experimentation.

    • You’ll also start benefitting from productization as the data from the products you’re building will help feed your research path by uncovering new insights.

Parting thoughts:

• The more experimentation and research you do the more comprehensive your understanding of the business will be.

• The deeper your understanding of the business the better able you’ll be able to create cohesive theories that describe its underlying mechanics and outcomes.

• Big, validated, cohesive theories are incredibly value for organizations as they allow for new strategies to be developed and investments to be made in productizing in accordance with those theories.

The cost of detecting and fixing issues increases exponentially the longer they remain undetected in the workflow. Issues introduced at the beginning of a project are often, as a result, the most expensive issues to detect and fix.

1 Researchers at Hewlett-Packard, IBM, Hughes Aircraft, TRW, and other organizations have found that purging an error at the beginning of construction allows for rework to be done 10 to 100 times less expensively than when it's done in the latter parts of a project.

For many teams, the user story is the consolidation point of an idea into an achievable piece of functionality. This makes user story creation a key point in the development cycle as it can significantly influence the project’s success and cost down the line. As a result, by delivering high-quality stories, you will be capable of effectively minimizing the downstream likelihood (and cost) of issues giving your projects a better chance to succeed.

User story disclaimer:

Not everyone needs to use user stories or benefit from them. I’ve found user stories to be one of the lowest-effort ways to have documentation, alignment, and tracking for the development of a solution.

I’ve also tried many other ways, all having worked to different degrees of success/failure. It is not the system that makes your work succeed, it is how the team works together that truly matters. User stories are just another way to help teams work together more effectively.

That being said, many organizations find user stories to be the way they prefer to work so understanding how user stories can be made into effective collaboration, documentation, and tracking tools is a valuable skill for a PM to have.

User story principles:

I’ve found that to maximize the utility of user stories there are a few core principles to keep in mind:

1. User stories are owned by the entire team:

   1. User stories represent agreements between stakeholders. They reflect the functionality the team has agreed to build along with the context behind that choice and its execution.

   2. While there may be one main writer for the story, each team member can, and should, provide input on the story itself and feel ownership over it.

2. User stories should be written in such a way that anyone, even those who weren't involved in the writing/review process, can quickly understand them.

   1. This ensures that anyone can easily pick up a story, regardless of their prior level of involvement on a project or how long it's been since the story was groomed. The following are the key items people should be able to quickly grasp when reading a user story:

      1. The affected user

      2. The functionality to be built

      3. The intended outcome

      4. The requirements to achieve that outcome

      5. How you will validate that the implementation achieves the intended outcome

      6. The effort required to tackle the story.

3. It takes teamwork to move stories through the workflow:

   1. For a user story to be moved across the workflow there needs to be understanding and buy-in for the work to be done. Without team alignment, the story itself is worthless as it represents a truth that only a fraction of the team understands and believes.

   2. Making this acceptance of a user story’s readiness into an explicit, workflow-based, action that the team performs together can help ensure each story is viable and maximized.

4. User stories represent an end-user outcome that can be enabled by executing the story:

   1. Stories are expressed from the perspective of the end user (which in some cases may be a system) and explain a feature and its outcome.

   2. Without an end-user value mindset, user stories can easily become ineffective and hard to grasp. Always ensure that the story actively describes an outcome to be achieved for the end user and the means to achieve it.

How to structure user stories:

I generally construct user stories using 4 key sections. These sections are:

1. User story

2. Additional context

3. Acceptance criteria

4. Testing plan

User story:

Template: 

As a {{user type}} I want to be capable of {{action to be performed}} so that I can {{value to be experienced}} 

Example: 

As a paid user I want to be capable of favoriting a publication from the news feed so that I can track publications I’m interested in hearing more from.

• What it is: A description, from the standpoint of the user, of the outcome to be delivered and the means to achieve it.

• Why it matters: This is the easiest, fastest way to create context for the entire team on the goal of the work item. By having this context anyone can understand and make effective decisions when executing on the story.

• Questions for quality control:

  • Can someone who didn’t write the story understand its goal just by reading this section?

  • Does the whole team understand the user, the action, and the value? Do different people have different interpretations?

  • Is the user type clear and identifiable?

  • Is the story specific enough? Is the story too specific?

  • Is the end goal clear and is it easy to understand why it would be valuable?

Additional Context

Template:

There’s no template for additional context but generally this section is needed if there is supporting context that can be helpful to make a story likelier to be effectively tackled. 

Things that fit this section:
• Links to relevant materials such as historical documentation on the feature the story touches.
• Explanations of the business context or motivation that lead to tackling the story.
• Related issues/stories that may be worth consulting. 

• What it is: The supporting context for the user story.

• Why it matters: Because sometimes user stories benefit from having additional context. This can help ensure that stories are fully understood.

• Questions for quality control:

  • Is the team likely to understand the context of the story?

  • Is there a context for the story that if understood reduces risk?

  • Is there additional information about the underlying feature or system that would be important to understand when tackling the story?

  • Is there anything that could easily be missed that could lead to the story being incorrectly addressed?

  • Have we tackled something like this in the past? Should that previous work be consulted when tackling the story?

Acceptance Criteria (ACs)

 Template:

There’s no template for ACs, different stories require different ACs. It’s up to the team to ensure these work for the story. A good starting point is:
Given {{the case in the story}}:
1. {{This is the state}}.
2. {{This action}} {{results in this outcome}}.
3. {{This action}} {{results in this outcome}}.
(…)


Example: 

As a paid user on the feed, I should be capable of:
1.Seeing the favoriting button in the news card. The button should accurately reflect if the media publication is favorited/not favorited.
2. If the button was inactive (not favorited), clicking it should lead to the publication being favorited and the button should update to active. 
3. If the button was enabled, clicking it should lead to the publication being unfavorited and the button should update to inactive.

• What it is: The high-level requirements that, if met, validate that the feature is working and delivering the user story outcome.

• Why it matters: Because it creates a shared understanding of what requirements must be met for the story to be deemed complete. This makes it simple for anyone to determine if a story is complete or not.

• Questions for quality control:

  • If this user story gives us trouble, what will be the reason?

  • Is the team capable of testing the ACs in order to validate that the feature works?

  • Did the team go through these ACs, review them, and agree they’re ready?

  • If someone wasn’t involved in the AC writing, would that person be capable of understanding them and testing the user story using them?

  • If just these ACs are met, does this story successfully pass testing? If not what’s missing?

  • Do we need a testing plan? Can we use the ACs to test it in this case or would we benefit from more info here?

  • Is there anything about testing these ACs that is messy/easy to miss?

Testing Plan

Template:

There’s no template for the testing plan, different stories require different testing plans or may not require them at all (if the ACs are clear and guiding enough). It’s up to the team to ensure these work for the story.

Example:

Checking that favoriting publications works

1.Login as a paid user
2. Go to the feed
3. Click the favorite button on the news card. This should flip the button to active. 
4. Check the DB for a favorite record from the media publication for that user. 
5. Click the favorite (unfavorite in this case) button for the same news card. This should flip the button back to inactive.
6. Check the DB for the same record. This record should now be soft deleted.

1. Login as a free user
2. Go to the feed
3. There should not be a favorite button on the news card. 

• What it is: A series of steps the team can undergo to validate that the feature is working as intended.

• Why it matters: For most tickets, the ACs are enough for the team to test. But for some tickets, you need more definition on how to test. This section allows the team to align on the testing plan when needed and document it for future use.

• Questions for quality control:

  • Do we actually need a testing plan for this?

  • How do you successfully test this feature step by step?

  • Are there cross-user considerations?

  • Is testing of this dependent on something else?

  • Do we need a manual action (from a dev or someone else) to test this? Like running a command.

  • Can this be tested on our regular testing environments?

  • What user accounts do we need, and what data do we need? Do we have a means of generating that testing data?

How to scope user stories

The hardest part of defining a user story is determining its scope. While this is more art than science, there are questions you can ask yourself to help find the right scope:

1. Is the user story the right size?

   1. User stories that are too large can slow down progress and hurt team morale as they make teams feel like the work is just not moving forward. Most times stories can be broken down into smaller stories which allow the team to more effectively, build, test, and release leading to faster value delivery.

   2. It’s important to stress test stories for size and to keep in mind that most stories tend to be over-scoped. If you’re already breaking stories down, keep in mind that the reverse can happen as well. Sometimes teams can get overzealous with breakdowns leading to stories that don’t deliver value, feel incomplete, or are generally overvaluing size over value. Think about story size in terms of the end-user and the context of that story in the project, ask yourself “Can it be smaller” but also “Should it be larger”.

   3. During the course of a project, it’s easy to bundle together items that don’t need to be bundled. It’s important to also ask yourself if there are pieces of the work that could be realistically done independently, or if there are parts that could hold back the story but shouldn't. This can help clarify story size and decouple the core parts of a story from the less essential parts.

2. Does the story make sense from the standpoint of the end user?

   1. It’s easy to lose sight of the end-user when optimizing user stories and this can easily lead to stories that are perfectly shaped for moving through a workflow but deliver poor or incomplete value on delivery.

   2. Ensure that the stories you’re writing deliver meaningful change to the end user be that end user an actual user or a system. Ask yourself if the story delivers actual value if achieved or if there is something missing.

3. Is the story testable?

   1. If a story cannot be tested you cannot ensure that it’ll deliver on requirements and as a result you cannot ensure that it is completed. For a story to be effectively deemed complete it needs to be tested which implies you have a clear way to verify requirements.

   2. If this is not possible, the team will need to determine why it’s not possible and problem-solve from there. In many cases, if a story is not testable it is because there are dependencies that need to be solved first which makes it so that the story needs to be tackled later on once those dependencies are met.

   3. Don’t move stories into the workflow without knowing exactly how to verify that they were completed or you run the risk of realizing the story isn’t achievable in its current state and needs to be rethought.

4. Is the story in the right place in the sequence?

   1. During projects, one of the main reasons why stories can fail is because they are being worked on in the wrong sequence which can make it so that the story is not testable or that if completed it does not meaningfully move the project forward.

   2. If tackling stories within the context of a project it’s key to understand the order in which stories are to be tackled as this will ensure that each story can be effectively tested and completed. Ensure that the team has thought about how the sequential execution of stories will lead to a finished deliverable.

5. Is there a clear delineation between what is to be covered in the user story vs elsewhere?

   1. At all times it should be clear what information is to be covered in user stories and what information is covered elsewhere. For example, it should be well understood by the team if user stories cover design specifications or if these are covered in design specks or other materials.

   2. Ensure that everyone understands what information the user story is responsible for and what materials cover the remaining information needed to tackle the story. It is very easy to miss requirements as a result of having ambiguity in terms of what information lives where.

Parting thoughts:

It is important to remember that processes exist to ensure that certain things happen in the workflow. As a result, processes should be evaluated against their ability to ensure those things happen. It is also important to evaluate if those things happening are actually making the workflow better.

User stories exist, mostly, to help teams better define the work to be done, document it, and create alignment on it. If user stories are getting in the way of the team effectively collaborating and defining work then consider if 1. you and your team are using stories correctly or if 2. user stories are a fit for your team overall.

Ultimately, team workflows work best when they are iterative and allowed to evolve based on the needs and structure of the team but they also need time to improve and find the right shape. Don’t be afraid to question if the stories you’re writing are adding value and remember why you’re writing them in the first place.

Internal tools get a bad rap. Many PMs find them to be “entry-level work” not worthy of their time and devoid of the magic and flash of consumer-facing products. The reality is, however, that high internal efficiency is a core aspect of a healthy company and a hallmark of a strong culture.

Growth is only possible when the team and products can support the scale of the user base. Without efficiency, you will create poor experiences for customers, drag down your product’s performance, and kill team morale (especially on customer-facing teams). With efficiency, doors open. Teams can then focus on key initiatives and you’ll get pulled less and less to deal with fires.

This guide covers how to maximize your time working with these tools.

A practical approach to building internal tools

I follow a basic framework for tackling internal tool projects composed of five steps:

1. Understand the problem and feel the pain yourself

2. Get your team to learn what you’ve learned

3. Find the right path for the solution

4. Execute with quality

5. Celebrate the improvement

1. Understand the problem and feel the pain yourself:

Quantitative Understanding:

The key question to understand with internal tooling is “What is the tangible value of solving this problem?”. If you understand the answer to this question very well, the solution will be easy to find. To answer this question you need to find the metric-based representation of the problem’s impact.

For Example:

Let’s say your company requires each new customer to be manually set up, and this setup is complex and time-consuming which results in customers not being set up within a reasonable time frame.

• Here you can measure the impact of this problem in a few different ways:

  • The total support time spent getting customers set up.

  • The number of days that it takes the average customer to get set up.

  • The percentage of customers being set up outside a reasonable time frame.

  • Ticket and call volumes from customers due to set-up delays.

  • Sentiment scores from customers who experienced this issue.

By creating a problem value statement that is metric-based all stakeholders can more easily align on the problem to be solved. This can create focus in your discussions as well as allow you to better understand how a proposed solution could impact the metric.

Not every problem can be reduced to a metric, many problems also have multiple value adds if solved, and some problems have immeasurable value. However, being capable of measuring a problem’s value in a clear way can make a huge difference in your approach and ability to focus.

Qualitative Understanding:

Sometimes PMs get overfocused on metrics as these can provide comfort to analytical-type individuals. However, a quantitative understanding of the problem is only a partial understanding. To truly understand the issue you need to experience it first-hand.

Here I look to do a few things:

Understand how the team feels and thinks about the problem.

• Get the perspective on the issue from stakeholders. In particular, talk to the individual team members facing the issue every day, their manager, and your manager. Find the misunderstandings and the overlaps, and understand how different people feel about the issue and why they feel that way. This will give you a clear understanding of your organization’s perspectives on the issue which can help you understand expectations and better navigate those expectations.

• If each stakeholder has a totally different understanding and evaluation of the problem to be solved, you’ll end up experiencing a lot of friction in the process as stakeholders may not understand why you’re going down a specific solution path.
  

Understand how the internal team is hoping the issue gets solved:

• 9 times out of 10 if you’re getting an ask on internal efficiency it’ll come prescribed as a specific feature that an internal team needs.

• While user-driven feature requests are many times not useful, internal team-driven requests can sometimes break this rule. In organizations that don’t invest much in internal efficiency you’ll often find that team leads will have created their own solutions to these problems in lots of clever, and sometimes not so clever, ways. This can help you fastrack your understanding of the problem and its solution.

• Don’t take the solution as gospel but don’t dismiss it outright. I’ve found with internal tooling that the average prescribed solution has tended to be a pretty good possible solution (although mileage may vary for you).

Understand the problem yourself by experiencing it first-hand.

• Upon taking all the feedback, you have to do the work yourself. There’s no better way to get your own opinion on the issue than living a day in the shoes of those dealing with it.

• Ask to take on the responsibility of one of the team members for a specific process or join a ride-along.

• See the problem for yourself and more importantly, feel the pain for yourself. This very hands-on experience will make the problem more tangible in your mind.

2. Get your team to learn what you’ve learned:

Depending on how your development process runs, you may have more or less of an opening to bring in other stakeholders. In the right environment, you can have most, if not the whole, development teams participate in the discovery process and gradually iterate on the problem through cycles of learning, building, testing/validating. However, I’ve found this to be exceptionally hard at most startups and in most cultures due to lack of time, process, skill, or all three together.

• If you’re in a more waterfall-type process, remember that you can bring your findings to the team as you’re consolidating them, while first-hand experience is best, a steady series of updates and documentation can help someone be more integrated in discovery even if they can’t participate directly.

• Another thing I like to do for these kinds of projects is to have the development team go through a simulated exercise that has them doing the work that the internal team is suffering through. Giving your development team first-hand experience of the issue and layering context to that experience makes the problem to be solved really tangible and gives them a direct relationship with that problem. This sets everyone up for a more effective discovery process and allows them to return to the exercise experience for guidance.

The key thing to achieve is to ensure the team understands the problem they’re solving to a similar degree as you do. If you’ve achieved this you’ll find that the outcome you and the team will deliver will in many cases be 10x what you’d get if only you understood the problem deeply.

3. Find the right path for the solution:

To find the right path you need to ensure that you’re properly weighing the variables of the problem and evaluating possible solutions accordingly. There are a few things you can do here:

Challenge assumptions, deconstruct and reconstruct your understanding of the problem:

Try to identify and deconstruct any assumptions that are being made about the problem and about a possible solution. Ask yourself

“What are we relying on to be true? Is that actually true”

“How do we make this problem completely disappear”

“What’s the 20% solution that delivers the most value”

“What’s the ideal solution to this problem”

I’ve found that pushing my thinking to extremes and challenging assumptions has yielded better solutions not because you end up building the extreme, you almost never do so, but because this type of thinking often helps cut through group thoughts, biases, and assumptions resulting in a clearer focus on the parts of the problem that add the most value.

Don’t worry about straying too far off course, it’s easy to get back on track if you understand the problem to be solved. But note that most teams tend to avoid challenging lines of thinking and intense discussions out of fear of conflict. Better solutions come from productive creative conflict, pushing the team’s thinking to challenge assumptions can help you do this in a positive way.

Test your solution hypothesis with the internal team, simulate it, and try to break it:

As you’re concepting the solution, bring in the internal team for feedback. Internal teams are the best kinds of users because most of the time you can bring them in as often as you’d like and they’ll spend as much time testing and talking to you as you need. Keep them in the loop at a minimum but consider taking advantage of their time and having them challenge the solution path you’re working through.

Once you have a clear path, try to simulate what the workflow would look like with it in place. If you can’t simulate it, run through the solution step by step with the internal team instead and evaluate how it performs or how the internal team would expect it to perform.

As much as you can, try to challenge the path you’re thinking of taking and try to make it fail. Look to find ways it could fail, and try to identify flaws by questioning it under different scenarios. If your path makes its way through, you probably have something that will work in the real world.

Go back to your problem’s key metric and evaluate how your solution would impact it:

For every internal efficiency project, I try to also put together a projection of what the world will look like with the solution in it. Go back to your initial quantitative metric for the problem and ask yourself “What is my expectation of the impact our proposed solution would have on the problem?” Putting together an estimate forces you to think through the value you’re solving for and the value you think you can capture. This type of calculation can help you refine your strategy and thinking around the problem.

Consider build, buy, change process, sunset, customer-facing options:

It’s very easy to fall into the trap of always solving problems by building, but remember building is one of many options to solve a challenge:

• Building: Ask yourself also if what you’re trying to solve is core to what you do or secondary (but still a requirement). Building isn’t a one-time cost, it presents a continual need to support and maintain what you’ve built as well as added complexity to your system. If you're building, ensure you understand the commitment involved and have determined it’s worth it over all other options.

• Buy: If something is not core to what you do and requires a high degree of expertise and maintenance, then buying may be a better path. Third-party solutions can be the right choice in situations where there are clear market solutions and where the problem to be solved is not core to your business.

• Change/sunset process: There are cases where you can change the process in such a way that much of the problem disappears. This isn’t very common but can be the case if the original process had a core assumption that you’ve proven to be untrue. If this is the case, sometimes you can work with the internal team to shift the process in a way where the problem you’re trying to solve essentially disappears.

• Make the solution customer-facing instead: While this can be a tougher sell, in some cases you can delegate part of the problem to the customer or position the solution in a way where the problem is removed via customer behavior. Consider if the problem you’re solving is actually just a symptom of a problem with how your product works for users and if a better end-user solution would mitigate the issue.

Ensure everyone is rowing in the same direction:

• Ensure that everyone who is a stakeholder in the project is bought in on your solution path and understands the reasoning behind it.

• Building buy-in will reduce the amount of friction you will feel during the project’s execution. If stakeholders don’t understand or believe in your proposed solution they’re likely to try to course-correct it which can lead to disruptions and tension.

• With internal tooling, internal buy-in is very important because internal teams have more of a say in product decisions than regular customers do (or at least are capable of communicating their thoughts and feelings directly). Before anything gets built, make sure the internal team is bought in as this will make the partnership smoother and more productive.

4. Execute with Quality

To ensure that your solution is the best it can be there are a few things to keep in mind when executing that make a big difference:

1. Test for real scenarios.

   1. Don’t just test and audit like a robot, think about how the tool will be used by your team and try to simulate it.

   2. If possible bring in the team for stress testing. Realistic testing will often find issues that pure auditing will not.

2. Get the internal team to be hands-on.

   1. Getting stakeholders to try out your solution in a testing environment early can uncover issues with your implementation. As development occurs, keep the internal team in the loop and provide them with visibility into the project as this will help catch issues but also increase morale.

3. Use realistic data and volume (stress test as much as you can).

   1. Don’t just test for an optimal scenario, make it as realistic as possible and test for extremes.

   2. It’s easy to miss something because you’re testing with data that was convenient. Ask yourself, what does the worst real-world scenario look like, now imagine it’s twice as bad. Test with the date from that imagined scenario.

   3. Also, ensure that your team members are testing with different data sets, it’s easy to miss issues when everyone is working with the same data.

4. Don’t throw your solution into the wild at the wrong time.

   1. Whatever you do, don’t try to rush a tool at the last minute to support a workflow. Tools need iteration and are generally never perfect on the first release.

   2. Give teams time to ease up to the tool. Don’t go live on the worst week of the year. Shipping at the wrong time can create a strain on teams and decrease morale.

5. Build some padding in your roadmap for follow-ups.

   1. It’s tough for some internal tools to be fully tested without having real-world usage. Give yourself some space after release for iteration and for addressing any issues found in the wild.

6. Be realistic and honest about what you can and can’t get done.

   1. It’s normal in any project to be constrained in terms of the investment you can make. Acknowledge that you won’t be able to solve every problem and be honest with internal teams about this.

   2. That being said, don’t lose perspective on the value you can create and how it’ll affect that team. Focus on that value and keep that focus during the development process.

7. Build support materials and documentation to ensure the solution is understood.

   1. This applies to any kind of work but it is especially important for internal tooling. You should ensure that you have clear documentation explaining what the internal tool was built for as well as what it does, and how it does it.

   2. Try to be as detailed and technical as possible in your documentation. Without documentation, the likelihood of forgetting key details about the tool is high which can lead to issues down the line.

8. Have a clear tracking plan around your metric to learn about how your solution is performing:

   1. Before going live ensure that you have a way to understand if the solution you’re putting in place is effectively addressing the problem.

   2. Understanding the solution’s real-world impact on the problem’s metric can help show you how to improve the solution and validate if you made the right decisions and assumptions in your process.

5. Celebrate the improvement:

Building internal tools is a highly collaborative, highly personal endeavor as the tools you’re building affect the internal team’s daily lives. Ensure that wins with internal tooling are recognized and come back to the value they’re delivering. Praise those who participated in making the tool better both on your team and on the internal team who will be using it.

Over time you’ll find that the internal team will be energized from the tooling and your working relationship will grow stronger. Remember that while projects like this can sometimes be draining, they often mean a lot to those who benefit from them.

Ultimately you’re trying to create valuable products for people, internal tooling may have a small audience in many cases, but the work you’re doing optimizes those people’s work lives and helps your business grow.

How to Experiment:

The easiest mistake to make with experimentation is to set the experiment up in a way where nothing is learned. The vast majority of the time, this happens due to improper setup.

In particular, there are three key areas to consider when it comes to designing an experiment.

• The people in the experiment.

• The structure of the experiment.

• The tracking and measurement of the experiment.

This guide will dive into each area and what to consider at each level.

The people in the experiment:

When you set out to conduct an experiment you’re generally trying to prove that something is true for a large group of people.

The experiment is then testing your assumption about/on that population. In most cases, you can’t/shouldn’t test on the full population as it will have adverse effects on the business. If you send a new test email to all your users, for example, you might be exposing everyone to a poorer experience and you will also have to wait in order to be able to test a new version of the same email. To avoid this problem, you can build sample groups from the larger population and test on those groups.

Sampling:

By taking a sample group from the larger population, you can ensure that the group you’re experimenting with will yield results that will also be true for the full population. At the same time, by breaking up your population into smaller groups you’ll be able to run more iterations of the experiment than if you just tried on the full population from the start.

To create a sample you have to define two things:

• The characteristics of the people in the sample.

  • Practically speaking what are the properties the population shares that make them distinct from other users?

  • Ideally, these characteristics are things you can easily use to segment your users in order to build the sample.

• The amount of people you need in the sample to get accurate results.

  • The sample size calculation for the majority of experiments can be done using a simple sample size calculator (here’s one from SurveyMonkey that I’ve used in the past).

  • These calculators can help you ensure that your sample is big enough to generate results that are very likely to be representative of the results you’d get if you were exposing the full population to the experiment.

  • They also can help you better understand how statistical analysis is done and how to interpret your results.
    

What if my population is made up of lots of different groups?

In some cases, you might be dealing with a population made up of different sub-groups each with their own unique characteristics.

In this case, you have to ask yourself if your hypothesis could be influenced by the particular properties of those groups. If it could be, the next question is if it warrants a different experiment or if you can just have that differentiation be something you include in your analysis. In most cases, this differentiation can just be a variable in your analysis as it won’t materially affect the experiment design. If your experiment is directly designed for a specific sub-group then you probably should segregate the user types.

What if there are very few people in my population?

In some cases, the population is too small to build a proper sample. In these cases, you’re generally better off just testing with the full population as the sample size requirement will essentially be very close to the full population size.

Sometimes, you can shift your experiment to focus on frequency to build confidence. For example, exposing the same user group to the experience several times over the course of a long window of time. That being said, I’ve found that going with the full population instead has tended to be the right path in these scenarios.

Focusing on a very small number of users:

There are cases (as I’ll cover below) where you’re trying to understand how to build something new. In these cases, sometimes your best path is to focus on how to manually support your intended experience for a small number of people and learn from their experience.

The thing to keep in mind in this context is that any learnings you’re making with a small number of people have higher risk since there is a chance the people you’re working with are outliers and the group is too small to cover the full variations of behavior you’d get from the larger population.

This shouldn’t discourage you from attempting this approach as you can always bring in a bigger sample to try your new experience later on.

The structure of the experiment:

The structure of the experiment is the mechanism through which the experiment is conducted. There are some common templates that people gravitate towards for conducting experiments:

1. White Glove Service / Concierge Treatment:

   1. Here you’re manually supporting an experience in order to understand what the same experience would look like and how it would perform if it were built.

   2. Imagine you’re trying to understand if it’s worth building a recommendations feature for your product. In this design, you’d manually curate recommendations for a small number of users yourself. This would teach you both how to best make recommendations as well as the user impact of having recommendations in their experience.

   3. Best suited for: Validating if something is worth building.

   4. Downsides: You’re generally limited in your sample size since you’re manually supporting this experiment.

2. A/B/Multivariate Testing

   1. These are scenarios where you’re running one or more versions of the same thing against one another to determine which one performs best. Within this format there are two main kinds of design:

      1. Big swing experimentation

         1. Here you’re testing out a completely different concept from the existing implementation.

         2. Imagine you have an onboarding experience that has mediocre performance. In this design, you’re coming up with a completely different approach for your onboarding experience and testing it in parallel with your current onboarding flow. This allows you to try to establish a higher floor of performance than what you previously had. You won’t know exactly why it’s better (because you’re essentially testing a totally new version and can’t measure what about it made it better) but better is better so you’ll take it.

         3. Best suited for: Cases where you feel like a big pivot is needed but aren’t ready to replace what you had.

         4. Downsides: Your new version has a tendency to underperform if it’s going up against something that has been deeply optimized. You won’t learn why you got better/worse.

      2. Variable based optimization

         1. Here you’re focusing on a key variable, changing it, and testing the changed version against your existing version.

         2. Imagine you have something that works pretty well but you feel like it can be better. This approach allows you to try to improve but still know exactly what led to the improvement by focusing changes on only a single variable.

         3. Best suited for: Optimizing something that’s already working.

         4. Downsides: If you’re trying to quickly improve something this generally won’t be an effective approach as it’s highly focused on a single variable and it can take many changes to improve significantly.

3. Buffet Style Experiment

   1. Here you’re presenting the customer with lots of options in a cohesive way and seeing what options they pick.

   2. Imagine you have multiple filters you’re thinking of building but you’re trying to figure out which one should be built first. In this experiment design, you are finding a way to expose the customer to all the options you’re thinking about and see which ones they gravitate to the most. You’ll know based on the engagement with the selection which ones are most immediately compelling to the user.

   3. Best suited for: Figuring out what things users gravitate towards.

   4. Downsides: It’s very easy to structure the experiment in a way where there’s a bias towards one of the options (leading customers to gravitate towards it0 which can mislead you.

Experiment Duration:

When it comes to experiment duration the key questions to consider are:

• What is the length of time that it would require for all users in my experiment to complete the experience/experiment path?

• Given that the necessary time has elapsed, have I achieved meaningful results?

  • For A/B types of experiments, you can use SurveyMonkey’s statistical significance calculator to determine if your results are significant.

  • Statistical significance will tell you that the results you’ve experienced with your sample group are likely to be experienced by the full population if that population were exposed to the same conditions.

  • For other kinds of experiments if you’ve built the sample correctly then you should be able to assume that the results you’ve seen with your experiment are representative of the results that you’d see with the full population.
    

If you don’t have clear results then you can either add more people to the experiment or run the experiment for longer. In cases where you have results that don’t seem better or worse than your baseline, you may be able to conclude that the new version is not better and focus on coming up with new experiments.

Experiment Medium and Fidelity:

For experiments around features, it can sometimes feel like you’re blocked from experimenting because you’re unable to deploy experiments in the same medium or with the same level of fidelity as your ultimate feature idea.

In many cases, you might be able to successfully experiment using other mediums (email being one of my favorites) you just have to ensure that it’s a reasonable assumption that the behavior would translate across mediums (ex: people would engage with this email in a similar way as they would engage with the in-product version of this feature).

For in-product experimentation, consider investing in feature flagging or other similar techniques to be capable of experimenting with select groups of users. In my experience, these capabilities have significantly increased the range of experiments available and have yielded the most accurate results since they allow you to get in-product engagement.

In terms of development try to have higher fidelity on the customer-facing portion of the experiment and hack together anything else that’s not visible (might as well because it’ll be cheaper and if the experiment doesn’t pan out you’ll likely have to throw it away).

Ultimately, it’s important to note that the experiment is a good enough proxy for a potential feature or process. It won’t ever perform exactly like the real thing, but it will ideally give you insight into how the real thing may behave if built.

The tracking and measurement of the experiment:

Sometimes in the excitement to try things we forget that without the right data at the end of the experiment, we won’t be able to take anything from it.

Before the experiment starts you should have a clear understanding of:

1. Key metrics and how to track them

2. What success and failure look like

Key metrics and how to track them

The experiment is essentially an exercise to extract data that is reliable for analysis and drawing conclusions. Without the data, the experiment is essentially throwaway. 

Generally, in terms of metrics, I like to have:

• One or two key metrics that are the clear guide for the experiment’s success or failure.

• A few additional metrics that are interesting and could affect my understanding of the results.

These metrics should all be retrievable by you at the point where the experiment is finished. This means that before starting you should know exactly how you will retrieve these metrics.

The key is to set up your data analysis process (how you will acquire those metrics and analyze them) before the experiment starts and not during or after (can’t stress this enough). Every time I’ve neglected this rule I’ve found myself missing key data points in my analysis and had to do further experimentation.

Quantitative + Qualitative

Many times I like to couple my quantitative analysis with qualitative analysis for a more well-rounded picture. Here you can often include surveys as part of your experiment (at key points that don’t disrupt the experiment) and schedule interviews/sessions with customers to further discuss their experience with the experiment. If you have session tracking tools, those can be very valuable for understanding user behavior in your experiment (although nothing beats getting feedback from customers directly).

What success and failure look like

It’s always valuable to determine what success and failure look like for your experiment as well as determining if you’d be happy with the success state. This success/failure should be based on the key metric(s) you’ve selected and be something tangible like hitting a certain threshold on that metric(s).

Once you actually run the experiment, coming back to this success and failure state can be enlightening. Evaluating your results against your initial expectations can help clarify what you’ve learned and how your expectations have changed since running the experiment.

Why some organizations are averse to experimentation:

You might have had the luck to work at an organization that doesn’t see experimentation as a valuable activity (you might be working there now). This write-up intends to provide some tools for breaking through that aversion and getting your experiments off the ground.

If you work at an experimentation-averse organization, the reasoning against experimentation sounds a bit like this:

“It’s too hard and too time consuming.”

“We already know the answer, why do we need to validate something we already know?”

“Experimenting is going to take away time from more important things that we need to focus on.”

“We need to move quickly and experimenting is going to slow us down.”

“This is going to create a poor user experience for our users.”

”We don’t have time for this.”

While the reasoning may sound different from situation to situation, it often translates to the same idea:

• The organization believes that its idea-to-value flow is already effectively yielding results. As a result, the organization doesn’t see a reason to introduce more mechanics to this flow.

While in some cases product managers and other individuals can “boil the ocean” with their research and experimentation, in most cases experimentation aversion stems from a lack of exposure to effective experimentation and a sense of security towards the company’s past way of working.

To get past this aversion, experimentation needs to be demystified and better understood.

Opening the door to experimentation:

In my experience, there’s always a gatekeeper for experimentation. They can be someone who’s directly managing you and is accountable for your output and time or they can be someone who is the owner of a key tool you need to leverage in your experiment.

In order to get experimentation on the map you need to convince this person that your effort is worthwhile while simultaneously aligning on a reasonable cost investment for your experiment. The following are a few things to keep in mind when having these conversations:

1. Make sure your reasoning for experimentation is rock solid:

   1. The key thing going into any conversation is to have clear-cut reasoning for the experiment. This means a few things:

      1. You have a deep understanding of the issue you’re looking to experiment on but you’re blocked from fully understanding the issue due to missing key insights.

      2. You’ve attempted to gain those insights via other suitable means and hit a wall and the experiment has a high chance of unlocking those key insights.

      3. Having those insights will improve your chances of future success or mitigate significant risk.

      4. You have a clear idea of how the experiment would work that is reasonable in terms of cost to setup and duration.

   2. If the above items are covered then you’re starting your conversation from a position of strength.

2. Make sure to explain your reasoning for conducting an experiment:

   1. In many cases, there’s an asymmetry of information between you and the gatekeeper when it comes to your work. While you’re spending considerable time in the weeds they might be highly focused on the “bigger picture” or overall not invested in your area of work.

   2. It’s then important to establish a shared foundation of knowledge so that the gatekeeper understands why you ended up deciding an experiment was necessary.

   3. Once that shared knowledge exists, it’s then time to explain why an experiment would be worthwhile (what risk you’re mitigating for, what upside you might verify, or overall what value you’re expecting from an experiment).

   4. Ideally, by doing this, you’ll have built enough buy-in to experiment, but some situations require further handling and convincing. In those cases, you need to be more thoughtful about the gatekeeper's concerns and philosophical disagreements with your reasoning.

3. Minimize cost/loss-of-focus concerns by addressing them head-on:

   1. Many organizations opt out of doing experimentation because they believe that the cost of setup for an experiment is too high. This leads to people opting to execute the project version of the experiment instead.

   2. There is an obvious paradox here, the project version of an experiment is, almost always, significantly more expensive to execute. What organizations really mean here is that running an experiment has costs outside of the costs they generally pay and requires effort outside of the effort they generally exercise.

   3. Here you have to align with the gatekeeper on what an acceptable investment looks like (in addition to the potential value and reasoning) since this will be an investment outside of the ones they’re used to making. If the gatekeeper believes you won’t over-invest they will often be open to the experiment (even if it’s just to appease you).

   4. Think about this as a resource allocation problem for the gatekeeper, and work with them to identify the acceptable time and capacity allotment for the experiment. If the gatekeeper understands you’re being mindful it’ll make them much more open to trying things out.

4. Don’t make it a battle of beliefs. Instead, focus on the upside/downside of the experiment:

   1. Many times the reasoning for not running an experiment ends up being about the gatekeeper’s belief around a key variable or idea the experiment is looking to validate. If the belief is not based on fact, and you can’t get the gatekeeper to realize that, then you might need to pivot away from beliefs and focus instead on the upside/downside of the experiment.

   2. Here are a few pathways for discussion that can be more productive:

      1. In cases where you’re experimenting to validate a feature. Focus on how much can be learned via the experiment and how expensive and risky it would be to only learn as a result of a release. The experiment may help you minimize scope, hone in on the most valuable aspects of the feature, and prevent you from having to do follow-up work.

   3. In cases where the experiment is only seeking to increase your knowledge base, identify the possibilities that could come out of the insight and use those possibilities to help the gatekeeper open their mind to the experiment. Most people love discovering new things especially if those new things can help create more value, key in on that value as fuel for your experiment.

   4. In cases where there’s risk, focus on helping the gatekeeper create a clear idea of the bad things that can happen if you move forward without experimenting. Make this risk very tangible to them so that they can imagine the world where the risk is realized. From here you can use experimentation as the risk mitigation tool that will help them prevent that risk from realizing.

5. Get other members of the team to lobby for the experiment:

   1. If other stakeholders of the team also believe in the experiment ask them to voice this to the gatekeeper. Having multiple voices echoing the same ideas can have a compelling effect on the gatekeeper’s thinking about the experiment and its value.

   2. Keep in mind that you’re not the sole owner of the experiment, the gatekeeper and the team working on the experiment are also owners of that work. If everyone rows together with conviction you’ll find that a lot of the friction in this process will start to dissipate over time.
      

Pick your battles:

In some cases, it might not be worth spending your equity on pushing experimentation. This is especially the case when the learning is low-upside and you’re still on shaky ground with experimentation as a concept within your org.

On the flip side, if you have something you really believe in, something that’s very polarizing in the organization, or something that feels very risky to the team then those tend to be good areas to invest in since you’re likelier to uncover insights and as a result validate the value of experimentation.

Being thoughtful about when to use experimentation can sometimes be the most valuable thing to counter aversion. Experiments with pay-offs are ultimately the best path towards combating aversion as they can be used as proof that experimentation is valuable.

Asking for forgiveness instead of permission:

It needs to be mentioned that there are pathways to experimentation where permission isn’t necessarily granted. Sometimes organizations work themselves into a dysfunctional workflow that prevents them from unlocking further value and become unable to improve on that workflow. In a few rare cases, learning the truth about something that is holding the company back is worth crossing a line. That being said experimentation with buy-in is always the best version of this work and the goal should be to make experimentation an aligned approach.

There are some key things here to keep in mind / avoid when swimming in these murky waters:

• Don’t run an experiment for something you’ve been explicitly denied permission for (changing a small variable on your original plan does not constitute a totally different experiment either).

  • You’re just going to piss people off and it’s kind of an asshole move. The upside is seldom justified and the downside is massive.

• Don’t do it as a practice since you’ll just get permanently shut down.

  • The undercover approach should be a rare occurrence, a card you can play when you really need it. If you keep doing it, again, you’re going to piss people off and come off as an asshole.

• Don’t do it if there’s a real negative downside.

  • If you’re affecting customers negatively, if you’re putting another team in a bad place, if there are real negative consequences to your experiment then it’s probably not the right path.

• Don’t talk about it constantly, don’t formalize it.

  • If you make a big fuss over the thing… You’re probably going to get the thing shut down.

• Don’t overinvest.

  • What’s worse than doing something you shouldn’t be doing? Doing it a lot. Think about this experiment as low-stakes research that you’re doing on the side, not as your big project.

• Know what you’re doing:

  • If you don’t understand your business, customers, teams, and strategy then don’t do it, you’re very likely to mess things up for everyone. Do the work before crossing the experimentation line.

Again, a bought-in group is always better than a group where everyone is going their own way. While experimenting without consent can yield results, it’ll almost never yield buy-in. Make people stakeholders and share the wins, this is most often the path toward an experimentation-focused culture.

Lexicon for convincing people to experiment:

I often find that a concept is only useful if I have the words to express it. With that in mind, I wanted to share some of my favorite lines when talking about experimentation and what they intend to achieve. Hopefully, this helps you build your own language and script for these kinds of conversations:

• “While we have a good idea that this is true, an experiment will help us learn more about how to build it effectively saving us from having to do more iterations after launch.”

  • Why it works: You’re moving the conversation away from the assumption and instead focusing on adding value (by better understanding how to solve the problem and doing it in a cheaper way than releasing).

• “I know that you’re not as high on this as I am and that to you this feels like a bit of a waste of time. Is there an amount of time I could invest into this that you feel would still be ok?”

  • Why it works: You’re disarming the gatekeeper by addressing a big concern head-on (by doing this giving them confidence you’re going to be thoughtful about this initiative). You’re then collaborating with them to find a suitable middle ground and asking them to problem-solve with you vs approve/deny you.

• “If we’re wrong about this, why do you think that would be the case? What’s the worst scenario for this? What could we have missed? Would we be open to testing that to be sure?”

  • Why it works: You’re pivoting the conversation to focus on the downside instead of assumptions and using the feelings/thoughts tied to the downside to help fuel the desire to experiment.

• “I really believe that I can find some valuable learnings here and minimize some potential risks… I’d like to spend some time on it unless you’re opposed to it?

  • Why it works: You’re making your belief and passion explicit and essentially asking the gatekeeper for an opportunity to pursue this passion. They may be inclined to allow you because they know you’ll be more bought in as a result of having been allowed to do this.

• “The thing that makes me want to do this isn’t necessarily just getting to this answer, but it’s instead what this answer unlocks. Think about what we could do if we validate X… We could do this thing, that thing, maybe even this other thing…”

  • Why it works: In cases where the key insight isn’t as interesting to the gatekeeper but it is to you, focusing on what it unlocks helps the gatekeeper understand why you’re excited. Making the possibilities more tangible can help the gatekeeper see your experiment in a totally new way and as a result, be more bought in.

Parting thoughts on getting buy-in:

• I’ve found most people are excited about uncovering new insights, being proven right/wrong, and preventing mistakes from happening. The experimentation process provides that potential for excitement and it’s partly your job to foster enthusiasm around that process.

• Cultivate excitement with those that are experiment averse and you’ll sometimes find that they’ll become advocates of experimentation over time. With enough successful experimentation, the culture of the organization might start leaning towards experimentation.

• While the road to get buy-in might be hard, it wouldn’t have been possible without the gatekeeper’s approval. Remember to always validate the investment the gatekeeper has made in enabling you to experiment as this will help them feel ownership of experimentation and can help turn them into a long-term believer.

• Finally, remember that most people are trying to do their best for the customer/product/company/team. Think about gatekeepers as partners and you’ll find ways to grow that partnership. Think about gatekeepers as blockers and you’ll often find yourself running into their walls.

1. How to: Identify a good experiment opportunity ← You’re Here

2. How to: Get buy-in for your experiment

3. How to: Run an experiment

4. How to: Report your experiment’s findings

Why experimentation matters

It’s easy to confuse facts with beliefs because both of them feel the same and sound the same. However, while facts are by definition always true, beliefs are only sometimes true.

The ability to discern fact from belief is then a key skill to uncovering the world’s truth and as a result, ensure your product matches that truth.

Experimentation is one of the key tools that allows you to explore and map that distinction. It is, in essence, a tool for fact-finding. It is also, in the world of product development, generally a lower cost, lower risk, and lower commitment option than its alternatives.

A general framework for what experiments are good/bad for:

It’s important to establish that experiments aren’t the perfect tool for every job. Experiments are very well suited for some things but poorly suited for others.

The following is a list of the different things experiments are well and poorly suited for:

1. Well Suited:

   1. Simulating how a feature may perform if built.

   2. Iterating on a pre-existing solution.

   3. Mapping customer behavior for a given scenario.

   4. Exploring the productization of an insight (along with testing that insight’s validity).

   5. Validating if a certain product or process can be removed.

2. Poorly Suited:

   1. Solving a business problem long term.

      1. Ultimately you’ll need a product/process to be created to fully solve most problems.

   2. Understanding users, markets, behaviors, and journeys (in some cases).

      1. Generally, you’re better off conducting user research than using experimentation to validate certain kinds of insights.

      2. Additionally, if you haven’t looked at data and/or talked to customers then you might be skipping a step by running the experiment.

   3. Simulating scenarios that are very long-term, intangible, or high risk to the company:

      1. Certain scenarios are not well suited for experimentation, if there’s considerable risk to your company, if the insight requires a lot of time to validate or if you’re trying to test big intangible decisions like partnering/acquiring/merging with another company then experimentation is likely a bad tool.

If your experiment idea falls under the “bad” list then ask yourself: 1. What is the goal for the experiment 2. Is an experiment the best way to get there?

Additionally, I want to stress that it’s worth experimenting when the insight is worth knowing in the first place. If the insight is trivial or can otherwise be learned in easier ways (like data analysis, customer interviews, usability testing, etc) then experimentation may not be worth doing in the first place. This seems obvious but it can be hard in practice to make this deliberation

A strong hypothesis

This is arguably the murkiest part of experimentation because different people can have different opinions on what constitutes a strong hypothesis. I like to think that a strong hypothesis is a hypothesis that:

1. Is logical and at least a few others would think your premise has validity.

2. Is based on first-hand experience or second-hand reporting from customers of the conditions, customers, and environments at play.

3. There’s a reason confirming/denying the hypothesis is valuable, knowing this thing will give you the necessary information to make an important decision.
   

Types of Hypothesis:

This is not an extensive list of the types of hypotheses but some of the more common patterns that come to mind:

1. X happens because of Y

2. When exposed to multiple options, customers will pick X

3. The outcome is better if x is changed

4. What if X existed/no longer existed?

X happens because of Y

Example: People buy the Caesar salad because it comes with croutons.

• In this context, the experiment is the method by which you validate that x does happen because of y and to what degree y can impact x.

• Notice that I used a simple relation of cause and effect, this is because the majority of the time multiple relationships of cause and effect are very hard to test in an experiment.

  • If you’re trying to prove multiple relationships most of the time you’re better off designing multiple experiments and performing those independently as this will give you the best odds of getting valuable data from your efforts.

When exposed to multiple options, customers pick x:

Example: If provided with multiple kinds of salads to pick from people would pick Caesar salad.

• In this scenario we’re essentially seeing what happens when we expose customers to multiple competing variables and we’re measuring which variables customers gravitate towards.

• This is a tricky type of hypothesis because the results are very experiment design dependent. It’s easy to tip the scales towards one variable or another. However, in my experience, it’s a very helpful question to ask that if validated allows the business to properly weigh the importance of different variables in their product.

• It’s also important to note that most times the experiment won’t give you the answers on why customers pick x, it’ll just tell you that they will. You’ll have to conduct research to truly understand the why.

The outcome is better if x is changed:

Example: We’d sell more Caesar salads if they came with smoked salmon.

• This is a classic optimization hypothesis here you’re just testing that a change to the product would lead to improved results.

• Again, valuable question to ask that is very dependent on the execution but when approached correctly can lead to real improvements in the product experience.

• The size of X is also variable. Many times people think of optimization as only being able to change a single variable to avoid inconclusive results. In some cases, if your performance is poor you’re better off experimenting with a completely different version of your experience to try to establish a new floor of performance that’s higher than your previous experience. While you won’t know exactly why the new version is better, in some cases, you’re better off optimizing something that performs well than something that performs poorly.

What if x existed / no longer existed:

Examples: What if we sold Cobb salad? / What if we stopped selling Caesar salad?

• If you’re thinking about building a new feature then an experiment can be a very valuable way to determine if you should follow through with your plan.

  • In this scenario, the experiment is then attempting to measure/understand how customers would interact with that feature if it was built by creating a simulated version of that feature.

• On the other hand, experiments are also really useful to prove that you can sunset features that you believe are no longer viable.

  • In this scenario, you’re in most cases hiding or removing the feature from some/all customers and measuring the results (seeing if feedback comes in, seeing if there’s a change in engagement, etc). This de-risks sunsetting features significantly.

Following your gut (or someone else’s)

I tend to find that good experiments are driven by very strong feelings or very negative feelings.

If I have very strong feelings about something it tends to be worth digging further. This isn’t because I’m some sort of savant but because strong feelings tend to be indicators of somethingness. What that somethingness is or if it’s actually there is unknown, the experiment will bring that out.

That somethingness many times is a set of patterns, behaviors, trends, or mechanics that we’re unable to quantify or explain at the time but that can eventually form the basis of something in the business.

Strong feeling driven experimentation can often yield positive outcomes in one way or another:

• When you’re validated in your feelings you can add more ammo to whatever case you’re trying to make.

• When the results are negative you’re then faced with confronting the reality of the outcome and learning from it. Many times these poor outcomes lead to important changes in perspective and direction.
  

While positive strong feelings about something are valuable, negative strong feelings are just as valuable. Annoyance, frustration, impatience, insecurity, fear are all great indicators of somethingness. People can be very right but they can also be very wrong about things, flipping your mindset to take advantage of both your beliefs and anti-beliefs is an easy lever to increase the amount of opportunities you’re identifying.

Leveraging other people’s gut feelings

All of this goes for people you trust, I find that in any team there are a few people who tend to have a general prescience about things. Those people’s feelings can also act as a compass for the direction to focus on. It’s important also to factor in the different exposure areas for different people on your team, a designer, a developer, a customer success specialist, a sales person with the prescience trait will often have feelings about very different areas of the business or the experience. Actively seeking to hear their opinions and thoughts about the product and business can often yield valuable opportunities.

It’s also important to highlight that things that many people consistently believe have a fun tendency to be wrong (often very wrong). This is because shared beliefs are often built on a tradition of knowledge and that tradition of knowledge is often based on opinion. These are very good experiment areas because they are often located at core areas of knowledge of the business and identifying misalignments in those areas can help the business find the things they should actually believe.

I have found that central ideas to mine and other people’s decision making were in some cases inaccurate or generally incomplete. They made sense conceptually and felt completely rational and beyond reproach and as a result we assumed they were self-evident. When you get a big enough idea like that wrong, you can spend months or even years going down less valuable paths. To find these I ask myself questions like: “What would really suck if we were totally wrong on”. This often yields the highest value areas to validate.

Story time:

For a period of about 2 years I was convinced that there was a pricing problem with a product I was managing and I had a strong belief that our pricing was a key blocker in our growth.

I debated and lobbied for an experiment to test this assumption and eventually was granted the opportunity. We found a clear way to test multiple different pricing options (all lower in different ways) in a cohesive experience to the customer. This was a several month experiment measuring short term conversion rate and long term retention.

When the results finally materialized the original pricing I had been so strongly against was by far the best option far outperforming everything else both in conversion rate and retention.

After a good amount of analysis trying to disprove this reality I settled on a new reality - pricing wasn’t the problem. This insight was fundamental for me and others to move forward in new directions which did prove fruitful.

Had we changed pricing from the start we would have likely taken significant losses and learnt that lesson the hard way. Had I not done the experiment I’d probably still be arguing for it today convinced pricing was the blocker for the growth I was seeking.