“Shifting To An Experimental Mindset: The Dos And Don’ts Of Hypothesis Testing” with Teresa Torres, Product Consultant & Coach, former CEO
By Lisa Rathjens
December 2014 Event
“Wisdom is the balance between knowledge and doubt.” It’s not often that product managers are encouraged to actively question their confidence in their own product ideas. But doubt is exactly what product managers need more of, according to Teresa Torres.
Teresa is a product coach who helps teams adopt user-centered, hypothesis-driven product development practices. On Wednesday, Dec 3rd, she challenged a packed room at the Oshman Jewish Community Center in Palo Alto, CA, to make doubt the center of their product management practice. Reminding the audience that pure confidence in a idea does not make it a successful one, she encouraged everyone to get uncomfortable, to step out of their “coherent stories”, and to challenge core assumptions.
Torres believes hypothesis testing is critical to the success of any product. As product managers, we need to keep the following truths at the center of our decision-making as we experiment to test and validate assumptions:
- We cannot predict the future.
- We will be wrong.
- We do not know what to build next.
- We do not know when we can deliver it.
Once you’ve acknowledged these truths, it follows that you’ll need to change your methods to be able to deliver great products. Instead of writing ~50-page PRDs, shift to building products iteratively (agile helps here). This allows you to learn and adjust, based on customer feedback and hypothesis testing. For instance, instead of building 10 features only to discover in the end that just 3 work, it’s better to run 10 experiments and then only build the 3 that work. This saves time, resources, and money. So, you see that it’s necessary to experiment at the feature level as well as the value-proposition level.
But how do you effectively test a product idea? According to Teresa, there are five components to a good hypothesis. In order to support (or refute) your idea by experimentation, your hypothesis must contain these attributes:
- What is the change that you are testing? Be very specific about what your expectations are and what you want to learn from the change.
- What impact do you expect the change to have? The expected impact should clearly define what you expect to see as a result of making the change. Be clear ahead of time with what measures you will use to gauge success.
- On whom do you expect it to have the most impact? Only test your hypothesis on those that you want and expect to impact.
- How much of an impact is needed to be successful? For instance, if you need a 25% conversion rate and your tests show you’ll only reach 19% conversion, then you cannot claim success.
- How long is the testing window? Your testing window cannot be arbitrary, or you might be fooled by false positives. Specify your test duration before you begin your testing and stick to it.
Then, when you set up your tests, be sure you can answer these questions:
- Is your expected impact specific and measurable?
- Can you clearly explain why the change will drive the expected impact?
- Are you testing with the right population?
- Did you estimate your targets based on something specific?
- Did you use a duration calculator?
These are the first steps toward successful hypothesis experiments, and ultimately toward more successful products.
Lisa Rathjens focuses on designing and building products and services that encourage and enable developers to build experiences that people love to use. She can be reached at lrathjens@yahoo.com and http://www.linkedin.com/in/lrathjens
#CX #UX #Design #Development #Startups #Entrepreneurship #Ideation #Market_Analysis #Metrics #Performance #Marketing #Tools #Readiness