Generally, experiments and hypothesis testing are a neat way to get forward within a very uncertain environment, if done right. A very common misconception is that once an experiment is conducted, the path on which to proceed gets super clear. Unfortunately, that is mostly not true.
When starting new ventures or products, be it in a corporate or startup, there are millions of assumptions in the room. If not tested and challenged, assumptions can be dangerous. They can ultimately even lead to failure of your endeavor. We very much like how Alan Alda put it:
“Your assumptions are your windows on the world. Scrub them off every once in a while, or the light won’t come in.”
Alan Alda
Some people are so overtaken by their assumptions that they take all of them as a given fact. No need to challenge them, they know how the solution looks like anyways. This is the single biggest learning that one can have going through the process of exploring opportunities to solve problems for a specific target group. You are biased and you don’t know nothing.
If you are still here, not neglecting what we just wrote, you are probably on the same page as us. Perfect! So, let’s get to the juicy parts. We would love to talk about two things: First, when should you use qualitative and when quantitative experiments and second, what are the most common pitfalls when experimenting?
Explore qualitatively, prove quantitatively
When founding a startup, or any corporate innovation project, start experimenting qualitatively. Why? Because just by talking to people, you can find out why they are doing, thinking & demanding certain things. In the beginning, it is not so much about proving different aspects, it is more about empathizing and understanding your target group.
Once you have a rough idea and feel confident enough to test certain parts of a business model or a product, you can start to verify or falsify assumptions that you might have via quantitative experiments. This is where you can rather observe and see if everything works and is understood the way you intended it.
Now that we got that out of the way, lets address the most common pitfalls of experimentation.
We would argue that if an experiment does not consider all those points, it is obsolete. Sadly, the majority does lack at least one of them, but hey… You did an experiment and that’s what counts, right? Quite the contrary. Gaining the wrong insights can just lead you either in a completely wrong direction, or worse, in the direction you wanted to go.
So what are those pitfalls now, you might be asking?
Find “the one specific thing”
The most important aspect of an experiment is that you want to achieve maximum insight with minimum resource input. We know, seems very obvious, but trust us, it is not. We have seen it multiple times, also when designing MVPs, that the natural thing is to add “just this one feature”, and one more feature, and just one more… We understand, this is very tempting to do and as long as you are not testing it with customers and talking to them, you are still in your comfort zone. And you didn’t fail, right?
If the customers are not inclined to use your solution for that one specific thing that creates benefit for them, even in a very rudimental version, they will not start using it once you add 4 more features into the mix.
Build, MEASURE, Learn
Always have in mind that if you don’t measure a specific outcome in quantitative experiments or ask questions that are not clearly targeted towards validating your hypotheses in qualitative experiments, you will still be guessing and basing decisions on your gut feeling, even if you did an experiment.
We would strongly advise to test your experiment a couple of times qualitatively to observe every step of it. Also, it can help to have one person focusing on measurability of the outcome of an experiment. Make sure to challenge it whenever possible!
Consider their situation
We saw tons of people conducting online experiments, completely disregarding how the customer feels and what they are looking for when going on a webpage, or when opening any link really. Especially when they get approached on the street, most people are initially irritated.
Make sure to always reach people in the right situation. Obviously, this is more relevant for quantitative experiments, like for example a fake door test, but for qualitative experiments it is also crucial that your opposite is in the right state of mind and not just hypothesizing.
The “skin in the game”
Think about their “skin in the game”. Whenever there is no commitment from people within an experiment, it is always easy to say yes. Never ask: “Would you buy this?” or “Do you think you would use it?”. Instead, get their commitment so that you know for sure they will buy or use it. Adding “skin in the game”, in form of an email address, other data or a commitment to testing your solution, really helps you to get the security you need to make a confident decision. It even helps you to think of next steps already.
A hypothetical “yes”, mostly a “no”
Last, and this goes hand in hand with what the Mom Test preaches. Never ask about hypothetics in the future. Even if people would love to, they will not always be able to give you the true answer. Always go for specifics that happened in the past so that people can relate to certain situations the experienced. Try to bring them in a mental state where they can remember a specific event, their mind should be there again. Remember, generic and hypothetical questions will just get you hypothetical answers.
Conclusion
So what did we learn now? Qualitative experiments are there to understand the target group, their needs and the situation they are in. Quantitative experiments come in handy whenever you try to verify or falsify certain aspects of your intended solution.
When it comes to experimentation, think to yourself: If we run the experiment like that and get the expected outcome, are we happy with it? Is that enough to confidentially proceed? If not, change the experiment.
What are your thoughts on this topic? We would love to hear from you!
