What do you associate the word experiment with? With a risky activity? Or a scientist in a cold laboratory? We associate it with gaining knowledge! And this knowledge is very useful, especially in developing business ideas and products. Let me explain why.
Hypothesis, experiment, and other boring words
It is not very often that we have the opportunity to implement classical experiments. However, we want to show that the basic principles of the experimental approach are widely applicable and work well for testing business ideas and products.
Experiment (lat. experimentum – experience, study) – as Wikipedia states – in natural and social sciences a set of activities inducing in material objects certain reactions and phenomena under conditions allowing to control all relevant factors, which are subject of precise observation.
The good news is that there are no experiments that go wrong as long as they are properly conducted. But there are certainly some that lead to the conclusion that we were wrong. In practice, such experiments are the majority! So every experiment is valuable because it brings new knowledge.
An experiment is an activity in which we have taken care to:
- a clearly stated QUESTION (hypothesis) that we want to answer,
- a TEST that will answer that question and be conducted under established CONDITIONS,
- to determine the CRITERIA for evaluating the test results,
- draw CONCLUSIONS – answer the question posed earlier.
What can be an experiment?
In the UX area, an experiment is usually associated with A/B tests. And very correctly, because A/B testing is a flesh-and-blood experiment, usually conducted on a living organism. It is based on quantitative data, conducted under controlled conditions, and the conclusions are drawn in accordance with the principles of statistics.
In the Design Thinking approach, however, we can talk about experimentation in a wider meaning and treat it as all kinds of activities that on the basis of data – both quantitative and qualitative – allow or help to verify the assumptions of the designer. The origin and nature of this data can be different: it can be previously available or collected by us, come from users, or be collected passively, based on observations of user actions, online and offline. The most important thing is that they are collected consciously and serve to answer specific questions, not just satisfy the curiosity of the researcher or designer.
Therefore, one can boldly turn to other sources of data than the classic experiment, sticking to the basic principles of the experimental approach:
- have clearly defined QUESTIONS,
- to plan a TEST that takes place under controlled CONDITIONS, e.g. we deliberately selected respondents in a qualitative study and we know what the consequences are,
- analyze the collected data according to predetermined CRITERIA,
- draw CONCLUSIONS and assess their reliability.
Need some support in developing your innovation?
What to what in an experiment?
When choosing testing methods, it is good to follow the rules suggested by Strategyzer specialists: in the first stages choose methods that are cheap and fast, only then invest in expensive and time-consuming ones.
If you are still in doubt about which method to choose, go for the one that provides stronger evidence (e.g., is based on hard, reliable data; on behaviours rather than statements; on information grounded in a real-world context).
If you don’t have access to methods that provide strong evidence, combine different ones to cross-validate them and reduce the risk of error.
How What to test?
When we talk about HOW to test, we often focus on specific testing methods. However, the method is often a consequence of WHAT we can test at a given stage. How to test when we do not have a finished product yet? What could be the IMPACT that we give the user in order to test their reaction?
CONCEPT – don’t even have a prototype yet, but have an idea? No problem, consider if you can prepare:
- A flyer or brochure describing your product
some key information about how the product will work, what benefits it will give. - Landing page
if you have a digital product, a good solution could be a landing page presenting the product, giving the possibility to sign up for early access. - Product card
if your product is very specialised, you may need a breakdown of technical data (even hypothetical) so users can easily review it and compare it to what they already know. - Storyboard
A pictorial story will be useful when you want to show the operation of some service, especially stretched in time. The story sets the product in the context of a specific situation, so it stimulates the imagination better than the description alone. - A video showing how a product/service works
We live in the era of video, so the video is becoming a standard way of telling about products. It can be useful if you manage to convey the most important functions and benefits in a short form.
PROTOTYPE – product still not ready, but already want to see how it performs? Consider:
- A paper prototype
You have an idea for a digital product and you want to test the interaction between it and the user? You want the user to focus on the main functions and not on the appearance? A paper prototype could be the solution.
- Wizard of oz
This is an idea testing technique, which assumes that we do not have a working product yet, but we “pretend” to have it, while the desired effect is achieved “manually”. (the classic example with OCR). - Functional prototype
This usually means that your concept is already quite well thought out. Although it lacks color yet, it can already be tested. - Clickable prototype
Nowadays, many tools allow you to design clickable prototypes right away. This is good because it allows you to test a solution that is already close to the final one. - Looks-like prototype
is a kind of mockup, which is supposed to show the shape, dimensions, and aesthetic values of the designed solution, but it does not work yet.
- Prototype works-like
A model that shows the operation of the designed solution, but may look very different from the final result.
ACTIVE PRODUCT – another version of the same site/product/service
- Feature bud
Putting a feature on the menu that isn’t there yet will measure interest in it.
Okay, you already have a feature bud? Then consider how you will explore it by:
- OBSERVING: what and how can you observe without asking your users so as not to interfere with the process? What data can you collect?
- QUESTIONING: what can you ask users during or after testing the concept or prototype to better understand their reactions?
1000 and 1 experiments to go
Innovation is a process of constantly being wrong and learning from mistakes. While this sounds like a good tweet, it can be difficult in practice because. no one likes to be wrong. Most organisations want innovation badly, but at the same time don’t want to abandon the approach of releasing “near perfect” products into the world in favour of trying out imperfect things.
Meanwhile, the most successful organisations in today’s world are often the ones that are not afraid to experiment. Did you know that at Booking.com there are about 1,000 experiments going on at any given time? 9 out of 10 of them will turn out to be a failure. That’s nothing, because 100 will turn out pretty good, and one can turn out fantastic and move the business forward. The key to innovation is creating a culture of large-scale experimentation in companies (e.g. by building Living Labs). And a culture of allowing for controlled error, i.e. error from which we draw conclusions. I would like to say at the end: good luck with being wrong!
