Some time ago we were asked to test the NaviExpert application. The research was exploratory in nature. Its aim was to check how users evaluate the application and how they use its most important functions. Why did we leave the laboratory and rent a car for this purpose?
We could conduct a usability test of the application in our laboratory. It would have been possible to talk about experiences, record errors, and point out incomprehensible and problematic elements. Theoretically, the test could be conducted in such a way that messages, audio instructions, and route suggestions would appear in the application. The result would be a detailed research report with recommendations and comments from the respondents.
Sometimes it is better to leave the lab
However, such an approach would not be complete. After all, phone navigation is rarely used at home, sitting on the couch, except to check the distance between two addresses or cities.
What better way to test the user’s reactions to the navigation method, the sound, the commands, the map, the changing day-night mode, if not during actual driving? Instead of showing subjects the two screen modes, we simply scheduled the test at sunset. This allowed us to observe the reaction of the test subject as the screen changed from lighter to darker while driving.
We arranged the tests in such a way as to be able to test the various functions of the application in different road and lighting conditions and in different traffic volumes. The tests took place between 6 a.m. and 9 p.m. Each respondent had to drive the same route, which was selected so that in the event of traffic jams the application would suggest at least two other options of how to get there. We, therefore, tested additional functionalities.
Advantages of exploratory freedom in NaviExpert
Respondents also had the opportunity to freely explore the NaviExpert application. During the study, we followed the respondents, asking additional questions about the elements we were interested in. The scenario assumed the examination of specific functionalities, but respondents had the opportunity to go through the tool on their own and look into the tabs that interested them. This allowed us to check which places the respondents enter, how they navigate the application, which elements intrigue them, and which may be incomprehensible.
The greatest reward during the test is the moment when the test subject forgets that they are in the middle of a usability test, that there is a camera in the back, and that the researcher is sitting next to them. This is the moment when the dance of gestures begins: the respondent discusses with the app, pulls out his phone, and starts comparing functions. The respondent’s spontaneous reactions are what we are waiting for. This is when the respondent leaves the level of declarativeness and shows how he really uses the device. He gets involved.
Research is not just an interview in the form of a question and answer. It is an in-depth observation of the respondent’s behaviour, their emotions, facial grimaces, shrugging shoulders, every sigh or pat on the phone. Conducting research in a natural environment allowed us to observe behaviours that we would not be able to capture in a research laboratory. We would not have paid attention to the way the user adjusts settings, tests and changes the sound, positions the phone in the holder. The moment when the test subject, while driving, asks to stop to change some of the settings, like turning off the points of view or changing the voiceover, would not have occurred. The in-car test made it possible to record surprising, unpredictable situations that would not have occurred under laboratory conditions.
Respondents were involved in the test of the application all the more as they installed it on their phones and received free access to it as part of their remuneration. Some respondents had so much to say, and were so involved in discussions about the tool, that on their own initiative the conversation stretched to two or even two and a half hours. This was twice as long as we had originally anticipated.
Difficulties on the field? There are, but they can be overcome
During an out-of-lab study, many external factors can occur that are beyond our control and can interfere with the study. This was the case at the recruitment stage, for example.
Respondents had to have a driving licence and could not be afraid to drive a rental car. Each respondent was enrolled for insurance, so sensitive data such as ID card number, and residential address were required during recruitment. Not everyone agreed to this, therefore opting out of the study.
Some respondents took a few moments to get used to driving a new car. A round in the car park, braking and starting tests became a normal warm-up part of the test. There was also a problem with the recording of the test. Every time we drove the car with NaviExpert we switched on the camera and voice recorder. After the initial recordings, we found that the camera lasted one and a half to two hours on a single battery. However, as the research dragged on, we had to change the battery during the interview. Consequently, we changed to a more stable way of recording – we connected the camera to a laptop.
Traffic jams and obstructions were factors beyond our control. On the one hand, they made it possible to test the function of avoiding them, but on the other, they prolonged the test time.
The study in natural conditions yielded exceptionally rich material in terms of quantity and quality. It allowed for very detailed and thorough testing of the application. We got to know the users well, their behaviour, reactions, and opinions. The best effect was the possibility to present the client with recordings of sincere and spontaneous reactions of the respondents. Sometimes one 30-second recording has more persuasive power than 50 pages of the report.
Need custom research? Let’s talk!
