How to ask why in usability testing

October 12, 2018 2:46 pm

How to ask why in usability testing



Similar to many children growing up, I always wanted to know why things were the way they were. This hasn’t rectified itself into adult life, therefore it’s ideal that I find myself working as a user experience consultant for SimpleUsability, where I can ask ‘Why?’ all day long.

But in the field of usability, constantly asking a research participant “why?” could become pretty annoying. Moderators of usability studies have to discover different methods to find out the reasons behind why somebody carries out a task in a certain way.

Maybe we should address another question first; “Why ask why?” Technology has provided us with many tools to find out what people are doing, particularly on websites. From analytics, we can see what people are clicking on and even watch where attention is driven to. What this doesn’t tell us is why people clicked there, what else they looked at first but were confused about, and also what they missed.

The fundamental challenge in website usability testing is to find out why in a natural way so that false findings are not documented. Traditional usability research involves a methodology called concurrent think aloud. This means that while the participant is carrying out a task, the moderator has to carefully coach the person to tell them what they are thinking as they go along. The benefit of this is that you get instant feedback, but a problem arises in the fact that they are being distracted from the task in hand. This leads to the user becoming stressed and forces a response without justification.

People generally don’t know why they do things, so moderators need to develop methods and techniques to find out what the client needs to know from the usability session. Here’s our top ten tips on how we do it:

  1. Leave the user alone during tasks

Discussions should take place after the user has finished their tasks. This requires the moderator to observe carefully and make notes (and also bite their tongue!).

  1. Replay

Within a discussion, provide users with evidence of what they actually did, so that you can guard against people speculating and creating opinions that are false. We use a method called ‘Retrospective Recall’, where we play back to them a recording of what they did with their eye-tracking and this acts as a visual cue for them to verbalise what they were thinking about at the time of the task.

  1. Open-ended questions

This is a friendlier, quicker and easier way of gathering feedback from the participant about tasks completed in a usability session than closed questions. The questions asked by the moderator require more of an answer than a single word such as yes or no. If the answer given is not sufficient a further question can be added that is less open, e.g.:

“Can you tell me about what you were thinking at this stage in the process?”

“This stage was all about finding out about X, how did you get on doing that?”

  1. When is a question not a question?

A question doesn’t have to be explicit but can be a statement that implies that you require a response. This is a very good way to ask a question that allows users to respond openly. You can do this by tailing off at the end of a sentence or changing the pitch of your voice so the user infers that you would like an answer, e.g.:

“You were doing this because…”

  1. Leave space for them to answer

Researchers are drawn to the field because they are interested in people so it’s not surprising that we love to talk with users in research sessions.  The risk with this is that we start to make it a conversation and fill in any silences instead of allowing space for the user to tell us about their experience. So when you ask a question, ask it and let it stand. Leave some space for a user to gather their thoughts and reply. And then, just leave a little more silence after they reply. This is a good way to encourage them to tell you a little more without the need for you to ask another question.

  1. Act like you’re not in the loop

As researchers, we let users know we are a neutral party. This means we are on the side of the user and allows the user to open up and tell you exactly what they think without worrying about offending anybody. You may be completely aware of the functionality of a feature or where users need to click to move forward towards what they are trying to achieve. Making it look to the user that you are no more aware than them, allows you to see their actual behaviour and therefore find out about their understanding and expectations of what they can see. For example, if you know that a button should allow them to access the next step, but they are not sure and ask for help, you could ask:

“I’m not sure what this button does, do you want to try it and see where it would take you?”

  1. Watch your language (and your body language)

Don’t give anything away in the wording of your questions. People pick up on language that you use within questions and often repeat it back to you without realising. You are interested in hearing their language rather than hearing your own words again. It can also be to imply a thought or opinion within a question without realising it. It’s all about subtlety in your choice of words so that the user doesn’t infer what you want to hear.

  1. Users who don’t want to talk

One-to-one usability sessions give shy participants a voice because they are not overshadowed by other members of the session, but there may be some users who just don’t like to talk and may struggle to open up. This user needs more help understanding where you are going with the questions and you may have to start with more targeted questions and then open up more as the user becomes more confident, e.g.:

“We’re really interested in how people get on finding a section.”

“How did you get on with getting started from this point?”

  1. Users who want to talk

Some users just want to please you and talk a lot but don’t cover the things that you are trying to find out about. If users have gone off target the challenge is to bring them back to the subject politely. Try obtaining a more specific answer by referring to an element of what you are testing and asking a more closed question, and then follow with a more open question, e.g.:

“Did you find this option useful?”

“Why was that?”

  1. Listen and respond

From observing the usability tasks, you may have a list of points or questions that you are desperate to know about. It may seem obvious but it’s important to listen to the person’s response to the first question before moving on too quickly. There may be a point that they raise that requires more clarification or can lead smoothly into the next question.