How to ask why in usability testing

October 15, 2010 11:50 am

How to ask why in usability testing



Similar to many children growing up, I always wanted to know why things were the way they were. This hasn’t rectified itself into adult life, therefore it’s ideal that I find myself working as a user experience consultant for SimpleUsability, where I can ask ‘Why?’ all day long.

But in the field of usability, constantly asking a research participant “why?” could become pretty annoying. Moderators of usability studies have to discover different methods to find out the reasons behind why somebody carries out a task in a certain way.

Maybe we should address another question first; “Why ask why?” Technology has provided us with many tools to find out what people are doing, particularly on websites. From analytics we can see what people are clicking on and even watch where attention is driven to. What this doesn’t tell us is why people clicked there, what else they looked at first but were confused about, and also what they missed.

The fundamental challenge in website usability testing is to find out why in a natural way so that false findings are not documented. Traditional usability research involves a methodology called concurrent think aloud. This means that while the participant is carrying out a task, the moderator has to carefully coach the person to tell them what they are thinking as they go along. The benefit of this is that you get instant feedback, but the problem arises in the fact that they are being distracted from the task in hand. This leads to the user becoming stressed and forces a response without justification.

People generally don’t know why they do things, so moderators need to develop methods and techniques to find out what the client needs to know from the usability session. Here are a few tips on how we do it:

1. Leave the user alone during tasks

Discussions should take place after the user has finished their tasks. This requires the moderator to observe carefully and make notes (and also bite their tongue!).

2. Replay

Within a discussion, provide users with evidence of what they actually did, so that you can guard against people speculating and creating opinions that are false. We use a method called ‘Retrospective Recall‘, where we play the user’s eye tracking back to them and this acts as a visual cue for them to verbalise what they were thinking about at the time of the task.

3. Open ended questions

This is a friendlier, quicker and easier way of gathering feedback from the participant about tasks completed in a usability session. The questions asked by the moderator require more of an answer than a single word such as yes or no. If the answer given is not sufficient a further question can be added that is less open, e.g.:

“Can you tell me about what you were thinking at this stage in the process?”

“This stage was all about finding out about X, how did you get on doing that?”

4. When is a question not a question?

A question doesn’t have to be an implicit question but can be a statement that implies that you require a response. This is a very good way to ask a question without leading. People pick up on language that you use within questions and often repeat it back to you without realising. You are interested in hearing their language rather than hearing your own words again. Tailing off at the end of a sentence and changing the pitch of your voice can infer that you would like an answer, e.g.:

“You were doing this because…”

5. Act like you’re not in the loop.

Being the neutral party puts you on the side of the user and allows the user to open up and tell you exactly what they think without offending anybody. Although you may be completely aware about the outcome of a feature on a product, making it look to the user that you are no more aware of what should happen then them can let you find out what the user’s understanding of an issue may be. For example, if you know that a button should allow them to access a particular part of a website you could ask:

“I’m not sure what this button does, where do you think it would take you?”

6. Watch your language (and your body language)

Don’t give anything away in the wording of your questions. It’s sometimes easy to imply a thought or opinion within a question without realising it. It’s all about subtlety in your questioning so that the user doesn’t become defensive.

7. Users who don’t want to talk

-One to one usability sessions give shy participants a voice because they are not overshadowed by other members of the session, but there may be some users who just don’t like to talk and may struggle to open up. This user needs more help understanding where you are going with the questions and you may have start with more targeted questions and then open up more as the user becomes more confident, e.g.:

“We’re really interested in how people get on finding a section.”

“How did you get on with getting started from this point?”

8. Users who want to talk

Some users just want to please you and talk a lot but don’t cover the things that you are trying to find out about. If users have gone off target the challenge is to bring them back to the subject politely. Try obtaining a more specific answer by referring to an element of what you are testing and asking a more closed question, and then follow with a more open question, e.g.:

“Did you find this option useful?”

“Why was that?”

9. Listen and react

From observing the usability tasks, you may have a list of points or questions that you are desperate to know about. It may seem obvious but it’s important to listen to the person’s response to the first question before moving on too quickly. There may be a point that they raise that requires more clarification or can lead smoothly into the next question.

Like to know more? Come along to one of our training courses and learn more about moderating usability tests.

Want to read more research articles?

All of our research thought pieces are published as printable PDF documents here.