The challenges of testing prototypes with users

February 9, 2017 11:17 am

The challenges of testing prototypes with users

Following on from our tips on building prototypes for testing, here we share some of the challenges which can come up while testing a prototype and how best to work around them.

Conducting usability testing can be challenging, especially when testing prototypes. This is because prototypes are not fully functional websites or applications and the level of detail within prototypes can range from simple paper prototypes to high-fidelity pre-launch prototypes. This article will discuss the challenges we face when testing prototypes from the point of view of a UX researcher and it will provide some tips that can make prototype testing more successful.

What is a prototype and why do we use them?

A prototype is an early sample, model, or release of a product built to test a concept or process or to act as a thing to be replicated or learned from. For UX researchers, they are used to evaluate new website journeys and designs and allow the product team to detect usability problems before implementing them into the live site or app.

What are the challenges when testing prototypes?

1. Visual design

Users can be distracted by the lack of visual design on a prototype because wireframes and other lo-fidelity prototypes are very basic. This can cause users to comment on the lack of design and colour and distract both themselves and the researcher from the true goals of the project. The extent of this challenge depends on the level of detail within the prototype.

How to get around this: Ensure the user is aware at the start of a session that the website they are about to view is at an early stage of development and so does not look and feel like they may expect. The research may need to be explicit with some users and point out it is not the visual design that we are interested in for today.

2. Partial journeys

Prototypes often cover only partial user journeys, meaning that users may have to be dropped into a journey at a specific point and may lose the context of the overall task or what they would be coming on the site to do.

How to get around this: As well as creating tasks which set the context, consider including some time at the beginning of the session for users to explore the prototype as they would normally do on that website/app, without giving them long enough to discover the prototype journeys. Introductory questions can also be asked at the start of the session to position the user in the right frame of mind for what the prototype will allow them to do, therefore helping to provide some context alongside the task wording.

3. Security concerns

Sometimes users may be required to enter personal details or bank details during the testing session. As we know users consider security to be important, and prototypes tend to lack a lot of visual design and thus may appear less safe than actual websites. This may cause users to become nervous when entering personal information or refuse to enter any at all.

How to get around this: Ensure users are aware that the website is safe and any personal details they enter will not be stored, saved or used to contact them. Of course, we can always be prepared with dummy details if they do not feel comfortable entering their own but this might not produce the same results as if they had used their own details.

4. Click happy

Users may become frustrated when they tap on parts of the prototype which are unclickable and so become ‘Click happy’. Instead of looking for the way forward in the journey, they start to look for any option which will allow them to move forward, and we lose the chance to observe their natural behaviour for the journey.

How to get around this: Explain to the user when something isn’t working. For example, the researcher should try to reassure the user so they are aware that their behaviour is still helpful: ‘I’m sorry that’s not working for us today but I’ll make a note you’d want to click there’. Similarly, when introducing the session to the user, ensure the user is aware that the prototype is in development and that they may have to be patient with it to avoid them becoming click happy.

5. Time

Testing on prototypes can take more time than testing the same journey on a live website. As well as the challenges noted above, there can be technical challenges such as those we discussed in the earlier article.

How to get around this: Pilot the study and test how long it takes for pages to load and how long it takes for someone external to the project to get through the journeys. Factor this into the session time. There is nothing worse than having problems with the prototype and so running out of time in the session.

The takeaway

We’ve seen there can be challenges when testing with prototype. These can be frustrating for UX researchers and observers from the product team who want to see how a new journey or design works. It can be easy to become so focused on the prototype journeys we forget we can also learn about user expectations, language and understanding which can feedback into our journey design. So prepare for the challenges, observe the sessions, take notes, be patient and be prepared to make changes. Keeping these potential issues in mind will help you both define your usability tests and understand the results better.