‘Myths about usability testing’ UXSheffield talk by Rolf Molich

October 7, 2016 9:12 am

‘Myths about usability testing’ UXSheffield talk by Rolf Molich



On Wednesday 28th September, UX Sheffield was hosted at The Electric Works, and featured pioneer of usability Rolf Molich. Rolf, who has worked in the field since 1984, came to discuss ‘Myths about usability testing’.

‘5 users will find 85% of usability problems’

Rolf began the talk by discussing the myth, made famous by Jakob Nielson, that ‘5 users will find 85% of usability problems’. Rolf went on to explain how it is impossible to say you have found ALL usability problems, as some usability problems will be specific to certain types of users, and therefore are unlikely to be spotted with a small sample of users. Rolf suggested instead, rather, that ‘5 users are enough to drive a useful iterative cycle’, where the key usability findings will be discovered, as they will be applicable to the majority of your users.

Rolf then went on to discuss the importance of usability testing on the correct users. While asking close friends and family can be a relatively cheap way of receiving feedback on a product, it most likely does not target your audience. Rolf provided the quote, ‘I know what my users need, I asked my sister’ – challenging the validity of asking the opinion of friends and family and to illustrate the importance of usability testing. He described how evidence based testing can help demonstrate problems in a convincing way, to encourage stakeholder involvement.

‘25% of comments in a usability report should be positive’

Rolf supplemented his discussion with an interactive online quiz about usability myths, which he encouraged the audience at UX Sheffield to partake in. One of those questions was: ‘Should at least 25% of comments in a report be positive’?

While this raised some questions amongst the audience – particularly from those who felt that sometimes it is not possible to say that many positive comments about a product – Rolf insisted that retaining positive comments within a report has two key benefits.

  • A spoonful of sugar helps the medicine go down – meaning that if 75% of the report is going to contain suggestions for things the product can do better, then providing some positive comments will make the client feel better about their product
  • If it isn’t broken, don’t fix it – this helps point out to clients what already works well with their product, which prevents them from going away and changing something that works well

‘Anybody can do usability testing’

Rolf then touched on usability research, and the qualities of a good moderator. Rolf suggested that the essence of a usability test is for the moderator to speak as little as possible, so the session clearly shows what the user can accomplish when using the product on their own. Hence, ‘Show me, don’t tell me’ is the key to a successful usability test.

During his interactive segment Rolf also included the statement ‘Anyone can do usability testing’, asking the audience to respond whether they agreed or disagreed. Rolf stressed that while yes, anyone can do usability testing, not anyone can do quality usability testing, and that quality is the key when it comes to a good usability practitioner.

‘UX professionals must set a good example, by making usable things’

Rolf then discussed what makes a good usability report. While he had already explained his stance on 25% of the report being positive, Rolf asked the audience: ‘If 60 serious problems are found in a usability test, should all 60 problems be included in the report?’. While it seemed the majority of the room agreed that if problems were found during testing, then the client needed to know, Rolf surprised the audience by saying that in fact, no report should extend beyond 25 pages.

Rolf explained that by including all 60 findings the report would become unusable, and referred back to the basic belief of UX professionals: ‘UX professionals must set a good example, by making usable things’. Rolf went on to further explain that while it may be tempting to include all of these findings, it can have a negative effect and overwhelm the developers.

This can lead to the research being shelved by the developers, as something which isn’t too important, or something which will take too much time to fix. Rolf suggests that by only including a small number of findings in a report, this helps clients focus on what they need to fix of most importance.

Our approach

While cherry picking a certain number of findings and defining how long a report should be is interesting, our approach is to provide our clients with deliverables that match the project needs. Our natural methodology including retrospective think aloud provides us with a high word count and a wealth of user needs and usability findings. Our preference is to provide our clients with the complete picture, keeping all the findings in context without assuming the importance of the findings to the client’s business. Rather than shortening the report we prioritise and identify key findings in an executive summary with recommendations, with the full findings report providing a reference and work with clients to rate and implement the findings.