Archive for July 2018
If you’re looking for guidance on how to recruit research participants, the internet is your oyster. But when it comes to understanding whether you should reuse your participants, you’ll find little advice on best practices. However, here at SimpleUsability and Research Helper we have 17-years experience recruiting participants and are happy to share what we’ve learnt. This article will outline best practices for reusing research participants across the contexts of different methodologies and different project needs.
Should we reuse participants?
Whether we should reuse participants is something that often gets asked by clients. “Should we invite the same users in for each round of research so that they can see our product improving” Or, “should we make sure we recruit a fresh set of eyes?”
There are a lot of questions to ask when considering whether to reuse participants or recruit a fresh sample, and the outcome is likely to depend heavily on the methodology you are using or the type of product you are testing. So first let’s have a look at the general principles that apply to re-using research participants:
When you should not reuse your research participants
- If the participant has taken part in a research session within the last 6 months. You want to avoid developing ‘professional participants’ who anticipate the types of problems they think they are expected to find and lose the fresh perspective needed. Both Nielson Norman Group and MRS recommend that participants should not be used more than twice in one year so we find that waiting 6 months before inviting a participant in again prevents them from becoming research masters.
- For iterative tests when the focus is on the ease of use or first exposure to the system. If a user has previously engaged with an earlier version of the site, system, or app, it’s likely they will remember things from the session, so will not approach the new designs with a completely fresh perspective. Avoid reusing participants where this is the case.
- Research on a similar topic. Even if the system itself is different, sometimes users may recall their previous experiences if the research is of a similar topic. For example, if users have previously completed research for home insurance, we would avoid inviting them in for another session relating to any kind of insurance.
- If the participant has previously been excluded from a study. Whether this is during the session itself or when you come to analysis, participants sometimes need to be excluded due to their dishonesty during the recruitment process, limited feedback during the session, or inappropriate responses. It is always important to record which users have had to be excluded, and a good idea to avoid inviting them in again.
- If the participant has previously been unreliable. We’ve all had times when our circumstances change unexpectedly and we have to cancel our plans. However, if a participant makes a habit of cancelling their sessions last minute, you might as well save yourself the hassle and avoid reusing them in future.
When you could consider reusing your research participants
- When carrying out research on an internal system. In certain cases you may be limited to a particular user group. For example, if evaluating an internal system, your user base is restricted to people who work for the company and use the system. In a situation where you can’t physically find alternative users, you will have to go back to the same participants to gain feedback on new designs.
- When your research is targeted towards a restricted target audience. You may want to see how your product addresses the needs of a particular target audience. For example, when testing for accessibility your audience is already restricted, so when coupled with limited time and resource, it may be difficult to avoid reusing participants. However, where possible we would still recommend avoiding anyone who has been recruited within the last six months.
- For studies on the same system that do not focus on ease of learning. If you’re looking to see how a system works over extended use, or how iterative designs can improve it over time then you might want to reuse the same participants. However, it would be best practice to still include some new participants to ensure learnability is not affecting the research findings.
Based on these general principles, our recommendation would be to avoid reusing participants when research is taking place more than every 6 months, or for systems requiring fresh user insight. However, as we’ve highlighted, there are exceptions where reusing participants is appropriate, such as if you are dealing with a restricted user base or wish to carry out a longitudinal study. Ultimately, these guidelines should be considered on a project by project basis in order to decide whether to reuse participants or recruit new participants for each round of research.
I’ve been watching people buy things on Amazon this week. Whatever the sector, when we’re researching users’ needs, Amazon is a great comparison to include in a session to trigger discussions because it is often held up as having some of the best interaction designs to help people find and buy the products they want quickly and efficiently.
But one thing that has come up several times this week is users saying how distracting it is.
> Read more