Isn’t it about time to take UX Research more seriously?

November 8, 2012 4:26 pm

Isn’t it about time to take UX Research more seriously?

User Research HighwayAs a business we are always challenging the UX space. UX seems to be the trendy thing that everyone wants to do, everybody says they do, but no one quite agrees on what it is. We presented this thought piece at the NUXONE conference in Bradford on 27th October 2012, and we’d like to share this in the hope that more professionals within this space will take up the challenge.

This article can be downloaded by going to Isn’t it about time to take UX seriously

How respected is UX research?

In October 2012 the MRS published a review that stated that the UK professional research and evidence market is worth up to £3billion. The purpose of this study was to identify the size, scale, market value and impact of the ‘evidence’ market.

As well as traditional market research companies, seven other sectors were identified as contributing to the total market worth. It’s always been hard to put a figure on the size of the UX research market, especially within the UK. The MRS review talks about the contribution of ‘traditional’ and ‘non-traditional’ working in the ‘evidence market’. It sparks our debate of who is conducting UX research, how it is being done, and how to make this a respected part of the ‘evidence market’.

“It concludes that traditional definitions of ‘market research’ no longer hold true, with many more organisations and businesses undertaking research activities than had been realised.”

Guidelines and standards

There are plenty of professional standards and guidelines out there that the UX research professional can adhere to. But when we hear about the wide range of methodologies that are taking place in user research, we do wonder how many people have read the MRS or ESOMAR guidelines.

“Guidelines are not binding. Their aim is to promote professionalism in the conduct of research.”

Excuses, excuses…

There are plenty of reasons out there for different parties involved in projects to push back on user research:

• Cost

• Time

• Internal buy-in

We’re not interested in a race to the bottom, and that’s where reducing costs with cheap options ends up. There’s a cost to failure and we want the research that we propose and carry out to make a difference. We help clients save development time by getting involved as soon as possible, improve conversion and the user journey, bringing teams together to see things from the user’s perspective. Good UX research is incredibly good value for money.

Projects push and pull and those involved will always be nervous that the user research involvement will stall and interfere with the project delivery. As a user research company, we have to be flexible with our involvement and tailor our services to suit. Adapting the research deliverables and the methodology allows the research to run concurrently with other project streams.

There’s plenty of help out there on how to demonstrate the benefit of user research. From the UXPA’s website to internal case studies, there’s always an argument in favour. We try to focus our efforts on those that understand the benefits of UX. Non-believers sap a lot of energy and time.

80% of products fail in the first year of release (more like 85%)

Burnt believers

There are stakeholders that have been burned by previous bad experience of research poorly conducted.

• Poor methodology

• Poor execution of methodology

• Poor recruitment

This is where the biggest opportunity lies. After well thought out and well conducted user research we have seen many a client converted, even after only observing a few sessions.


In the following section we focus on a few methodologies where there are things to look out for. This is not a full overview of all UX research methodology, but a few examples.

Less focus on the focus groups

The concept of ‘Group think’ needs to be guarded against. A lot of emphasis is placed on the outputs from a group, and this can be misplaced. The idea of a focus group is that the discussion has a cascading and chaining effect. But this often means that the more introverted members of the group don’t have a say.

We are often approached to provide insight from real user sessions as focus group output is questioned by clients. If clients are not bought into the process, then they won’t be bought into the outcomes.

When is a focus group not a focus group? There can be misapplication of the focus group setting by using it for usability testing and web development rather than exploration.

One aspect of user research that we are passionate about is the matching and preparation of the methodology. We are constantly questioning and planning in order to match the client’s objectives to how we are going to conduct the research. Some traditional market researchers are adapting focus groups and turning them into group usability testing – often referred to as group surfs. This simply doesn’t emulate a natural environment enough and can lead to false findings.

Pop-up interruptions

Online surveys have a habit of appearing at the wrong time. Whether it’s right at the start when the user hasn’t interacted long enough with the website to give an opinion, or right at a conversion point of interacting with the navigation or about to click on a crucial call to action.

Remotely cheap

Remote user testing is defaulted to as the cheap, cheap, cheap option – but is it really? Along with our lab research we often add remote testing as a way of gathering more statistical significant numbers to a quantitative study. A huge amount of work goes into designing, reviewing and aligning the findings to what happened in the lab. This is time consuming and therefore not always cheap.

With remote testing you lose an element of control. This is often linked to the kind of person who wants to take part in these types of studies. It’s often the budding expert reviewer who wants to impose an opinion rather than naturally react.

The analysis relies on inference and self-reporting without the check of a user sitting beside a moderator.

What would you do for 75p a survey?

Again, it comes down to who and how you have recruited, but also the incentive when dealing with online panels. Ask yourself how much effort and brain power you’d use for something that’s going to be a relatively small gain.

Referring to our previous point of losing control of the study in the hands of the remote user, it can come down to how that person wants to be perceived on that day. Does what they say match into actual behaviour?

Users who do this on a regular basis tend to be quite tech savvy. They get better with practice, eliminating your ability to gather novice usage and initial reaction findings. They are good with browser cookies, can install user cameras and are generally happy with downloading software and allowing you to observe their browsing experience. You have to ask yourself if this is your target audience.


Recruit right, recruit once

With participant recruitment we’re always guarding against serial attendees. These people are on their best behaviour and want to become semi-professional at taking part in research. We’re not interested in this as we often teach people a lot about how they think. They also learn the kind of things that you are interested in, and the next time will set out to please you with what they think are the ‘right’ answers.

Don’t keep it in the family

Screening for participants should not include friend, family and colleagues. There is a familiarity that is hard to eliminate and a fear of offence that will lead to bug false findings.

In market research there are strong guidelines regarding testing with people that you know.

Testing your own work is hard. It’s difficult to remain objective and therefore produces user research outcomes that can be seen as trusted evidence.

MRS Code of Conduct: Qualitative exercises – B.40

“The issue of anonymity and recognition is a particular problem in business and employee research. If guarantees cannot be given then members must ensure that observers are fully introduced before the group/interview begins and respondents given a chance to withdraw.”

“Employee research has all the problems of conducting research in a small universe. Sample sizes in specialised areas may be very small to the point where employees themselves could be identified”.

SimpleUsability comment – when testing internally, colleagues may be worried about giving feedback to the team in case of offence.

Wrapping up

If you find yourself saying the following phrases:

• I don’t need to do research I know what my users need

• I’ll sketch up a prototype and then prove it’s better by testing

Stop and think. What effect are these thoughts on your project and what effect will this have on UX research being taking seriously?

We’ve written this piece with the aim of challenging how UX research is being carried out, rather than being detrimental about what others do. We ask you to keep an open mind when planning your user research. Research is about learning. Where in the past some user research was better than nothing, not having a go just isn’t good enough.