UX research – are we ready for the Wizard of Oz research methodology?

May 27, 2021 9:29 am

UX research – are we ready for the Wizard of Oz research methodology?



As prototypes become more expensive to produce but project deadlines show no signs of slowing down, are businesses missing the point of what good user experience (UX) research is fundamentally about?

Guy Redwood, UX and Research Advisor at SimpleUsability – part of CDS – delves into why organisations must now get back to basics to best serve users’ needs, and uncovers whether or not Wizard of Oz testing could help organisations successfully venture down their own ‘yellow brick road’.

Recently it’s felt like the discovery phase of research has been slowly sliding into the realms of proof-of-concept testing. With the pressure on to deliver products quickly and UX design shifting their focus towards ‘form over function’, we’re seeing that businesses – often through no fault of their own – are rushing through projects in order to complete them rather than taking the time to unpick every element of a customer’s online journey with their brand.

So, we decided to put this into practice using the Wizard of Oz model of testing a system that doesn’t yet exist. For many people who might not be aware of this type of research, it might seem like they’ve stepped into the L. Frank Baum children’s novel, but we knew this could unlock insights.

The research approach

Utilising the Wizard of Oz UX approach, we set up discovery phase projects on how customers would feel about using their banking facilities through the globally recognised AI virtual assistant, Amazon Alexa.

The same technical set up was then used to explore complex journeys that moved between website, SMS and an Interactive Voice Response system.

Following these early tests, we discovered that users who moved through tailored journeys – with a researcher responding to each of their online decisions – made for a more holistic experience. Exploring this way to interact, inform and analyse the emotion of the customer, with minimal technical effort, was hugely beneficial to all concerned.

Where the Oz technique really came into its own happened when we evolved our systems to record and monitor all aspects of the session – including the participant’s phone screen and allowed the facilitator to listen in on the call through headphones, as well as communicating separately via chat with their research assistant in real-time.

When is Oz relevant?

We found that Oz was easiest to deploy in the early phases of the project – especially when quickly prototyping ideas and insights into screens, designs or systems that can be changed ‘on the fly’ and placed in the hands and ears of the users themselves.

Additionally, Oz allows you to swiftly understand how a customer felt about your product after using it – there was nothing hypothetical about it. And it’s that type of real reaction that means organisations can easily explore variations with different people, and with minimal technical investment.

Moving between channels paved the way for a more joined-up journey and reduced the need for a collection of teams to create stimulus for the research. However, for businesses looking into this type of model, it’s worth noting that you’ll need to engage with more stakeholders to ensure they’re aware that you’re respectfully including them in the process.

What are the barriers to Oz?

Prototyping tools appear to have a binary life – they’re used to either learn about engagement or validate design. And with that, some are better than others.

With Oz, you have to learn fast, fail forward and innovate in real-time to ensure you’re truly getting a more complete picture. For businesses struggling to quickly create journeys in their research, it’s worth looking into where you might need a refresh and if this type of model is for you.

Are there any tech challenges with Oz UX research?

With anything, there are always going to be obstacles and difficulties to overcome. We learnt a few lessons, including:

  • Always use your own equipment with users. You need full control of the environment with prototypes to ensure the bulk of your time is spent running the session and not wrestling with setting up participant’s phones.
  • It’s best to place the mixing desk with the wizard to ensure you can control the sound for all inputs and outputs – and don’t forget to label the dials, or draw diagrams to show where all audio signals are going.
  • Provide the facilitator with a live feed from the mixing desk to a pair of headphones so they can hear what the participant is listening to.
  • The delivery of SMS is a real problem. We found it could take up to give minutes for a message we’d sent to arrive on a person’s phone.
  • And finally, not all cables are equal in terms of specifications – so Amazon reviews are your friend here!

In conclusion…

There will be many organisations that aren’t ready for adopting the research methodology of Oz. However, it does shine a light on how important it is to never forget the essence of UX research. If we continue to agree to delaying user research because the wireframes or designs aren’t ready, maybe it’s worth pushing for a different relationship with the things we’re putting in front of users. And if we’re to run research outside of silos and making whole organisations user-centred, then we must make research more accessible and part of our day-to-day responsibilities.