Wizard of Oz testing – a method of testing a system that does not yet exist.

August 30, 2018 4:13 pm

Wizard of Oz testing – a method of testing a system that does not yet exist.



You’re probably thinking, what does Toto have to do with user research? This method of research gets its name from the story by L.F. Baum. In the story, the Wizard fools everyone by creating a vision of himself to look powerful by using a set of controls while he hides behind a curtain concealing the reality.

What is Wizard of Oz testing?

The Wizard of Oz method is a process that allows a user to interact with an interface without knowing that the responses are being generated by a human rather than a computer by having someone behind-the-scenes who is pulling the levers and flipping the switches.

Wizard of Oz testing – The listening typewriter IBM 1984

The Wizard of Oz method allows researchers to test a concept by having one practitioner – the ‘Moderator’ – leading the session face to face with each user, whilst another practitioner – the ‘Wizard’ – controls the responses sent to the user via the chosen device. In the image above from IBM, we see an example for testing the concept of a ‘Listening typewriter’. The user is sitting in one room talking into a microphone while the ‘Wizard’ sits behind the scenes typing what the user is saying so it appears on the user’s screen as if it was done by the computer.

Why?

As UX researchers we often remind people to test systems at every stage of development, and that includes testing before development has even begun. This can save time, money and those ever so embarrassing moments when products are launched before they are fit and ready for users.

The Wizard of Oz methodology allows you to test users’ reactions to a system before you even have to think about development. This could be a new concept you are unsure will work for your users or a project that would require a substantial amount of effort to create, but we want to learn more before it makes sense to invest the time and money, and it cannot be tested with the usual prototype tools. Wizard of Oz is a flexible approach that allows concepts to be tested and modified without having to worry about potentially tiresome code changes, breaks in a daily testing schedule or full development costs.

How?

Prototype, prototype, prototype! The easiest way to conduct Wizard of Oz testing is to build a simple and easy-to-use prototype that allows the ‘Wizard’ to quickly react to the user’s gestures or actions with the designed response with a single click.

Similar to any methodology, creating a Wizard-of-Oz prototype starts with us having to determine what we want to test or explore. Then we need to figure out how to fake the functionality needed to give the user a realistic experience from their viewpoint. For example, you could prototype a jukebox without creating the mechanics and use a hidden person to play the selected songs to the customer.

Our experience with the Wizard of Oz methodology

At SimpleUsability we have used the Wizard of Oz method when testing IVR and SMS systems. For IVR we saved development time and money by creating an Axure prototype with integrated voice files to playback to the user once they have selected the option that is most suited for their needs. This allowed our clients to see, in real time, how users got on navigating their IVR systems, and what prompts had a negative effect on the user’s journey. This also allowed us to run a more complex project, trialing different versions of the same prompt within a single project to see which prompts work the best for a specific audience.

We have used the ‘Amazon Polly’ automated voice generator to generate automated voice files that are quick to make and consistent. This speeded up the project process because we didn’t have to wait around for voice files to be recorded by a human. It also saved costs of having voice files professionally recorded before testing showed which versions worked best.

How can we help you?

We have used the Wizard of Oz methodology for clients from a variety of sectors including financial services and telecoms and can tailor the approach to meet varying business requirements.

In the financial services sector, we worked with our client Arrow Global on a broader programme of work to optimise their online portal by providing customers with an additional channel to perform key actions relating to their account.  To further encourage channel shift, we explored the current IVR system and the way in which this could be adapted to appropriately direct customers to the self-service portal.

Our work for EE was focused on exploring various concepts for enabling customers to take control of their accounts; firstly, to understand whether this was something they would like to do and secondly, whether they were able to do it using their IVR system. By using the Wizard of Oz methodology at the concept stage, we were able to save EE significant development costs and time and then help them optimise the flow for customers managing their account over the phone, reducing the need for call center support.