Behavioural Research Consultancy
Leeds 0113 350 8155       London 020 3393 0651         info@simpleusability.com

Menu
  • Home
  • What we do
    • beInsight
      • bewebsite-iconbeWebsite
      • bemobilebeMobile
      • beadvertisingbeAdvertising
      • beinstorebeInstore
    • Close
  • Our Services
      • Accessibility
      • Retail insight & In-store environments
      • Usability testing
      • Mobile usability testing
      • Usability training
      • User-centred design
      • Eye tracking services
      • Gaming UX Research
      • Neuromarketing
      • Print/pack & advertising testing
    • User-centred design
      We can support you with User Centred Design services throughout the whole project lifecycle.
      • card-sortCard sorting
      • expertExpert review
      • iaInformation Architecture, Prototyping & Wireframing
      • personaPersona development
      • taxonomyTaxonomy
    • Close
  • Why we’re different
      • Methodology
      • Our facilities
      • Participant recruitment
      • Technology
      • The team
      • Environmental Policy
      • Our clients
    • Our Technology
      Cutting edge neuro-marketing technology is our enabler.
      • eraEmotion response analysis with EEG
      • eyeEye tracking
      • gsrGSR
      • glassesMobile eye tracking
    • Close
  • Case Studies
  • BeInspired
      • Articles
      • News
      • White papers
      • Video gallery
      • Sign up to our UX Knowledge email newsletter
    • Close
  • Contact us
      • Directions
      • Get in touch
      • Join Us
      • Market Research Volunteers
    • Close

5 important things to think of when conducting usability testing of voice interaction using voice controlled assistants.

Jan 11

by Natalie Crook

In: Articles, Research Thoughts, Usability

Comments Off on 5 important things to think of when conducting usability testing of voice interaction using voice controlled assistants.

Over the last year there has been a significant increase in the use of voice controlled assistants such as the Amazon Echo or the Google Home, with over 17 million devices estimated to have been purchased over the last three months alone.

As sales of these devices have boosted more companies are starting to develop systems to work with voice interaction and we are seeing an increase in different ways people can use their devices, some of these are new services such as making calls or sending message others are new channels for existing services such as asking the weather, ordering their online groceries or even ordering a taxi.

As researchers, we’re interested in the usability of these devices and how we can develop different ways for testing voice interaction. The key difference between mobile app or website and a voice interaction system is that the user has no visual cues. Users have to rely on the audio response given by the device rather than seeing visual feedback and this is something that must be taken into account when conducting usability sessions with voice interaction.

As this is new technology there is not a lot of advice around how to run a session using voice interaction devices and how this might be different to a usability session of a website or app. So, drawing on our experience of testing IVR and voice controlled assistants, here are our five top tips.

1. Use participants with different levels of experience using voice controlled assistants.

When planning sessions, it is important to think about the participants you will recruit to take part. As this is a new technology, it is important to recruit users that do not yet own voice controlled devices as well as those more experienced. By doing this you can see any issues with the system for beginners while noting down and problems within the user journey with someone who is already familiar with the device and how it works.

2. Introduce the device to users

For beginner users, it’s a good idea to start he session with an introduction to the device nd how they can start talking to it. This will allow the user to become familiar and relaxed and settle them before you move onto testing your app or Skill. This introduction will also be useful for users who have experience of voice controlled assistants, but may not have used the device you have chosen to test on.
Below are three things you could do to introduce the device to the user dependant on the participants level of experience:

3. Consider providing users with secondary device such as a mobile phone in the session.

If you have ever used a voice controlled device, you’ll know that some of the commands needed for the different ‘Skills’ built into the device to work successfully can be quite specific as the example above shows ‘Ask the BBC to play Radio One.’ If you do not say ‘Ask the BBC’ the device is just simply unable to complete the task. In this case, it is unfair to expect the user to think of this on the spot and get it correct. Setting users up to fail can often lead to having a negative impact of research sessions. A number of Skills have an onboarding experience that uses both the voice controlled device in conjunction with their smart phone. This process helps users learn the language of the skill. So, to minimise negative impact on the session, even if you are testing post-onboarding experience, consider providing users with a smart phone so they can refer to it for basic language.

4. Allow users to have time in the session to use it as they would naturally

This is important as it allows the participants to use the device in their own time as they would if they were at home. As users don’t have any visual indicators when using these devices, it can often take longer to begin with, so it can be useful to provide users with this time to allow them to try different commands. As users do this, you can gain some insight into features they expect to be available and can use this to ask how they had expected these features to work. This also allows you to follow up with questions around how they felt if this command was not available or if it was, what did they think to the way this worked? This gives feedback that can help with further development.

5. Test at all stages of development.

We all know how important it is to do usability testing on throughout the product cycle whether or be a website or mobile app, but this is even more important when it comes to new technologies. As more people are purchasing trying these devices it is essential that the user journey is ready for them. By testing early in the development process you are able to trial out different ideas without the expense of fully developing an idea to find that it does not work for users.

Exploratory

Starting testing at this stage allows you to trial out different ideas without the cost of developing the system. Testing methods at this stage in the process could be:

  • Command prioritisation: Giving users a list of commands and asking them to rank them from high to low as to whether this is something they would or would not use a voice controlled assistant for.
  • Trial commands: Give users a list of commands, this could even be the top prioritised from the previous task and ask them to talk through how they would expect them to work.

Prototyping

Testing at prototype stage allows you to build up a basic system without the cost of full development. Testing methods at this stage could be:

  • Wizard of Oz method: Something we have done before when testing IVR systems, the Wizard of Oz method allows the testing of the commands by having one practitioner – the ‘Moderator’ – leading the session face to face with each user, whilst another practitioner – the ‘Wizard’ – controls the prompts and responses sent to the user via the device.
  • Voice interaction prototype platform: When thinking of creating your prototype there are tools available to help you with this. Sayspring is a tool that allows you to build up a voice interaction prototype that allows you to create commands and phrases personal to your project and allows you to enter the responses that can be played back to the participants.

Final system test

Testing at this stage is more about the whole experience and should include cross platform testing to ensure the user journey from voice controlled assistant to app on mobile or tablet is consistent and easy to follow for users. As these devices are linked to an app that provides assistance when it comes to learning new skills or settings needed to ‘enable’ certain skills it is important testing across this journey is completed.

Takeaway

Plannning usability sessions with voice controlled assistants is similiar to traditional usability testing so you can draw on past experience to plan sessions, and it is different so there are new aspects to consider. Like all usability research, testing should start early in the product lifecycle and continue through, but as it is a new technology you should consider including beginners as well as experienced users, allowing time to introduce the device to users, provide secondary devices to help them become familiar with language and allow a little more time in the session to allow users to complete the tasks. I’m sure as this technology grows, so will the ways in which we can test its usability and we must continue to do this throughout the product cycle to ensure meets user’s expectations and needs.

Tags: Alexa, Amazon Echo, Google Home, Voice controlled assistants

Knowledge Share Newsletter

newsletter banner Sign up to our regular UX knowledge email newsletter.

Recent articles

User testing & Agile – Enemies or Friends? Our Leeds Digital Festival event co-hosted with Bolser

Often thought to be a contentious relationship, Bolser and SimpleUsability came together for a presentation

> Read More
Do It by the Book: How to Use Storytelling in UX Research

When was the last time you bought something online? Did it go as expected? How

> Read More
User experience research vs. market research for product design and development – a blurring line?

You say ‘to-may-to’, I say ‘to-mah-to’ You often hear the terms ‘user research’ and ‘market

> Read More
How to use storyboards effectively in UX projects

‘A picture tells a thousand words’ This is certainly the case when it comes to

> Read More
Homepage fundamentals: Handy tips to entice your users

A homepage is something that any internet user will be familiar with. Whether it be

> Read More

Categories

  • Articles (74)
  • Awards (2)
  • Conversion Rate Optimisation (4)
  • ECommerce (10)
  • Emotion Response Analysis (5)
  • Eye Tracking (8)
  • Gaming (2)
  • News (62)
  • Research Thoughts (28)
  • SimpleUsability events (5)
  • Usability (31)
  • Usability Reviews (22)
  • UX Design (19)

Archive

  • April 2018 (2)
  • March 2018 (2)
  • February 2018 (2)
  • January 2018 (4)
  • December 2017 (1)
  • November 2017 (2)
  • October 2017 (2)
  • September 2017 (2)
  • August 2017 (2)
  • July 2017 (1)
  • June 2017 (3)
  • May 2017 (2)
  • April 2017 (3)
  • March 2017 (2)
  • February 2017 (2)
  • January 2017 (2)
  • December 2016 (1)
  • November 2016 (1)
  • October 2016 (3)
  • September 2016 (4)
  • August 2016 (4)
  • July 2016 (1)
  • May 2016 (1)
  • March 2016 (1)
  • October 2015 (1)
  • September 2015 (1)
  • July 2015 (1)
  • June 2015 (1)
  • May 2015 (1)
  • March 2015 (2)
  • January 2015 (2)
  • November 2014 (4)
  • July 2014 (1)
  • June 2014 (2)
  • May 2014 (2)
  • April 2014 (1)
  • March 2014 (1)
  • February 2014 (2)
  • December 2013 (1)
  • July 2013 (1)
  • May 2013 (1)
  • April 2013 (1)
  • January 2013 (1)
  • November 2012 (3)
  • September 2012 (1)
  • June 2012 (2)
  • May 2012 (1)
  • March 2012 (1)
  • February 2012 (1)
  • January 2012 (2)
  • December 2011 (3)
  • November 2011 (2)
  • October 2011 (1)
  • September 2011 (1)
  • July 2011 (1)
  • May 2011 (2)
  • March 2011 (2)
  • February 2011 (2)
  • January 2011 (4)
  • December 2010 (7)
  • November 2010 (5)
  • October 2010 (5)
  • July 2010 (1)
  • June 2010 (1)
  • May 2010 (4)
  • March 2010 (1)
  • January 2010 (1)
  • September 2009 (1)
  • July 2009 (1)
  • May 2009 (1)
  • March 2009 (1)
  • February 2009 (1)
  • January 2009 (1)
  • March 2008 (1)
  • February 2007 (1)
  • January 2007 (2)
  • November 2006 (1)

Research tags

accessibility Advertising Effectiveness ApplePay argos Big Brother christmas contactless payment Conversion Rate Optimisation CRO Eco ecommerce eeg emotion response analysis Eye Tracking eye tracking glasses gaming insight Internet Retailing iPhone Leeds Market Research Mobile multichannel neuromarketing neuroscience NUX Oculus Rift Online Experience Index outsource Prototyping psychology recruitment research buyer social media twitter twitter brand page usability usability day usability methodology usability testing Usability test of prototypes venturefest Virtual Reality Waitrose X Factor

SimpleUsability

  • Usability Home
  • What we do
  • Our services
  • Why we're different
  • beInspired
  • Contact Us

Sign up for our Knowledge Share Newsletter

We aim to email you once every two weeks with curated content on UX, behavioural research and more.

Our Services

  • Accessibility
  • Retail insight
  • Usability Testing
  • Mobile Usability Testing
  • Usability Training
  • Eye Tracking Services
  • Neuromarketing
  • Games testing
  • Print & Pack Testing
  • Viewing Facilities in Leeds

Latest news

User testing & Agile – Enemies or Friends? Our Leeds Digital Festival event co-hosted with Bolser

24 April 2018

Do It by the Book: How to Use Storytelling in UX Research

4 April 2018

User experience research vs. market research for product design and development – a blurring line?

22 March 2018

How to use storyboards effectively in UX projects

8 March 2018

Homepage fundamentals: Handy tips to entice your users

22 February 2018

Get in touch

  • SimpleUsability Ltd, Marshall Mills, Marshall Street, Leeds, West Yorkshire LS11 9YJ

    Leeds Tel: 01133 508 155
    Int Leeds Tel: +441133508155

  • SimpleUsability Ltd,
    27/31 Clerkenwell Close,
    London, EC1R 0AT

    London Tel: 020 3393 0651
    Int London Tel: +442033930651

  • Email: info@SimpleUsability.com

Copyright Behavioural Research Consultancy 2018

facebook rss twitter youtube