Moderating usability testing with people with disabilities

Posted on Tuesday, 4 January 2022 by Ela Gorla in Design and development

All user research projects should include people with disabilities. Representing at least 15% of the world population, according to The World Bank, and potentially your target market, it doesn't make sense to exclude them. This post provides user researchers with tips on how to best moderate usability testing sessions where participants have a disability.

As an experienced researcher, your existing skills will help when conducting research with people with disabilities. Combining moderation best practices with flexibility, quick problem-solving and consideration for participants will allow you to run sessions smoothly and collect valuable insights.

While careful planning, familiarity with assistive technology, and technical skills are also required to successfully design and carry out inclusive user research, this post focuses only on session moderation.

Take your time

We have all been there: you have several journeys to test and many questions to ask in a short amount of time. Rushing through the welcoming and introductory sections and reading out scenarios and instructions as quickly as possible can be tempting. However, this will only result in people getting confused and anxious, with some not understanding what they are required to do. This is particularly true for people with certain disabilities, such as people with hearing loss or cognitive disabilities. You must take your time, talk slowly and clearly, and pause to check people's understanding.

It is just as important you give people the time they need to process information and answer your questions. Some people with disabilities may need slightly longer to understand instructions and formulate answers than other participants. Allow silence, do not interrupt people as they think or talk, and only move to the next question when you are sure they have nothing else to add. In sessions with people who have a speech disability, such as stammering, do not complete words or sentences for them. Not only is this disrespectful, but it can also worsen people's anxiety making it more challenging to answer questions.

Similarly, allow people as much time as they need to attempt tasks. Some assistive technology users may take longer to navigate and operate digital products than others. People with cognitive disabilities may spend a while on pages reading and processing content. Never put pressure on participants; let them use the product as they usually would, even when this means spending longer on a page than you had anticipated.

Finally, some types of disabilities can cause people to get tired quickly. Offer as many breaks as needed during the session and end the session if they are too tired to continue. Participants' wellness should be any researcher's priority.

If you are concerned you may not be able to go through all tasks and questions in your moderation guide in the allowed time, remove some. It is not uncommon to have a shorter moderation guide for sessions with people with disabilities. This also allows for additional time to troubleshoot technical issues that may arise during the sessions (more on this in the Use your problem-solving skills section below).

Be flexible

When running research with people who have a variety of disabilities, you may need to adapt your moderation style based on each person's abilities and preferences.

Your instructions, questions and scenarios may not work for everybody. People may not be familiar with words you have used, some may find your questions difficult to process or not specific enough, and tasks may not feel realistic to them. Be ready to make adjustments on the fly: you may need to use simpler words, shorter sentences or gestures, ask more precise questions or modify scenarios slightly.

This happened to us in a recent research project where one of the tasks required people to enter details from an ID document such as a passport. Some of the participants did not own one, so we quickly adapted the task so that it better reflected a real-life scenario for these participants.

The way you plan to share content may also pose difficulties. For example, some people may struggle to navigate to or open a chat and read instructions posted in there. Be prepared to try out alternative communication channels. In the example of the chat, it may be as simple as reading out the instructions for the participant, which may also work better for some people with reading disabilities. Similarly, offer people alternative ways to complete activities, such as communicating scores verbally if entering them in an online form is challenging.

A technique commonly used in moderated usability testing is to ask participants to "think aloud" as they use a website or app. It is a simple and effective way to capture people's thoughts, reasoning and emotions. However, it may not work well with all participants. Think of screen reader users, for example, who may struggle to talk while listening to their screen reader. If a participant lets you know they cannot or prefer not to think aloud, do not pressure them and assure them it is fine not to. You can still learn a lot by observing them.

If you plan to use rating scales as part of a research session, keep them as simple as possible. This means writing questions in simple language and using a small scale, for example a 5-point scale rather than a 7-point one. Even so, scales may be challenging for some people. Be prepared to rephrase the questions and offer people a different way to express their opinion: a thumb up or down may be all you need.

Use your problem-solving skills

We have all been in testing sessions where the website or app being tested isn't working, or the test device keeps crashing. When running sessions with people who use assistive technology or adjust their accessibility settings, you are even more likely to encounter problems. In fact, using AT or adjusting accessibility settings adds a layer of complexity to the technical set-up of a session. You must be able to think on your feet and find solutions to continue the session should technical issues arise. This may mean finding a way to bypass a screen, switching to a different version of the test product, or changing a task on the fly.

In a recent research session we were testing both the website and app version of a user journey. In one session the app kept crashing when used with a specific assistive technology. After a few attempts, we decided to move away from the app and ask the participant to complete tasks using the website instead. We then asked a different participant who was testing the website later in the week to test the app instead. This way we were still able to collect useful information on the content presentation and understandability.

At other times, technical accessibility issues with the website or app being tested may be complete showstoppers for participants. Again, you may decide to divert from the moderation guide or provide assistance to proceed. As long as you accurately document where and why the problem occurred, this is the best approach to collecting as much feedback as possible.

In another project we worked on recently, we observed a few participants who were using a very large font size becoming stuck when key content including the main call to action ended up being off-screen. With no option to scroll the content and make it visible, there was no way for these participants to continue the journey. Rather than ending the session, we decided to help them move to the next screen: we asked them to reduce the text size in their device settings so that the content became fully visible on screen again, then we directed them to the button they needed to activate, which was too small for them to see. Once on the next screen, they went back to the device settings and selected their preferred font size again, before proceeding with the task. When documenting the research findings for the client, we made it clear that the issue with the enlarged content not fitting on the screen was a complete showstopper for these participants.

Be respectful

Researchers must always show respect to participants. This means using inclusive language when talking about disabilities and people with disabilities. For example, avoid negative words or phrases, such as "suffers from" or "wheelchair-bound". Instead, say "has epilepsy" or "uses a wheelchair". Never identify people by their disability, such as saying "the blind". Instead, say "people who are blind". A disability is what someone has, not what someone is.

Referring to people without disabilities as "normal" or "able-bodied" should be avoided, as it implies that people with a disability are "abnormal". The Etiquette: Interacting with People with Disabilities article on the RespectAbility website provides valuable tips on how to communicate with people with disabilities.

Being respectful also means respecting people's privacy. While the information on the assistive technology or adaptive strategies used by participants is important, details about their health are often not needed. Unless people volunteer this information, limit your questions to what is absolutely required. If participants' health happens to be discussed in the session, remove all details from the recordings before sharing them together with other personal information. Such sensitive information must not be available to people outside of the research team working on the project.

Recognising the uniqueness of each participant is another way to show respect. Because two people have the same disability and use the same assistive technology, it does not mean they use products in the same way. People's personal preferences, past experiences, attitudes toward technology and many other factors play a role. Refrain from asking questions that generalise participants' experience, for example, "Do you think that other people who are blind would find this website easy to use?"

Summary

No user research is complete when people with disabilities are excluded. Next time you run a research project, make sure to recruit people with various disabilities and use the advice from this post to make each session enjoyable for both you and the participant.

Next steps

Find out more about agile user experience testing at TetraLogical and how people with disabilities browse the web with assistive technologies.