Inclusive user research: analysing findings

Posted on Wednesday, 8 June 2022 by Ela Gorla

In moderating usability testing with people with disabilities we covered the skills and techniques that help researchers run sessions smoothly and collect valuable insights. The second post in our Inclusive user research series discusses some of the unique challenges posed by findings from sessions run with people with disabilities, and advice on how to analyse them.

Researchers have a wealth of data that needs organising, understanding, and filtering once the usability testing sessions are complete. For research that includes people with disabilities, the analysis stage often requires a bit more time and consideration. Properly investigating issues, and considering the needs of different people, is crucial for coming up with recommendations that will improve the product for all.

Challenges

When running usability testing with people without a disability, participants tend to encounter similar issues, and often provide similar suggestions for improvements. For example, they may all miss a link towards the end of a screen and mention it should be moved up or made bigger. You would therefore provide designers with this feedback, together with a few suggestions on how to improve the visibility of the link.

There may be times when participants make suggestions for changes based on their preferences. They may not like the colour scheme or images used on a website, for example. As an experienced researcher, you are able to distinguish personal preferences from usability issues and make an informed decision on whether to report them.

Findings from sessions involving people with disabilities can be more varied, which makes the analysis stage interesting and challenging at the same time. Here are a few of the reasons behind such a variety of results.

Unique ways to use products

Each person is likely to interact with the test product uniquely. Some may use assistive technology, others may enable accessibility settings, and others may have their own way of exploring and processing content. This often results in each participant experiencing very different issues. What works perfectly well for a person may be a complete blocker for another. Collecting, reviewing and making sense of such a wealth of information takes time.

Personal preferences versus accessibility issues

If you are not familiar with how participants use assistive technology or accessibility settings, it's more challenging to separate personal preferences and actual accessibility issues. For example, a blind person using a screen reader may comment that a data table is difficult to navigate. This could be because the table is not accessible or the participant dislikes or is unfamiliar with navigating data tables.

Conflicting suggestions

People will have ideas on how to improve the test product. Feature requests always provide valuable insight, but they may sometimes seem to conflict with one another. Because each person has different needs, a change that would make a product easier to use for one person may negatively impact someone else's experience. Finding a solution that benefits all may be challenging.

Strategies

Learning to make sense of a wide variety of data, some of which may be contradictory, and coming up with solutions that best address them all takes practice. Here are some strategies to get you started.

Learn about technical accessibility requirements

When you understand how content should work with accessibility settings or assistive technology, you can more easily recognise irregularities. This makes differentiating between personal preferences and technical issues easier.

Going back to the data table example, if screen readers announce its content as expected, there is probably nothing wrong with the code. Asking developers to change it based on the feedback of one participant may actually break its accessibility. Suggesting that the content is also presented in an alternative format would be a far better recommendation and benefit all people who struggle to interpret tables.

Our browsing with assistive technology videos give an overview of how popular assistive technology is used to access web content. The Web Content Accessibility Guidelines (WCAG) 2.1 techniques provides useful information on how content should be coded in order to work well with assistive technology.

Be aware of the needs of different people

Implementing a new feature or modifying an existing one based on a participant's feedback may be tempting, especially when it clearly improves that participant's experience. However, before you do that, consider the impact of that change on other people. Could it make it more difficult for other participants to use the product? What about people with disabilities that were not part of the participant sample?

In a research session we conducted, a lady with Cerebral Palsy suggested that we replaced a few drop-downs with input fields. Containing a long list of options, the drop-downs required a lot of scrolling, which she found challenging and tiring. However, we are aware that input fields pose other types of challenges to other people, and are more prone to errors. As per the data table example, offering people alternative ways to complete the same task would be a better solution. This also meets offer choice, from the Inclusive Design Principles.

Running usability testing sessions with people with a wide variety of disabilities is a great way to learn about different people's needs. The W3C Web Accessibility Perspectives Videos may also provide a good introduction.

Try reproducing issues

For issues experienced by people using assistive technology or accessibility settings, always try to reproduce them following the testing sessions. This is to verify the issues were:

not caused by a temporary glitch with the technology
a code error that has been resolved since the testing
an error on the participant's part

Failing to investigate technical issues properly may once again result in developers breaking code that was accessible.

Occasionally we have been unable to replicate issues following user research. In some cases, the code causing the issues had been replaced; in others, participants, still new to their assistive technology, had used an incorrect command to interact with content.

If you are not familiar with the assistive technology used by participants, or do not have access to it, you can ask someone else either within or outside your organisation to test the content for you. Ideally, the same version of assistive technology, accessibility settings, browser, operating system and device model should be used.

Writing recommendations

Once you have finalised a list of issues to report, write accurate and clear recommendations on how designers, content authors, and developers should address them.

Including links to relevant Inclusive Design Principles may inspire and help designers come up with ideas on how to improve the product for all. For technical accessibility issues, include details on the portion of code causing the issues and how to change it, and details of the assistive technology or accessibility settings used, browser version, operating system and device model. Links to relevant Techniques from WCAG 2.1 may also be useful.

If assigning issues and recommendations a priority level, carefully consider the impact of each observed issue on the product's users, and the effort required to solve it. We often recommend that clients focus on the issues with the most severe impact first, as well as those issues, large or small, that can be fixed quickly. This allows to start improving the accessibility of the product straight away.

Summary

Thanks to the variety of people who take part in inclusive user research, findings are always interesting, sometimes surprising, and at times conflicting. Analysis can take time, and requires accessibility knowledge and skills. Take your time to learn about technical accessibility requirements, assistive technology and accessibility settings, and how different types of disabilities impact people's experiences with digital products. With the right knowledge and experience, you can come up with recommendations that will improve the product for all people, with and without disabilities.

Learn how to moderate usability testing people with disabilities, or find out about usability testing at TetraLogical.