XR Accessibility: for people with hearing disabilities
Posted on by Joe Lamyman in Design and development
Extended Reality (XR) experiences tend to focus on providing immersive sounds and directional audio to convey information. But we need to consider how we convey the information in these experiences to people who can’t hear them.
In this post, we’ll explore considerations for designing and developing inclusive XR experiences for people with hearing disabilities.
If you haven't already, you can also explore other articles in this series including:
- Introduction to XR accessibility
- XR Accessibility: for people with seeing disabilities
- XR Accessibility: for people with moving disabilities
- XR Accessibility: for people with thinking disabilities
You can also watch my Introduction to XR accessibility talk from InclusiveDesign 24 (#ID24).
Who does this affect?
People with hearing disabilities might include:
- People who are Deaf and have been born with permanent hearing loss
- People who are deaf and have age-related hearing loss
- People who are in a loud environment and are experiencing noise-induced hearing loss
- People who have an ear infection and are experiencing hearing loss
- People who are experiencing migraines, who might find sounds uncomfortable and be unable to focus on listening to sounds
As we mentioned in Introduction to XR accessibility, there are different types of disability that might be permanent, temporary, or situational. Regardless, we need to create experiences that are usable by everyone.
Provide sign language
People who are D/deaf may use a sign language as their primary language. Therefore it's important that sign language is available in our services and products.
Here's a user story that summarises this need:
As someone with permanent hearing loss, I want to be able to view content in my primary language, British Sign Language (BSL), so that I can easily understand content.
It's about using the Inclusive Design Principle of provide comparable experience, to make sure that people can access content in their preferred way. For context, British Sign Language (BSL) is the first language of 87,000 people in the UK. Using their primary language might make the experience more usable for people. For people who use sign language, the expressions and tone information communicated by the interpreters, may make information easier to understand than reading text.
When adding sign language videos in XR we need to make sure that sign language interpreters are clearly visible and appropriately sized. The XR Accessibility User Requirements (XRAUR), which we covered in Introduction to XR accessibility, require that any signing videos are at least a third of the size of the original stream's size. Signing video sizes are important for helping to ensure that gestures and facial expressions are easily visible.
It's important that you design for and implement videos appropriately. Test your experience with the videos, make sure that they don't cover important information. Make sure that the videos can be enabled from the start of the experience. If you're unsure of how best to add or author these videos, consult people with hearing disabilities in your audience.
It's important to note however that sign languages are a lot more regional than spoken languages. The Game accessibility guidelines, notes:
However be aware that sign language is highly localised, due to it not having a written form it has much greater regional differences than speech. Regional accents are very strong, and international differences are extreme, with separate localised signing required for BSL, ASL and AUSLAN (British, American, Australian) for example, even though the original audio / captions would be almost identical.
This isn't to say that you can't include sign language. It's about understanding the needs of your audience and including languages that are the most relevant.
Identifying sounds and audio
We also need to consider the different types of information that we convey with audio. We may use audio as alerts to convey the position of items or people and identify objectives.
Examples of user needs for this requirement might include:
As someone with hearing loss, I want to understand any audio identifiers used in the experience, so that I can understand what I need to interact with.
As someone with ear pain, I want an alternative way to access audio information, so that I can continue to use the experience.
To address these needs, we can take a multi-modal approach to designing our experience. Throughout this XR Accessibility blog post series, we've covered different ways to present information to people. The core focus is to make sure that we don't rely on a single way of providing information.
When designing for people with hearing disabilities, we can pair audio information along with visual affordances. If we want to help people navigate to sounds, we can include arrows that point in the relevant direction. Such patterns will already be familiar to people who play video games. An example of this can be found in Fortnite, in which sound effects are displayed visually, allowing people to quickly understand the location of sounds.
There may also be ways that you can incorporate sounds into your experience. For example, characters or objects could respond when audio is played to help convey information. Characters in the game could help by using sign language if appropriate. Alongside this captions could be used to help describe any environmental sound effects. If available, haptic feedback could correspond to sounds to help people identify information.
Captions
Captions and subtitles are a great way of conveying information in XR, similar to the way they're used in other mediums. A user story for this might be:
As someone with hearing loss, I want audio content to be presented as text, so that I can easily understand the audio information.
Types of captions
Currently, there are two main approaches for presenting captions in XR:
- Head locked captions which are locked to the devices movements, and move with the hardware, so that they are permanently visible
- Fixed captions which are tied to a virtual or physical object and remain in proximity to that object, regardless of the device's movements
Each has their own benefits and drawbacks, and will depend on the context of use. If you're creating an Augmented Reality (AR) experience in which people can view different models, head locked captions may be more appropriate. Whereas if you're developing a Virtual Reality (VR) experience with characters, tying captions to the character that is speaking might work better.
Head locked captions are always visible, and alongside them, we can include directional prompts to inform people as to where the audio is coming from. Directional prompts could be included as arrows, pointing to the location of the source of the audio.
While fixed captions can allow you to easily identify speakers and the position of audio, we also need to consider their size. If the character speaking is far from the user, we need to ensure a minimum caption size is included to maintain legibility. Similarly, if the character is off-screen but can be heard, ensure the captions remain visible.
An example of great caption design in XR can be found in Owlchemy Labs' games. In the talk Subtitles in XR: A Practical Framework, Andrew Eiche explains how they use a blended approach to XR captions. The way this works is by using fixed captions when the character that is speaking is visible, but when the character is out of view, the experienbce switches to head locked captions. The head locked captions follow the VR device's head movements, and use an arrow to point to the off-screen location of the speaker.
Regardless of the method used, captions must be available to people using assistive technologies such as screen magnification as people might be both low vision and hard of hearing.
Customising captions
As always, give control to people to allow them to customise captions. People may want the names of characters to be included in captions, they may want directional information, and they may want to change the presentation of captions. Ensure that people are able to choose between different font types and different font sizes to find a presentation that works best for them. Similarly, there should be a choice of sufficiently contrasting text colours and backgrounds, that can help people to better perceive the captions. All of these different options allow people to make the experience more usable based on their own needs.
Another of the Inclusive Design Principles is relevant for this functionality, be consistent:
Familiar interfaces borrow from well-established patterns. These should be used consistently within the interface to reinforce their meaning and purpose. This should be applied to functionality, behavior, editorial, and presentation. You should say the same things in the same way and users should be able to do the same things in the same way.
Your captions and available options should match those that people are familiar with when using other forms of multimedia. People might be very familiar with the way that captions work in their favourite streaming platforms and video games, so it's important that your implementation is consistent. Morgan L. Baker's fantastic article on the topic provides an in-depth summary of best practice for subtitles and closed captions.
Mono and stereo audio
A final consideration is to make sure that experiences can provide information using both stereo and mono audio. While stereo audio can make use of multiple speakers to replicate how directional sound works, this information can prevent people from hearing sounds if they have hearing loss in one of their ears. This is because in delivering stereo audio, sounds might be quieter, or silent, through one speaker or earphone. If this is the case, people might miss this audio cue.
Provide an option that can combine multiple audio channels used for stereo audio, into a single, mono audio track. This mono audio can then be provided through all speakers, so people can hear all the audio information. A user story for this would be:
As someone with hearing loss in one ear, I want to be able to switch on mono audio, so that I can hear all the audio.
Summary
XR allows people to experience immersive environments. We must create these environments in a way that includes the needs of people with hearing disabilities. This can be done by providing information in people's primary language, designing in a multi-modal approach, providing captions and subtitles, and allowing people to switch between mono and stereo audio.
More information
- Accessibility options for The Last of Us Part II, by Playstation
- Accessibility Virtual Reality: How to Add Captions Using Open Source, by Equal Entry
- Inclusive XR: accessible 3D experiences, by Joe Lamyman
- Inclusive XR: accessible augmented reality experiences, Joe Lamyman
- When/How to use Deaf Accessibility Tools (a11yTO Gaming 2020), by Morgan L. Baker
- Xbox Accessibility Guideline 104: Subtitles and captions, by Microsoft
- XR Accessibility User Requirements, W3C
Next steps
If you're currently designing an XR product, our design review service will provide you with our accessibility expertise and guidance to build an accessible experience. If you already have an XR product, our assessments can help you to understand whether your product meets accessibility standards.