Representative samples for accessibility assessments

Posted on Friday, 14 August 2020 by Léonie Watson in Strategy, Testing

When a website is assessed using the Web Content Accessibility Guidelines (WCAG), it is rarely feasible to test every single page. Instead, the assessment can be based on a sample of pages that are representative of the website as a whole, to keep things as efficient and cost-effective as possible.

Why testing every page is not feasible

The reason it is almost never possible to test every page is simple logistics. There are 30 Level A and 20 Level AA Success Criteria (SC) in WCAG 2.1, another 28 if you include Level AAA. Leaving Level AAA aside for a moment, that's 50 checks that need to be made to every page included in the assessment.

Some of those checks can be done quickly and some can be made easier through the use of tools, but others take more time because they are complex or need careful inspection.

For example, SC 1.2.2 says that captions must be provided for prerecorded video (unless the video itself is an alternative to some text content). Checking if captions are available can be done quite quickly, but in the information to help you avoid failing the SC there is F8: Failure of Success Criterion 1.2.2 due to captions omitting some dialogue or important sound effects.

Every SC has a number of "sufficient" and "failure" techniques. The techniques are not a formal part of the SC, but they are intended to help people understand, interpret and test it. So to test SC 1.2.1 properly it is necessary not only to check that captions have been provided, but also to check that the captions are an accurate representation of the dialogue and sound-effects in the video - and that takes time.

Let's say that on average, it takes about 3 minutes to test an SC, or 2.5 hours to test a page against all 50 Level A and Level AA SC. That might sound achievable, but it doesn't consider the following:

1. Time to write up the findings
2. The scale of the website

There is little point in assessing a website unless the findings are reported, so let's increase the average to just over 5 minutes for each SC or 4.5 hours for every page.

For a simple brochure website of a few pages this is manageable, but at the other extreme the BBC has around 12.5 million pages. Most websites sit at the lower-end of that range, but even a website with 100 pages would take 450 hours to assess. That's a little more than 69 days (assuming an intense 8 hour day with reasonable time for breaks).

That is a lot of time and a lot of money. It also includes a lot of unnecessary duplication.

That is where the idea of a representative sample of pages comes in.

About representative samples

Step 3 of the Website Accessibility Conformance Evaluation Methodology from the W3C covers selecting a representative sample of pages:

During this step the evaluator selects a sample of web pages and web page states that is representative of the target website to be evaluated. The purpose of this selection is to ensure that the evaluation results reflect the accessibility performance of the website with reasonable confidence.

A good representative sample should include examples of different types of page layout and design, content and functionality. It should be large enough to be representative of the website as a whole but not so large it is wasteful in terms of the time and money invested in the assessment.

Factors that can influence the size of the representative sample include:

• The size of the website; the larger it is the more likely it is to have many different page layouts, content and functionality types
• The age of the website; older websites often have areas that use earlier or different page designs, and pockets of forgotten content
• The complexity of the website; websites with lots of rich interactive content, or content that is responsive or available in different ways (light/dark mode or multiple languages for example) tend to mean a larger representative sample is needed

When the sample of pages is chosen well, it should be representative of the website as a whole and therefore representative of the different accessibility issues that affect it. Any accessibility issues that are identified within the accessibility sample need to be fixed there, and then the same fixes need to be applied to all other pages across the website with the same layout, design, content or functionality type.

Next steps

Find out more about the Web Content Accessibility Guidelines in our WCAG primer and how assessments can help you identify issues in your websites, mobile applications, design systems, and other products and services.