Accessibility Testing with pa11y

Knowing where to start with accessibility testing can be difficult and overwhelming. In my article on Tools for Developing Accessible Websites, I mentioned 5 tools you can use to get started developing more accessible websites. More recently, I have discovered an incredibly powerful tool - pa11y

Introducing pa11y #

pa11y, pronounced pally, is a set of free and open source tools that aims to make designing and developing accessibility easier. They have a number of different projects to help with this, for example a web dashboard interface, but I will be focusing on 2 of their projects: pa11y and pa11y-ci.

Getting Started with pa11y #

As a beginner, the easiest way to get started with accessibility testing is by installing the core package, pa11y.

npm install -g pa11y

This gives us access to the command pa11y, which we can use to run a simple accessibility audit on a particular url. Let's take this blog's homepage as an example.

pa11y https://bitsofco.de

Running the above command, however, will likely send you into spiral of doom because every single error, warning, or notice will be outputted into your terminal. But don't fret, because there are a few simple options you can pass to ease you into the process.

Only display critical errors #

When first starting out, you will likely just want to see the crucial errors that you need to fix, and not just every single accessibility-related message related to the page. pa11y divides their messages into three types:

1. A notice is general message about an element's accessibility. It isn't necessarily something that is wrong with your implementation, but rather a notice to be aware of concerning a particular element. For example, a notice can be "Notice: Check that the title element describes the document."

2. A warning is something that pa11y considers may be an issue, so you should take a look at it. For example, a warning you may receive on an <img> element with a null alt attribute is "Warning: Img element is marked so that it is ignored by Assistive Technology.". Using a null alt attribute isn't necessarily an error, as null attributes can be intentional (see Alternative Text and Images), but it is something that you should look at to make sure

3. An error is a critical issue that should be fixed. For example, an error you may receive on an unlabelled for input is "Error: This text input element does not have a name available to an accessibility API."

For your first audit, you probably want to exclude notices and warnings, so you only see the crucial errors on your page. This can be done by using the --ignore flag -

pa11y https://bitsofco.de --ignore "warning;notice"

Set thresholds for failure #

Once you've set your option to only see critical error messages, you can specify the amount of those errors that should be permitted in order for the page to "pass" the accessibility test. This is done using the --threshold flag -

pa11y https://bitsofco.de --threshold 10

The command above, for example, will allow up to 10 errors on the page before the page will be deemed to have failed the test.

Hide elements #

Finally, another great option for beginners is to hide certain elements from testing. This can be useful if, for example, you have third party widgets such as ads on your page that you can't control. Or, you can use it to hide elements that are visual or screen-reader only.

pa11y <url> --hide-elements "#ads .sr-only [aria-role='presentation']"

Using this blog as an example, I want to hide the errors related to Carbon Ads widget (#cardbonads) -

pa11y https://bitsofco.de --hide-elements "#carbonads"

Putting it together #

Putting all of these together, you should get a useful, non overwhelming audit of your page, with actionable errors to fix.

pa11y https://bitsofco.de --ignore "warning;notice" --threshold 10 --hide-elements "#carbonads"

Doing more with pa11y #

Using pa11y in the way described above can be great when you're getting started, but pa11y can be used for much, much more than that.

The basic pa11y command starts to fall a bit short when you need to test multiple pages, each with different options. Although the core package can be configured in a more advanced way by creating a dedicated node script, I think the pa11y-ci package does a better job at handling the more complex audits.

npm install -g pa11y-ci

This will give you access to the pa11y-ci command. It can be used in a similar way to pa11y, but most of the configuration options are stored in a .pa11yci JSON file.

Here's an example of the same basic configuration mentioned above, but in a .pa11yci file:

{
  "defaults": {
    "hideElements": "#carbonads",
    "ignore": [ "notice", "warning" ]
  },
  "urls": [ "https://bitsofco.de" ]
}

The only option that is still specified as a command line option, is threshold. So, we can execute all our options using the following command:

pa11y-ci --threshold 10

(Note that pa11y-ci expects your .pa11yci file to be in the root directory.)

Once we've got the default configurations down, we can move on to some more advanced customisations.

Testing multiple URLS #

The first thing you might notice about the .pa11yci configuration, is that there is an array of URLs we can pass. This allows us to test multiple URLs in one go. Using this blog example, I may want to test the home page, the newsletter signup page, and a single article page.

{
  "urls": [
    "https://bitsofco.de" ,
    "https://bitsofco.de/subscribe",
    "https://bitsofco.de/collapsible-margins"
  ]
}

What's even greater, is that we can specify custom configurations for each page. We do this by passing an object into the urls, instead of a simple string. In addition to the url, we pass any of the pa11y configurations we want to override for that particular page.

For example, I want the #carbonads element to be hidden globally because it shows up on all pages. On an article page, I also want to hide the comments (#disqus_thread), so I can set a custom hideElements configuration only for that page.

{
  "defaults": {
    "hideElements": "#carbonads"
  }
  "urls": [
    "https://bitsofco.de",
    "https://bitsofco.de/subscribe",
    {
      "url": "https://bitsofco.de/collapsible-margins",
      "hideElements": "#carbonads, #disqus_thread",
    }
  ]
}

Customise accessibility standards and rules #

A useful configuration option is the standard option. The default accessibility standard used by pa11y to test a page is the WCAG2AA. However, we can change it to any of the following options - Section508, WCAG2A, WCAG2AA, or WCAG2AAA.

{
  "defaults": {
    "standard": "WCAG2AAA"
  }
}

Additionally, we can specify a set of specific WCAG 2.0 guidelines to include. For example, you may want to follow the WCAG2A standard in general, but include a couple of specific guidelines that would normally apply to a stricter standard.

{
  "defaults": {
    "standard": "WCAG2A",
    "rules": [ 'Principle1.Guideline1_4.1_4_6_AAA' ]
  }
}

The above requires that the specific guideline 1.4.6, "Contrast (Enhanced)" must be followed.

Testing user actions #

Besides a basic audit of your markup, pa11y can be used to test actions that a user might take on your page. Using their custom set of keywords, you can demo a flow of actions.

For example, on the subscribe page of this blog, I may want to test that a user can actually fill out the form to subscribe.

"urls": [
  {
    "url": "https://bitsofco.de/subscribe",
    "actions": [
      "set field #mce-EMAIL to pa11y@testing.com",
      "set field #mce-FNAME to Tester",
      "click element #mc-embedded-subscribe",
      "wait for url to not be https://bitsofco.de/subscribe"
    ],
    "timeout": 60000
  }
]

When testing actions, it's important to set a long timeout to allow pa11y to go through all the steps.

Device-specific testing #

pa11y allows us to specify the viewport dimensions to test a page with, allowing for device-specific testing.

{
  "defaults": {
    "page": {
      "viewport": { "width": 320, "height": 480 }
    }
  }
}

Doing even more with pa11y #

pa11y has what seems like an endless list of features and configuration options so we can test our sites in any number of ways. I haven't gone over every feature or option available, such as implementing continuous integration which the the pa11y-ci package is actually built for. If you want to find out more, check out the repositories for pa11y and pa11y-ci. If there are any particular features you would like me to cover in more detail, leave a comment below.

You can see a full example of how I use pa11y to test this blog on my repository, ireade/bitsofcode-pa11y.