Testing methodology

How Allie produces its results, and how to read them.

The engine

Allie uses axe-core, a widely used open-source accessibility testing engine. Every report states the axe-core version that produced it. Allie loads the page in Chromium, waits for it to finish loading, then runs axe-core on the rendered page.

Which rules run

Allie enables axe-core rules tagged for these WCAG levels:

• wcag2a and wcag2aa: WCAG 2.0 Level A and AA
• wcag21a and wcag21aa: new criteria in WCAG 2.1 Level A and AA
• wcag22aa: new Level AA criteria in WCAG 2.2, as far as axe-core can test them automatically (for example target size)

Together this tests against WCAG 2.2 A and AA. Experimental rules and general best-practice rules that are not tied to a WCAG criterion are not included, and Level AAA is not tested.

Reading the results

Fail (an issue)

axe-core is confident that an element breaks a rule. Each failing element counts as one issue. Severity (critical, serious, moderate, minor) is the impact rating given by axe-core for that rule.

Pass

A rule applied to the page and found no problems on the elements it checked.

Needs review

axe-core could not decide automatically, for example when text sits over an image or gradient and contrast cannot be measured reliably. These are not counted as issues. A person should check them by hand, so they appear as a count in the report.

Not applicable

The page contains nothing the rule applies to, such as a video rule on a page with no video.

Scan conditions

• One page per scan, in its state right after loading, as a visitor who is not logged in.
• A 1280 by 800 pixel desktop window in the light colour scheme.
• Time limits apply: 30 seconds for the page to load and 60 seconds overall.
• Content behind logins, dialogs, tabs, menus and other interactions is not tested unless it is visible after load.

What automated testing cannot do

Automated tools cannot judge whether alt text is meaningful, whether content is understandable, whether the focus order is logical, whether captions are accurate or whether a task is usable with a screen reader. Automated testing covers only part of WCAG, so a page with zero automated failures is not necessarily accessible or conformant.

Allie's suggested fixes are generic starting points. A developer should review them in context.

Not legal advice

Allie reports are technical test results. They do not certify conformance with WCAG, the ADA or any other law, and do not protect against claims. For legal questions talk to a qualified professional. Questions about our method? hello@a11ie.io.