Accessibility You Can Show
What a check can prove, what AI can’t, and the conformance report a buyer can open.
By Adam Stankiewicz, founder of Variantly · A Variantly white paper, companion to Every Product Is Designed Twice · 12 October 2026
Variantly sells a product that builds these reports. We say where every figure comes from, and where a figure is still missing.
A worked example: the invitation confirmation
| What was meant | After sending, the admin knows the invitation was sent, and to whom. |
|---|---|
| What shipped | “Invitation sent to sam@example.com.” appears under the heading and is gone after three seconds. |
| What the evidence shows | A capture 0.8 seconds after Send, in Ledgerly, our own sample app: the accessibility tree shows the message as plain text, with no status role or live region; axe-core 4.13.0 reports 0 violations. That is what WCAG 4.1.3 Status Messages addresses. The capture was made by a Playwright script on 12 October 2026. No screen reader was run. |
| Who decides | A person sets the 4.1.3 row with the evidence open. Rules that pass don’t set it. |
| Stage | Item (sample from the full paper’s checklist) |
|---|---|
| Gather the evidence | Attach what the rules found to each criterion. A pass alone doesn’t show the criterion is met. |
| Set each answer | Have a person decide each answer with the evidence open, and record who decided and on what. |
| Questions to ask of any ACR | What evidence does “Supports” rest on, and can I open it? |
Summary
This is the companion to Every Product Is Designed Twice, which argues for one review of the product people actually use, shared by design, research, accessibility and engineering. This paper takes the accessibility seat. It asks what that review has to record before someone outside the team can trust a conformance answer, and what a report needs before a buyer can check it.
Universities, public agencies and large companies ask vendors for an ACR, usually on the VPAT® template. Most are self-reported: a vendor, or a consultant it hired, fills in each criterion. A buyer who reads “Supports” can’t tell whether it rests on an expert’s afternoon with a screen reader, a clean automated scan, or a hope.
The answer isn’t a better adjective. It is a report where every row says how it was set and opens the evidence it rests on. That needs four things, one per chapter of the full paper:
- Know what a check can prove. A failure is evidence. A pass isn’t conformance.
- Measure the checks in public. Against shared test cases, with the bar set first.
- Keep one record. Accessibility evidence comes from the same captures the rest of the team reviews.
- Show how each answer was set. And what the report covered.
1. What a check can prove
Automated accessibility checks are useful and limited, and the limit is precise. The W3C’s ACT Rules Format 1.1 (February 2026), which defines how accessibility test rules are written, says that when every outcome of a rule passes, “further testing is needed”. A rule can show that a criterion fails. It can’t show that the criterion is met.
That matters to anyone who writes a conformance report. A “Supports” that rests on a clean scan claims more than its evidence shows: the scan found nothing it was built to find, on the pages it reached.
So every answer should say how it was set:
- Rules attach what they found to a criterion. A failure is evidence; a pass means further testing is needed.
- AI can propose an answer, with its evidence. A person checks a random sample; outside it, only a failing answer is accepted, marked as AI proposed.
- A person decides with the evidence open. That is the only source of “Supports”.
- Coverage is stated with every answer: what was captured, and what wasn’t.
And per WCAG-EM 2.0 (July 2026), the W3C’s method for evaluating a website, a claim for a whole site can’t rest on a sample alone. A report says what it covered: which pages, which signed-in areas, which tasks, and what it didn’t reach.
3. One record, every seat
In the review Every Product Is Designed Twice describes, accessibility isn’t a separate audit. It is one seat at a shared record, and each seat asks its own question of it.
| Seat | Asks | Decides | Takes away |
|---|---|---|---|
| Designer | Did the flow match what I meant? | Fix it, not sure yet, or working as intended | The issue on its screens, in order, with what was meant and what people met |
| Researcher | What should we study with people? | Which open questions go to research | The issues marked not sure yet, with the screens that raised them |
| Product | What first, and did the change work? | Priority | A counted summary with its limits, and what was seen after each change |
| Engineer or coding agent | What do I change, and how will I know? | When a change is ready to check (an agent proposes; a person decides) | The issue’s evidence, its location, and how to check it again |
| Accessibility lead | Can I show a buyer? | Each WCAG answer, with the evidence open | A conformance report whose every row says who set it and on what evidence |
For the accessibility lead, this changes where the evidence comes from. Each state of a task is captured five ways: the screenshot, the accessibility tree, the focus trace, a reading built from that tree (no screen reader is run) and rule output. The capture that shows a designer a confirmation vanishing after three seconds shows the accessibility lead that it isn’t exposed as a status message: the pattern WCAG 4.1.3 Status Messages addresses, on a page where the rules report zero violations. One issue, one fix, one check afterward.
The full paper adds
- How accurate the checks are. Our rule checks measured against the public ACT Rules test cases, the bar we set before seeing results, and how the figure could mislead.
- A report a buyer can check. What each row says about how it was set, why an AI walkthrough doesn’t set a row, and the dates buyers are working to.
- Putting it into practice. Six steps for the team writing the report, and four questions to ask of any ACR.
- The benchmark, criterion by criterion, with how to reproduce the run.
Or email me a copy
How Variantly does this
Everything above works without us. Variantly exists for teams that want the report to stay current without rebuilding it by hand.
- Crawl and replay. Public pages, signed-in areas with a test account, and key user journeys replayed step by step, every state captured with its five records.
- Grouped by cause. Failures are grouped into the components that cause them, so one fix closes many.
- Every row says how it was set. Rules attach what they found; AI proposes and a person checks a sample; a person decides with the evidence open.
- A report on a public link. VPAT® 2.5 and OpenACR, with version history, kept current by nightly replays that catch drift.
Paste a URL for a free, private accessibility check of your public pages.
Ship what you meant.