Skip to content

A method you can inspect.

Variantly turns an intended task into browser evidence your team can inspect. Here’s the research behind our design choices, how a review works, and what its findings can tell you.

Start with what someone is trying to do.

A review needs a purpose. You describe the task and how you’d know it worked. That gives the review a concrete question to investigate, from the first action to the last confirmation.

Intended
sam@demo-target.test is listed, and the admin knows it was sent.
Observed
After Send invitation, the confirmation disappears on the next screen.

Ledgerly invitation example. Illustrative demo, not evidence from a customer review.

Read the example review →

A goal gives critique something to answer.

UICrit asks designers to identify the interface’s task before critiquing it, then links comments to regions of a screenshot. That work informs our emphasis on task context and evidence tied to a specific place. [1]

Variantly follows the task across browser states. The useful question is whether the invitation reached its intended state and what got in the way. A static screenshot can show a label; the next screen can show that the confirmation disappeared.

Make the perspective explicit.

You describe the knowledge, role or interaction constraints to examine. A new administrator and someone using only a keyboard can expose different conditions in the same task. Stating those assumptions makes the attempt easier to interpret.

Wang, Morgenstern and Dickerson show how identity-conditioned simulations can misrepresent groups and flatten their diversity. That informs our preference for concrete task conditions and context you can substantiate, rather than treating an identity label as evidence of someone’s behavior. [3]

A perspective is a stated test condition. It does not represent a population or establish how people with that background or disability will behave.

Keep the observation close to the claim.

TaskAudit investigates mobile accessibility barriers that appear through interaction, using execution traces to examine what happened. It informs our choice to follow a task through its changing states. Its mobile study and screen-reader proxy do not establish web conformance. [2]

Variantly keeps the screens and recorded steps behind an issue so your team can inspect the condition it describes. An attempt stopping is not, by itself, evidence that the task worked. AgentRewardBench treats evaluating a browser attempt as a separate problem and documents limitations in both model judges and benchmark rules. [4]

WCAG-EM recommends documenting coverage and findings, with supporting details such as screenshots, DOM snapshots, navigation steps and tool versions. This informs our emphasis on evidence someone else can examine. A capture still says only what was available in that state. [6]

Give rules and interpretation different jobs.

Deterministic checks look for defined failure conditions, such as a control without an accessible name. AI can propose context-dependent issues, such as an unclear confirmation. Both keep their evidence; your team decides what to do with it.

ACT Rules distinguishes a check’s applicability, expectation and outcome. A rule can cover only part of a WCAG criterion, so a pass can still require further testing. Variantly’s rule findings therefore attach to the report; they do not decide its conformance rows. [5]

Recheck the condition after a change.

A recheck looks for an issue’s condition again and records what matched, what changed and what could not be compared. Repeatability means keeping the task and comparison visible. It does not mean an AI attempt will always take the same path.

τ-bench studies consistency across repeated attempts as a separate question from success in one trial. That distinction informs how we read rechecks: one observed result is one piece of evidence. The paper’s API-based tasks do not measure Variantly’s repeatability. [11]

Recorded accessibility tasks can also replay when their schedule is enabled. Changed evidence can make a report row stale and require another decision. Finding a condition absent in one run does not show that it is absent everywhere.

What runs, who decides, and what isn’t covered.

The operational method, from a captured state to a decision your team can check.

CaptureThe machine

A headless browser opens the pages and task states included in the review. It records a screenshot, the accessibility tree, the focus trace, a screen-reader transcript derived from the accessibility tree, and the results of axe and Variantly’s own rules. Short-lived states, like a message that disappears, are captured too.

AttemptsThe machine, from a perspective you describe

An attempt works through a task from a perspective you describe, signed in with a test account you control when the task needs one. It records each step and what it saw. An attempt isn’t a person: it shows how the interface behaves, not how people feel about it.

IssuesProposed, then decided

Rules report what a rule can detect. AI proposes the rest, and each proposal keeps the screens it rests on. A person decides each task-review issue: Fix it, Not sure yet, or Working as intended. Issues from task reviews never set a row in the conformance report.

Conformance reportA person; the edition, for one row

Each WCAG criterion reads Supports, Partially Supports, Does Not Support, Not Applicable or Not Evaluated. Rules attach what they found to a row; they don’t set it. The AI proposes a value and a person checks a sample, or a person decides it with the evidence open.

Supports comes only from a person who opened the evidence, with one exception: the WCAG edition sets Parsing (4.1.1), Supports in WCAG 2.1 and Not Applicable in WCAG 2.2, and no page is read to set it. Every row says which.

When you publish the report (VPAT® 2.5, OpenACR), it appears on a public page that keeps its version history, and its link keeps working after a report is withdrawn.

RechecksAfter a change; recordings on an enabled schedule

Recorded accessibility tasks can replay when recording and the plan’s schedule are enabled. Availability and cadence depend on the current setup and plan. A change is reported, never silent, and a report row whose evidence came only from a changed recording is marked stale until a person decides it again. After a change to your product, a recheck looks for an issue’s condition again and records whether it was seen again, not seen, not checked or couldn’t tell, and why. It never marks an issue fixed.

LearningCounted across customers; approved by a person

When a person marks an issue Working as intended, they can say why: the check is wrong, it’s on purpose, or it doesn’t apply. Each week those choices are counted by check, criterion and element type across companies. Screens, notes, page text, company names and a company’s own rules are never counted or shared. A pattern people at three or more companies keep calling wrong becomes a proposed example the checks and the AI are measured against, and a person at Variantly reviews it before it is used.

Not coveredNamed, not assumed

Whether real people hit what an attempt found. Real screen readers: the transcript is derived from the capture. Pages or tasks you didn’t add. Anything a test account can’t reach. A clean scan does not establish conformance.

Use the finding to make the next decision.

Inspect the evidence, change what needs changing, and check with people when the question is about their experience.

A review can show a captured failure condition and suggest a question worth investigating. It cannot measure how often customers encounter it, predict conversion, or establish people’s comprehension, effort or satisfaction. Those questions need research with people and data about actual use.

An accessibility report needs stated coverage and human judgment over its evidence. WCAG-EM is explicit that evaluating a selected sample alone does not establish conformance for an entire website. Unvisited pages, unreachable states and real assistive-technology behavior remain separate work. [6]

These sources inform our design choices. They do not validate Variantly’s accuracy or show that its attempts are equivalent to research with real users.

How each report row is decided · How to add recordings

How the rule checks score on known cases.

We run the engine’s rule checks, with no AI and no person, on the public ACT Rules test cases. Each case is a small page with a known outcome.

The cases the checks were built against

ACT rules a check can decide
203 of 212 (96%, 92–98%) failing examples caught
ACT rules that need a person or the AI
1 of 74 failing examples caught by the checks, 19 more sent to review
Passing examples flagged
1 of 235 (0%, 0–2%)
Run
12 October 2026
Engine commit
9f2624c1785a
axe-core
4.13.0
Chromium
153.0.8010.12
Case set commit
f39b7f81cee6
testcases.json SHA-256
49051da603afa6d28b94ef3e758146dafcbe72bade7202a24636bab3ea4c0036
Cases
876
CriterionHow the row is setFailing examples caughtPassing examples flaggedNot run
1.1.1 Non-text ContentAI proposed18 of 26 (69%, 50–83%)0 of 19 (0%, 0–17%)0
1.2.1 Audio-only and Video-only (Prerecorded)AI proposed0 of 6 (0%, 0–39%)0 of 00
1.2.2 Captions (Prerecorded)AI proposed0 of 2 (0%, 0–66%)0 of 2 (0%, 0–66%)0
1.2.3 Audio Description or Media Alternative (Prerecorded)AI proposed0 of 4 (0%, 0–49%)0 of 00
1.2.5 Audio Description (Prerecorded)AI proposed0 of 3 (0%, 0–56%)0 of 00
1.3.1 Info and RelationshipsAI proposed21 of 21 (100%, 85–100%)0 of 30 (0%, 0–11%)0
1.3.3 Sensory CharacteristicsNeeds a person0 of 4 (0%, 0–49%)0 of 00
1.3.4 OrientationRule checked4 of 4 (100%, 51–100%)0 of 3 (0%, 0–56%)0
1.3.5 Identify Input PurposeAI proposed10 of 10 (100%, 72–100%)0 of 9 (0%, 0–30%)0
1.4.2 Audio ControlAI proposed2 of 2 (100%, 34–100%)0 of 3 (0%, 0–56%)0
1.4.3 Contrast (Minimum)Rule checked11 of 11 (100%, 74–100%)0 of 12 (0%, 0–24%)0
1.4.4 Resize textRule checked12 of 12 (100%, 76–100%)0 of 9 (0%, 0–30%)0
1.4.5 Images of TextAI proposed0 of 5 (0%, 0–43%)0 of 00
1.4.12 Text SpacingRule checked14 of 14 (100%, 78–100%)0 of 20 (0%, 0–16%)0
2.1.1 KeyboardRule checked3 of 3 (100%, 44–100%)0 of 6 (0%, 0–39%)0
2.1.2 No Keyboard TrapAI proposed5 of 5 (100%, 57–100%)1 of 7 (14%, 3–51%)0
2.1.4 Character Key ShortcutsAI proposed0 of 2 (0%, 0–66%)0 of 00
2.2.1 Timing AdjustableNeeds a person4 of 4 (100%, 51–100%)0 of 3 (0%, 0–56%)0
2.2.2 Pause, Stop, HideAI proposed0 of 1 (0%, 0–79%)0 of 00
2.4.1 Bypass BlocksAI proposed0 of 1 (0%, 0–79%)0 of 12 (0%, 0–24%)0
2.4.2 Page TitledRule checked6 of 9 (67%, 35–88%)0 of 6 (0%, 0–39%)0
2.4.4 Link Purpose (In Context)AI proposed11 of 25 (44%, 27–63%)0 of 11 (0%, 0–26%)0
2.4.6 Headings and LabelsAI proposed0 of 10 (0%, 0–28%)0 of 00
2.4.7 Focus VisibleRule checked1 of 1 (100%, 21–100%)0 of 00
2.5.3 Label in NameRule checked16 of 16 (100%, 81–100%)0 of 16 (0%, 0–19%)0
2.5.4 Motion ActuationAI proposed0 of 2 (0%, 0–66%)0 of 00
3.1.1 Language of PageRule checked8 of 13 (62%, 36–82%)0 of 3 (0%, 0–56%)1
3.1.2 Language of PartsRule checked9 of 13 (69%, 42–87%)0 of 5 (0%, 0–43%)0
3.3.1 Error IdentificationAI proposed0 of 5 (0%, 0–43%)0 of 00
4.1.2 Name, Role, ValueAI proposed49 of 52 (94%, 84–98%)0 of 59 (0%, 0–6%)0
All criteria204 of 286 (71%, 66–76%)1 of 235 (0%, 0–2%)1

Percentages carry a 95% Wilson interval. A case counts once for each criterion it maps to, so the totals count some cases more than once. Per-case results (CSV): each case’s ID, ACT rule, expected outcome, the criteria the engine flagged, and how it scored.

The checks were developed against these cases, so this run shows the most they catch, not what they catch on pages they haven’t seen.

Repeat the run

You can fetch the same cases and check them against the per-case results:

git clone https://github.com/w3c/wcag-act-rules
git -C wcag-act-rules checkout f39b7f81cee60efe1f23205a893e031a7a02821b
shasum -a 256 wcag-act-rules/content-assets/wcag-act-rules/testcases.json

The hash matches the one above when you have the same cases. Each case’s page and expected outcome are in that file, under its ID.

You can’t run our engine: it isn’t public. Our CI runs it with pnpm benchmark:act, which writes the figures on this page and the results file, and holds every change to the checks to them.

Read the original work.

The sources cited above come first. Further reading adds context on browser agents and repeated attempts. Research papers and accessibility standards answer different questions; each has its own setting and limits.

Research cited in the method

  1. UICrit: Enhancing Automated Design Evaluation with a UI Critique Dataset

    Duan et al. · UIST 2024 · UI critique dataset

    Task context and localized design feedback. Its single-screen study does not establish how people use a complete task.

  2. TaskAudit: Detecting Functiona11ity Errors in Mobile Apps via Agentic Task Execution

    Zhong et al. · CHI 2026 · Mobile accessibility research

    Interaction traces reveal barriers beyond static checks in the studied apps. Its screen-reader proxy and task selection have limits; the study does not establish web conformance.

  3. Large language models that replace human participants can harmfully misportray and flatten identity groups

    Wang, Morgenstern and Dickerson · Nature Machine Intelligence · 2025

    Identity-conditioned responses can misrepresent groups and their diversity. The study concerns the tested models and questions, not every possible use of a perspective.

  4. AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories

    Lù et al. · 2025 · Version 2 preprint

    Compares automated judgments of browser attempts against expert annotations. Both model judges and benchmark rules have limitations; its results do not validate Variantly.

Accessibility standards and guidance

These define requirements, testing structure and report coverage. They are standards and guidance, not studies of agent accuracy.

  1. Accessibility Conformance Testing (ACT) Rules Format 1.1

    W3C Recommendation · 5 February 2026

    Defines applicability, expectations and outcomes for checks. A rule can cover only part of a criterion, so a pass can still require further testing.

  2. WCAG-EM 2.0

    W3C Group Note · 23 July 2026

    Guidance for coverage, evidence and reporting. Evaluating a sample alone cannot establish whole-site conformance.

  3. Web Content Accessibility Guidelines (WCAG) 2.2: Conformance

    W3C Recommendation · 12 December 2024 · Section 5

    Read the requirements for full pages, complete processes and accessibility-supported technology. A finding about one captured state is narrower than a conformance claim.

Further reading: browser agents and consistency

Complementary work for a deeper look at task execution, reviewing agent output and what repeated runs can tell you.

  1. WebArena: A Realistic Web Environment for Building Autonomous Agents

    Zhou et al. · ICLR 2024 · Conference paper

    See how web tasks are defined by an intended result and checked against the resulting state. This is a controlled agent benchmark, not evidence of customer experience on a live product.

  2. UXAgent: An LLM-Agent-Based Usability Testing Framework for Web Design

    Lu et al. · CHI 2025 Extended Abstracts · Conference companion paper

    A browser-agent workflow for rehearsing study designs before involving people, with traces and recordings for researchers to inspect. Its researcher study also reports concerns about bias and unrealistic behavior.

  3. UXCascade: Scalable Usability Testing with Simulated User Agents

    Holter et al. · 2026 · Version 2 preprint · UIST conference forthcoming, November 2026

    A workflow linking goals, interaction traces and proposed issues. Its study uses a seeded site and a deliberately minimal human baseline; the recheck tests one step for one agent. It does not establish general equivalence to human research or a fix’s downstream effects.

  4. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

    Yao et al. · 2024 · Preprint

    Separates success in one trial from consistency across repeated trials. Its API-based customer-service tasks and simulated conversations do not measure web usability or Variantly’s repeatability.

Start with what you meant.

Describe one task and how you’d know it worked.