Every Product Is Designed Twice

By Adam Stankiewicz, founder of Variantly · A Variantly white paper · 12 October 2026

Variantly sells a product that runs this review. We say where every figure comes from, and where a figure is still missing.

A worked example: the invitation confirmation

Executive summary

Making software got cheap. Knowing it works for people didn’t. A designer can prototype in code in an afternoon, and a coding agent can build a flow from one sentence, but the review that tells you whether a person can finish the task still runs at human speed: a study each quarter, an audit each year. We call this the verification gap.

It has a simple shape. Every product is designed twice. The first design is what the team meant, in a file, a prototype, a spec or a prompt. The second is the one people use, made by everything on the way to production: real data, states nobody drew, copy edited in a pull request, and thousands of small decisions by whoever, or whatever, wrote the code. Teams review the first design carefully and the second in pieces, with design, research, accessibility and code review each keeping their own record. None of them checks the shipped product against what was meant, and the problems that stop people happen inside tasks, where page scans don’t look.

This paper sets out one review of the second design, shared by everyone who owns how a product works. We call it the second-design review:

1. Intent. Write the task, and how you’d know it worked.
2. Walk through. Work through the task on the shipped product, as the people who’ll use it would.
3. Capture. Record every state, and judge usability and accessibility at equal weight on the same screens.
4. Decide. A person decides each issue with the evidence open.
5. Check again. After a change, walk through it again and record what you saw.

It needs no particular tool. A team can run it by hand, with people, or with AI agents doing the walking, which makes it cheap enough for every release. The AI doesn’t stand in for the people you design for, and it doesn’t make the call.

Where to start in the full paper. If you lead design or product, read chapters 1 to 3 and 7. If you research with people, read 3 to 5. If you ship the code, with or without an agent, read 3, 8 and 9. If you answer buyers’ accessibility questions, read 2 and 5, then the companion paper, Accessibility You Can Show.

Principles

The rest of this paper argues for these eight principles.

1. The product people use is the design. Whoever or whatever made it, that’s the version that counts.
2. Write down what you meant. Before the change, in words anyone could check. It is the spec, for people and agents alike.
3. Review the experience, not the artifact. Design, research, accessibility and engineering look at the same thing: the product as it ships.
4. Accessibility is usability. Same screens, same weight, same list.
5. Show the evidence. A problem comes with the screens that prove it, and a review says what it didn’t look at.
6. Machines find. People judge. AI can explore, repeat and flag. It doesn’t stand in for the people you design for, and it doesn’t make the call.
7. Whoever builds it should see it. Designers who code, and the agents that write code, get the same review as everyone else.
8. Check again after every change. The second design keeps changing, so the review keeps running.

1. The second design

Take a small, ordinary flow: an administrator invites a teammate. This is a redrawn example, composed from the kinds of changes we see in reviews.

In the design, it is clear. The admin types an email address, picks a role and presses Invite. A confirmation says the invitation went to sam@example.com as a Member. The teammate appears in the list as Pending.

Six months later, the shipped version differs in four places, and nobody decided any of them.

• The confirmation is now a toast that disappears after three seconds. An admin who glanced away never sees it, and nothing marks it as a status message for a screen reader.
• The role picker defaults to Admin, because an engineer changed the default while fixing something unrelated.
• On a slow connection, pressing Invite twice sends two invitations.
• On a phone, the Invite button sits below a keyboard that covers it.

None of these is a mistake on its own. Each came from a reasonable change made by someone doing their job. Together they mean the screen people use is a second design that nobody reviewed.

The places a first design never shows (real data, a slow connection, the back button, a keyboard, a screen reader) are where people get stuck. They are also where accessibility fails, the first sign that usability and accessibility are one problem seen from two sides.

The drift grows with every release, and a review that happens once a quarter falls further behind each one.

When designers code and agents build

Two changes might seem to end the drift: designers and product managers prototype in code, and coding agents build interfaces straight from a prompt. Both make the second design more important.

• A coded prototype is still a first design. It shows the path its author walked, with sample data, on a fast laptop, signed in as the right user. Production adds every path the author didn’t walk. It says nothing about the error state, the second click or the screen reader.
• An agent is a designer nobody reviews. Asked to “add an invite flow”, it decides the default role, the confirmation, the error copy, the focus order and what happens on a double click. Each looks plausible. The prompt was the first design; the agent made the second, with more design decisions per week than any team has time to look at.
• More people shipping means more seams. When a designer, a product manager and an agent all change the same task, no one person holds the whole of what was meant.

This is the verification gap: the second design is made faster and by more hands than before, and reviewing it hasn’t sped up.

Automated checks catch part of it. WebAIM found detectable WCAG failures on 95.9% of the top million home pages in 2026, up from 94.8% in 2025, and that counts only what a machine can find on a page that loads without signing in. The vanishing confirmation, the changed default and the hidden button happen inside a task, between states, where a page scan doesn’t look.

2. Four disciplines, four partial views

Four practices care whether a product works for the people using it, and each sees part of the second design.

Each practice works. The trouble is in the space between them.

They look at different things at different times. Engineering reviews every change, but it reviews the code, not the experience; the other three look rarely. On a product that ships weekly, no release gets all four.

They keep different records, and trade off against each other. The same broken confirmation becomes a research note, an audit row and a design comment: three tickets with three owners, or none. Accessibility, reviewed late and apart, arrives as a defect list and gets treated as a cost, though many of its failures are usability failures some people meet first and hardest.

All four come down to one question. Can this kind of person finish this task on the product as it ships? When the question is shared, the review can be shared too.

3. Start from intent: the task, and how you’d know it worked

Every usability study, walkthrough and design review starts with a task: one thing a person needs to get done, and the sign that they did. Write both down before anyone looks at a screen.

Name what the person is trying to do, in their words. “Invite a teammate with the right access” is a task. “Use the Members page” is a tour of a screen. Keep it to one outcome; if the sentence needs “and”, it is two tasks.

Say where they start, and as whom. The same task is harder signed out from the home page than from a familiar dashboard. A new administrator on their first day and a member of a year meet different products.

Say how you’d know it worked. Write what a person could point to afterward. “Sam appears in Members as Pending, with the Member role, once” can be checked. “The invitation works” and “it feels intuitive” can’t. Say what would count as failure even if the task ended, such as two invitations. Prefer what survives a reload.

Write what you meant, screen by screen. One sentence per screen about what the person should understand or be able to do. “The admin knows the invitation was sent, and to whom.” An intent written this way survives a restyle, and it is what the shipped screen gets compared against.

Leave out the steps. A task that says “open Settings, then Members, then choose Invite” checks your instructions. Whether a person finds Members on their own is what the review is for.

A task written like this is the unit every discipline can share: intent to a designer, a study task to a researcher, an acceptance criterion to an engineer.

It is also the spec. Teams that build with coding agents increasingly write a spec first. A task with how you’d know it worked is that spec for the experience: an agent can build toward it, and anyone, person or agent, can check the result against it. Research on web agents defines tasks the same way: WebArena (Zhou et al., 2024) states each task by its intended result and checks the state the agent leaves behind.

Related guides: Write a task, and how you’d know it worked; Review what shipped against what you meant.

6. What AI can do here, and what it can’t

AI changes what this review costs. An AI agent can walk through a task from several perspectives on every release, capture every state and give a first reading of each. On the invitation, walkthroughs from three perspectives (a first-day admin on a phone, keyboard only, and a screen reader) would surface all four changes above in one run, including the toast a person stepping through by hand might never see. That is what we propose, not a measured result.

What it does well

• Works through tasks repeatably from defined perspectives, including behind a sign-in, and captures every state.
• Proposes issues on usability and accessibility, each tied to its screen and evidence.
• Repeats the same walkthrough after a change, so before and after compare.

What it doesn’t do

• It doesn’t stand in for people. An AI agent working through a task represents no group of people. That includes disabled people, whose experience no agent reproduces.
• It doesn’t measure people. It can’t tell you how many users would struggle, whether they would convert or how they feel. Those questions need research with people.
• It doesn’t decide. Every issue it raises is a proposal. A person decides it.
• Its explanations aren’t evidence. In our research, an agent’s notes credited its persona in 147 of 150 runs, including runs where its action went against that persona. We read what an agent did, with its notes beside the action, never instead of it.

What the research says

Research on simulated users sets the same limits. Researchers who tried UXAgent (Lu et al., 2025) for rehearsing usability studies raised concerns about bias and unrealistic behavior. AgentRewardBench (Lù et al., 2025) found limits in both model judges and rule-based checks of agents’ web attempts. τ-bench (Yao et al., 2024) showed that success on one try says little about the next. None of it validates any product, ours included. It says: use agents to find and repeat, measure them in public, and keep a person on the decision.

What this paper leaves to its companion

How accurate automated accessibility checks are, what a passing check can and can’t prove, and what a conformance report needs before a buyer can check it are the subject of the companion paper, Accessibility You Can Show.

8. Coding agents close the loop

Chapter 1 called the coding agent a designer nobody reviews. The fix isn’t to slow it down. It is to give the agent the same review everyone else uses, so it can see the second design it made.

Today an agent learns about a problem from a ticket, a failing test or a pasted screenshot. None carry what was meant, the screen people met or what the accessibility tree exposed, so the agent fixes the symptom it was told about.

The review needs an interface agents can use

Coding agents now reach outside tools through the Model Context Protocol (MCP), an open standard Anthropic introduced in November 2024 that the major coding agents support. Any review a team wants its agents to use should be reachable this way, and it should hold to four rules.

• The agent reads evidence, not summaries. Each issue comes with its task, the intent for the screen, where it happened, the element, the WCAG criterion if there is one, and how to tell when it’s gone.
• The agent can ask for a review. With a person’s permission, it can describe a task, start a review and ask for a check once its change is deployed. The loop runs at the agent’s speed, not the next audit’s.
• The agent never decides. It can say a change is ready to check. It can’t close an issue, mark something as working as intended or set a row in a conformance report. Access is read-only unless a person grants more.
• Page text is data, never instructions. Anything captured from a page could have been written by someone trying to steer the agent. A review hands it over as labeled, length-capped data.

The loop

A person writes the task and the intent. An agent makes the change. The task is walked through on a preview, and the issues go back to the agent with their evidence. The agent changes the code and asks for another check. A person reads what was seen and decides.

On the invitation, the agent reads the toast issue, its screens and the accessibility-tree reading with no status message. It replaces the toast with a confirmation that stays on screen with a status role, then asks for another check on the preview. The check records seen for both: the confirmation stays on screen, and the accessibility tree has it as a status message, “Invitation sent to sam@example.com as Member.” A designer opens the screens and closes the issue. This step is a redrawn example.

The natural place for this is the pull request: the same task walked through on the version before the change and the version after, so a change is checked against what was meant before it merges, not after a customer finds it. It is fair only when both sides run the same task, perspectives and rules; anything that can’t be lined up counts as not checked.

Designers and product managers who ship code get the same loop: whoever wrote the change, by hand or by prompt, sees the second design before people do.

9. Putting it into practice

You can run this review next week with what you have. This checklist works with any tools, or none.

Before the release

1. Pick the three tasks that matter most this release. Write each with where it starts, as whom, and how you’d know it worked.
2. Write one sentence of intent for each screen the task passes through.
3. Choose perspectives that change how someone gets through: at least one keyboard-only, one screen reader, and one narrow screen.

Back to the invitation

Six months ago, four things changed in a small flow and nobody decided any of them. With one review, the same flow is written down as a task, walked through as the people who use it would, captured at every state, decided by people who know what was meant, fixed by whoever or whatever writes the code, and checked again on the next release. The second design still gets made. Now someone looks at it.

How Variantly does this

Everything above works without us. Variantly exists for teams that want it to run on every release, without someone running each review by hand.

• Intent and attempt. You describe a task and how you’d know it worked. AI attempts run it on your live product or staging, from the perspectives you describe, and keep every state with its five records.
• Decide and recheck. A person on your team decides each issue. After a change, a recheck records what was seen.
• Coding agents. Claude Code, Cursor, Codex and others connect over MCP to read issues with their evidence, and, connected through your account, to start a review or request a recheck.
• The report. A conformance report on a public link, built from crawled pages and recorded tasks, never from AI attempts, where every row says how it was set.

Paste a URL and a task, free: we check your public pages for accessibility, and an AI attempt tries the task once as a first-time visitor, look-only.

Ship what you meant.

• Try it free on your product
• See an example review