Skip to content

Learn

What AI agents do with the person you describe: our research

Some research tools now put an AI agent through your product “as” a described person. We ran three preregistered studies to see what those agents do with the description. Here is what we found, and what it means if you use them.

Research

What we ran

5,388 attempts by AI web agents in a real browser, on one synthetic small-business accounting app, across six models from two providers: Claude Opus 5.5, Sonnet 5.5 and Haiku 4.5, and GPT-6-Astra, GPT-6.1-Sol and GPT-6-Luna. Each case was a choice the page left open, such as inviting a bookkeeper with full access or view-only access. Each agent got no persona, one of two opposed preferences, or a persona unrelated to the choice, such as “Grows tomatoes on weekends”. Half the cases were written by a model family blind to our earlier studies.

1. A stated preference decides the choice

On all six models, a preference like “avoids choices that can’t be undone” decided which option the agent took. It did so completely (3 of 3 runs against 0 of 3) in 174 of 192 model-and-case pairs. It held when we reworded the preference to share no words with the options, so the agents weren’t matching keywords. No case moved against the prediction on any model.

What it means: a persona agent turns the preference you write into a choice, reliably. That makes it a useful probe of which way a preference pushes a choice your page leaves open. The output reflects what you wrote. It is no evidence of what your users would do.

2. Text that shouldn’t matter can change the result

An unrelated line, like the tomatoes, changed the agent’s most common choice in up to 44% of cases on one model, and in none on another. Across five models, unrelated personas shifted held choices about two to three times as often as run-to-run noise.

What it means: one run of one persona proves little. Repeat each persona with a different irrelevant background line, and report a result only if it holds, for example in at least 2 of 3 runs. On our cases, that rule kept nearly every real preference effect, but kept irrelevant text’s effects under 10% on only three of the six models. Check the rule on your own model and screens.

3. A persona makes the agent more consistent

With a preference, 89–100% of model-and-case pairs gave the same choice in every run, against 63–94% with no persona. Agents varied less with a preference than without one. These studies did not measure how much people with that preference vary on the same cases.

What it means: one persona repeated is not a population. Never report persona results as a share of users, such as “40% of hurried users would…”.

4. Agents follow facts they’re given

Told where a setting lives, all six models went to it. That shows agents follow navigational facts. It doesn’t show that they model what a person who knows the product would do.

5. An agent’s explanation isn’t evidence

In 147 of 150 runs, the agent’s notes credited the persona for its choice. Our prompt asked agents to name the trait they acted on, and in some runs the notes credited the preference while the choice went against it. Read notes beside the action they explain, never instead of it.

An early look at real people

A fourth, preregistered study compares two of the models with 300 people from a public dataset of survey and economic choices. It is paused partway, and these results are interim: we scored them before the full sample was in, which departs from the preregistration.

  • A one-sentence persona exaggerated differences between people. On risk, people’s most and least risk-taking thirds differed by 0.49; the agents’ differed by 0.78 and 1.00.
  • A persona built from each person’s own earlier answers came much closer. On risk and time, it about tied a simple rule that uses no model at all.
  • Demographics alone made almost no difference.

What to do with it

  • Separate what an agent observed under conditions you enforce, such as a signed-in role or keyboard-only use, from what it chose because of a described preference.
  • Report a preference’s result only when it holds across varied repeats, show the count, and never turn it into a share of users.
  • Make no claim about people without evidence from people. Take extra care with personas that stand for disabled or marginalized people.
  • Re-check whenever the model, the prompt or the screens change.

Limits, and who wrote this

The first three studies had no people in them. The expected choice in each case is ours, not observed behavior. Preferences were single, explicit trade-offs on one screen, on one synthetic app, in cases written by models. The paper is a draft and hasn’t been peer reviewed. Variantly builds a product that runs these agents. That is a conflict of interest, and the reason we publish the limits with the results.

Stankiewicz, A. (2026). The Persona Is the Prompt: Synthetic Users Act on What a Persona Says, Even the Parts That Shouldn’t Matter. Draft, October 2026.

See it on a product