Skip to content

Learn

Why people take wrong turns, and what the words predict: our research

People find their way by the words on a screen: they choose the label that reads closest to what they want. We checked whether a score built only from the goal and a screen’s labels predicts where people go wrong, on human navigation and on real websites. Here is what we found, and how task reviews use it.

Research

The idea: information scent

Information foraging theory (Pirolli and Card, 1999) describes people following “scent”: the cues in a link’s words that suggest the goal is that way. The Cognitive Walkthrough for the Web (Blackmon and others, 2002) used it to predict where visitors choose the wrong heading. When a wrong label reads closer to the goal than the right one, people tend to follow it.

What we ran

Two public datasets, with the method fixed before each run and nothing tuned between runs. Wikispeedia (West and Leskovec, 2012) records people clicking from one Wikipedia article to a target article: we scored 396,465 of their decisions, wrong clicks and backtracks included. Multimodal-Mind2Web (Deng and others, 2023) records experts completing tasks on real websites: we scored 717 of their clicks, on 47 sites, all of them correct.

At each step, a small language model compared each link’s words with the goal. We measured how much closer the best wrong link read to the goal than the right one.

1. The words predict wrong turns in human navigation

Where a wrong link read closer to the goal than the right one, people went wrong at 71% of steps, against 36% elsewhere. Where a wrong link read much closer, in the strongest tenth of steps, 87% of moves were wrong. The score added to what the page’s structure alone predicted, such as how many links it had. It didn’t predict giving up.

A 30 MB model that only looks up each word’s meaning came close to the sentence model. Plain word vectors carried little of the signal.

2. As a rule for every step, it raises false alarms on real sites

On Mind2Web, the same rule with the 30 MB model flagged 51% of correct, uneventful clicks, and 44% counting only links, menu items and tabs. On a real site, the control a person needs often serves a step on the way, like “Quick Tools” on the way to booking a passport appointment, or a plain “Continue”. Its words rarely match the whole task, even when the step is obvious. Every Wikipedia link names a topic; most product controls don’t.

What it means: scoring every screen against the task would mostly flag screens that work.

How task reviews use it

So we don’t score every step. In a task review, an attempt sometimes leaves a screen by one control, comes back and finishes by another, or switches from one tab to another. The wording check then compares both controls’ words with the task. When the control taken first reads at least as close to the task, the review raises a “Misleading wording” issue that names both controls, the screen and both scores.

The check is a word-similarity measurement run on our own servers. No AI model judges it. An attempt isn’t a person: one wrong turn shows the wording can mislead, not how often it does. A person decides each issue, and a recheck after a change shows whether a fresh attempt takes the same wrong turn.

Limits

  • Wikispeedia’s links are Wikipedia text, not product screens, and “wrong” means off the shortest path. A sensible detour through a general article counts as wrong.
  • The sentence model was trained partly on text from Wikipedia, which may help it there.
  • Mind2Web’s experts explored each site before recording, so its false-alarm rates are an upper bound.
  • We found no public dataset of people’s wrong turns on product screens, and haven’t yet measured the wording check against people using a product.

Pirolli, P., and Card, S. (1999). Information foraging. Psychological Review. Blackmon, M., Polson, P., Kitajima, M., and Lewis, C. (2002). Cognitive walkthrough for the Web. CHI. West, R., and Leskovec, J. (2012). Human wayfinding in information networks. WWW. Deng, X., and others (2023). Mind2Web: Towards a generalist agent for the web. NeurIPS.

See it on a product