Post
WebArena: Why Does GPT-4 Achieve Only a 14.41% Success Rate on Realistic Web Tasks?
As large language models become more capable, a natural question arises: can we simply tell an AI what we want and let it open websites, search for information, click buttons, and complete the task on its own?
Examples include finding when I last bought shampoo, locating the highest-rated ergonomic chair, publishing a post on Reddit, checking the GitLab merge requests assigned to me, or planning a route across several locations and writing the result into a GitLab README.
Systems that perform these kinds of tasks are commonly referred to as Web Agents, GUI Agents, or Computer-Use Agents.
Before WebArena, however, there was a major evaluation problem: we did not really have a sufficiently realistic environment for testing whether these agents could actually use the web.
The ICLR 2024 paper WebArena: A Realistic Web Environment for Building Autonomous Agents addresses this problem. Instead of proposing a new agent algorithm, it does something more fundamental:
It builds a realistic and reproducible “training ground” and “exam environment” for Web Agents.
1. What Problem Is WebArena Trying to Solve?
The authors argue that many earlier agent benchmarks suffer from three major limitations: the environments are too simple, the tasks are too short, and the evaluation criteria are often too rigid.
Many web benchmarks use small, artificially constructed websites. A task might simply ask the agent to click a button, search for a product, or add something to a shopping cart. These tasks can test basic web interaction, but they do not capture the complexity of real websites, which often contain many pages, search and filtering interfaces, user histories, permissions, dynamic data, complicated navigation structures, and multiple valid ways to achieve the same goal.
As a result, a high success rate on a simplified webpage does not necessarily mean that an agent can reliably operate realistic systems such as Amazon, Reddit, or GitLab.
The second issue is task length. Real-world tasks are rarely completed in one or two actions. For example:
Find all art museums in Pittsburgh, design an efficient route starting from Schenley Park that minimizes driving distance, and record the visiting order in my GitLab repository.
To complete such a task, an agent may need to use Wikipedia to identify museums, search their locations on a map, compare routes, open GitLab, find the correct repository, edit the README, and save the result.
This is a long-horizon task. Its difficulty comes not only from the number of steps, but from the requirement to continuously remember the original goal, intermediate results, and previously completed actions.
The third problem is evaluation. Many earlier benchmarks compare an agent's action trajectory against a human-annotated reference trajectory. For example:
Click A
Click B
Type C
Click D
But realistic web tasks often have multiple valid solutions. An agent might perform an extra search or take a different navigation path and still complete the task correctly.
WebArena therefore emphasizes Functional Correctness:
Do not judge whether the agent followed the same path as the reference. Judge whether the task was actually completed.
If the task is to create a GitLab repository and write specific information into its README, the evaluator checks whether the repository exists and whether the README contains the correct content. The exact action sequence is secondary.
2. How Can an Environment Be Both Realistic and Reproducible?
The most realistic approach would be to let agents directly operate live websites such as Amazon, Reddit, GitLab, or Google Maps. However, this creates major problems for scientific evaluation.
Real websites constantly change. User interfaces are updated, products disappear, content changes, account states evolve, CAPTCHAs may appear, and automated agents may be blocked. If Agent A is evaluated today and Agent B is evaluated several months later, they may no longer be interacting with the same environment.
In short:
Live websites are realistic, but difficult to reproduce.
The opposite approach is to build small toy websites. These are easy to reproduce but do not reflect real-world complexity.
WebArena resolves this trade-off by using open-source software underlying real websites, hosting local copies, and populating them with realistic data.
| Category | WebArena Implementation |
|---|---|
| E-commerce | Magento |
| Social forum | Reddit-like platform |
| Software collaboration | GitLab |
| E-commerce administration | Magento CMS |
| Maps | OpenStreetMap |
| Knowledge base | Wikipedia |
| Utilities | Calculator, Scratchpad |
These websites are deployed through Docker and can be reset to the same initial state. This is important because many tasks modify the environment. If one agent deletes a post, creates a repository, or changes a product, the next agent should not inherit those modifications.
The core design philosophy is therefore:
Realistic enough to resemble real web applications, but controlled enough to support reproducible evaluation.
3. How Does an Agent Interact with WebArena?
The paper formalizes WebArena as:
Here, State represents the true state of the environment, such as the contents of a shopping cart, the repositories in GitLab, or posts stored in a database.
Observation represents what the agent can currently see, such as the current URL, webpage content, and open browser tabs.
Action represents the operations available to the agent:
click
type
scroll
hover
new_tab
tab_focus
goto
go_back
Transition describes how an action changes the environment:
Shopping cart is empty
↓
click "Add to Cart"
↓
Product appears in shopping cart
WebArena is therefore not a static collection of webpages. It is an interactive environment following the loop:
Observe → Act → Environment Changes → Observe Again
4. What Does the Agent Actually See?
WebArena supports three main webpage representations: screenshots, HTML/DOM, and accessibility trees.
A screenshot is closest to what a human sees. HTML provides rich structural information but can be extremely long. The baseline agents in the paper mainly use the accessibility tree, which preserves important semantic information while remaining relatively compact.
For example:
link 'Outdoor Patio'
StaticText 'Rating:'
StaticText '82%'
button 'Add to Cart'
Each interactive element can also be assigned an ID:
[1582] button 'Add to Cart'
The agent can then issue:
click [1582]
This design reduces the burden of pixel-level coordinate prediction and focuses the experiment more directly on whether the agent understands what it should do next.
WebArena also supports multiple browser tabs. This matters because realistic tasks often require comparing information across websites, using a map while reading another page, or moving between a knowledge source and a target application.
5. How Were the 812 Tasks Designed?
WebArena contains 241 task templates and 812 instantiated tasks. The authors require the tasks to be high-level, creative, and template-based.
High-level tasks should not directly specify the next action. Instead of asking the agent to “click the Science subreddit,” the benchmark might ask it to “post a greeting message in the Science subreddit.” The agent must infer the required sequence of actions.
Tasks are also designed to include meaningful constraints. For example:
Create a Reddit account with the same information as my GitLab account.
This requires the agent to inspect one website, remember relevant information, move to another website, and recreate it there.
Finally, tasks are represented as templates. For example:
Create a {{site1}} account identical to my {{site2}} one.
Different websites can be substituted into the template.
This is important because it allows researchers to test whether an agent has actually learned a reusable procedure or merely succeeded on one particular instance.
6. What Types of Capabilities Does WebArena Test?
The benchmark mainly divides tasks into three categories.
Information Seeking tasks require the agent to return a textual answer. For example:
When was the last time I bought shampoo?
Unlike ordinary question answering, the answer may be hidden inside a user's private order history, requiring actual website navigation.
Site Navigation tasks require the agent to locate a specific page or piece of content, such as:
Show me the merge requests assigned to me.
The agent must understand the site's structure and use the appropriate search, links, or filters.
Content & Configuration tasks require the agent to change the environment, for example by publishing a Reddit post, editing a README, deleting a review, or changing a setting.
These tasks are particularly important because success cannot be determined by what the agent claims to have done. The environment itself must actually change.
7. How Does WebArena Evaluate Success?
For information-seeking tasks, WebArena uses three main matching strategies.
Exact Match requires the answer to match the reference exactly.
Must Include checks whether required pieces of information are present.
Fuzzy Match uses GPT-4 to determine whether two answers are semantically equivalent. For example:
2h58min
and:
2 hours and 58 minutes
should be treated as equivalent.
For tasks that modify websites, evaluation is more direct. The evaluator inspects the environment state.
Suppose the task is:
Post a message containing “a car in NYC” in the NYC subreddit.
The evaluator can check whether the latest post is actually in the NYC subreddit and whether its body contains the required text.
This captures the central philosophy of WebArena:
The agent saying “Done” is not enough. The requested state change must really exist.
8. Unachievable Tasks
WebArena also intentionally includes tasks that cannot be completed.
For example, a user might ask for contact information that the website simply does not provide. In such cases, the correct behavior is not to fabricate an answer but to respond that the task is unavailable or return N/A.
This evaluates whether an agent can recognize when a task is impossible rather than hallucinating a result.
However, this design also reveals another problem: an agent may become too conservative and give up on tasks that are actually achievable.
9. How Well Does GPT-4 Perform?
The authors evaluate TEXT-BISON-001, GPT-3.5, and GPT-4. They also compare direct action prediction against Chain-of-Thought reasoning before action selection.
| Agent | Success Rate |
|---|---|
| TEXT-BISON-001 + CoT | 5.05% |
| GPT-3.5 Direct | 6.41% |
| GPT-3.5 + CoT | 8.75% |
| GPT-4 + CoT | 11.70% |
| Human | 78.24% |
Chain-of-Thought helps. GPT-3.5 improves from 6.41% to 8.75%. However, the improvement is modest compared with the gap between agents and humans.
The most important result is therefore not simply that GPT-4 performs better than GPT-3.5. It is that even GPT-4 remains far below human performance on realistic long-horizon web tasks.
10. Why Does the Paper Also Report 14.41% for GPT-4?
The 11.70% GPT-4 configuration includes an instruction telling the model:
If you believe the task is impossible to complete, stop and return N/A.
The paper calls this the Unachievable Hint, or UA Hint.
The authors find that this instruction makes GPT-4 overly conservative. It incorrectly classifies 54.9% of achievable tasks as impossible.
When the UA Hint is removed, GPT-4's overall success rate rises from:
11.70% → 14.41%
This is the 14.41% figure highlighted in the paper's abstract.
However, removing the hint also reduces the model's ability to recognize genuinely unachievable tasks.
This reveals an important trade-off:
An agent that is too aggressive may hallucinate, while an agent that is too conservative may stop too early.
11. The Agent Does Not Learn Stable Skills
The authors also examine different task instances generated from the same template.
In principle, if an agent genuinely understands how to “fork a repository,” changing the repository name should not completely change its ability to perform the task.
The results show otherwise.
Among 61 templates with at least one successful execution, GPT-4 achieves a 100% success rate on only four templates. GPT-3.5 achieves a 100% success rate on none of them.
This suggests that success is often highly instance-specific:
The agent may solve one task variation without learning a stable, reusable procedure.
The paper therefore points toward ideas such as memory and reuse of successful strategies.
From today's perspective, this connects naturally to an important GUI Agent research direction:
Long-horizon Memory + Skill Learning
12. Why Does GPT-4 Fail?
The paper's error analysis reveals several recurring failure modes.
One is Observation Bias. The agent often latches onto the first piece of information that appears relevant rather than actively verifying whether it satisfies the user's full request.
For example, if the task asks for the top-selling product in 2022, the homepage may show recent best sellers. The correct solution requires opening a historical report and selecting 2022, but GPT-4 may simply use the visible homepage information.
This points to weaknesses in active exploration and verification.
Another problem is fine-grained observation interpretation. A page might show:
Projects 0
Users 1
The correct reasoning could be that there is no matching project but there is a matching user, so the agent should inspect the Users tab. Instead, the model may incorrectly conclude that the search failed.
The agent may also repeat actions even when the observation clearly indicates that the previous action has already taken effect. For example:
DMV area
may already be present in a search box, yet the agent repeatedly types the same query until the step limit is reached.
This exposes weaknesses in state tracking and the use of interaction history.
13. What Research Problems Does WebArena Really Expose?
The most important contribution of WebArena is not simply the number 14.41%. Its deeper value is showing that strong language-model capabilities do not automatically translate into reliable agent capabilities.
A robust long-horizon GUI Agent still needs to solve several major problems.
Planning: How should a complex goal be decomposed into subgoals and executed over many steps?
Memory: After dozens of actions, how does the agent remember the user's original goal, what has already been completed, and what information has already been discovered?
Exploration: When the current page does not contain the answer, does the agent know that it should continue searching rather than immediately respond?
Verification: When the agent finds something that appears relevant, does it verify that the information actually satisfies all constraints?
Failure Recovery: After a wrong click, failed search, or unexpected page transition, can the agent recognize the failure and change strategy?
Skill Reuse: If the agent has successfully completed a procedure before, can it extract that procedure into a reusable skill rather than reasoning from scratch every time?
These are the problems that turn a system from a one-step interface predictor into a genuinely autonomous agent.
14. Why Is WebArena Still Important for GUI Agent Research?
WebArena represents an important shift in the way Web Agents are evaluated.
The question is no longer merely:
“Can an LLM click the correct button?”
It becomes:
“Can an LLM reliably complete a realistic task over a long interaction horizon?”
A model may be very good at understanding webpage text, choosing the correct button for a single step, or reasoning about the current screen. But a reliable autonomous agent must also plan, remember, explore, verify, recover from mistakes, and preserve task progress over time.
WebArena demonstrates a key distinction:
Single-step intelligence is not the same as long-horizon agency.
That is arguably the most important takeaway from the paper.
Conclusion
WebArena makes three main contributions:
- It builds a realistic yet reproducible environment for Web Agents.
- It introduces 812 complex, long-horizon tasks inspired by realistic web usage.
- It evaluates functional correctness rather than requiring agents to imitate a fixed reference trajectory.
The paper finds that GPT-4 achieves a best success rate of only 14.41%, compared with 78.24% for humans.
The result suggests that although LLMs already have strong local language understanding and reasoning abilities, they remain far from being reliable autonomous agents.
The central challenges exposed by WebArena—
Planning, Memory, Exploration, Verification, Failure Recovery, and Skill Reuse
—remain core problems in GUI Agent and Computer-Use Agent research.
Paper
WebArena: A Realistic Web Environment for Building Autonomous Agents
Shuyan Zhou, Frank F. Xu, Hao Zhu, et al.
ICLR 2024