A computer use agent is a model that operates software the way a person does: it reads a screenshot, decides where to click or what to type, and checks the next screenshot. It needs no API from the application it drives, which suits legacy desktop tools and internal portals. The vendors' own documentation is candid about the costs: latency, screenshot tokens and prompt injection from whatever is on screen. Below: the loop as the Claude, OpenAI and Gemini docs describe it (read on 4 November 2026), what the OSWorld benchmarks measure, where agents fail and when to use something else.
How a computer use agent works
Anthropic's documentation is explicit that "Claude doesn't directly connect to this environment." Your application receives the model's tool use requests, translates them into actions, captures the results such as screenshots, and returns them. Anthropic calls the repetition the "agent loop".[1] One iteration:
- Your code sends the task with the computer use tool enabled. Google's flow includes a screenshot; OpenAI notes the model's first call "may contain only a
screenshotaction."[4][3] - The model returns one or more actions: click at a coordinate, type, press a key, scroll, wait or take a screenshot.
- Your code performs them on a real or virtual machine and returns a new screenshot.
- The loop repeats until the model stops requesting actions or your code hits a limit. Anthropic's reference loop sets
max_iterationsto 10 by default.[1]
How it differs from ordinary tool calling
Mechanically, computer use is tool calling. Anthropic's tool is a client toolset: one {"type": "computer_toolset_20260801"} entry in tools gives Claude 17 member tools.[1] In ordinary tool calling the model fills in typed arguments defined by your JSON Schema, such as an order ID. Here the arguments are screen positions and keystrokes, and the result is an image. Our reading is that this moves validation out of your schema and into the model's perception: nothing in a coordinate pair tells your code whether the click hit the right button. Only the next screenshot does.
What each vendor documents
The APIs share the loop and differ in almost every name (pages read 4 November 2026).
| Item | Anthropic (Claude) | OpenAI | Google (Gemini) |
|---|---|---|---|
| Tool entry | computer_toolset_20260801 for Claude 5.5 and later, no beta header; the older computer_20251124 needs the computer-use-2025-11-24 header[1] | The computer tool in the Responses API; examples use gpt-6.1-sol[3] | computer_use, with a browser, mobile or desktop environment[4] |
| Actions | 17 members, including screenshot, zoom, five click types, scroll, type, key and wait[1] | click, double_click, drag, move, scroll, keypress, type, wait, screenshot[3] | Browser actions include click, type, scroll, drag_and_drop, hotkey and navigate; the mobile set includes open_app[4] |
| Coordinates | Pixel space of the screenshots you return[1] | Not stated on the page read[3] | A 0 to 999 scale that your client converts to the screen[4] |
| Several actions per turn | Run them in order and stop at the first failure[1] | An ordered actions array in each computer_call[3] | A function_result per call, with the current URL and a screenshot[4] |
| Built-in safeguard | Classifiers scan screenshots for potential prompt injections[1] | Pending API safety checks, which stop the guide's sample unless you add a human approval callback[3] | A per-action safety_decision such as require_confirmation; prompt injection detection is opt-in[4] |
| Status | Stable on the Claude API[1] | Not labelled beta or preview on the page read[3] | gemini-3.8-flash is the recommended model; only gemini-3-flash-preview is marked preview[4] |
Anthropic recommends 1024x768 or 1280x720 for desktop tasks and 1280x800 or 1366x768 for web applications, and says to "Avoid resolutions above 1920x1080"; small targets and detail lost in downscaling are listed among the usual reasons clicks miss.[1] OpenAI documents a second path in which the model "writes code that uses a library such as PyAutoGUI or Playwright to operate the interface", and states: "For GPT-6 Astra, we recommend code execution."[3]
Screenshots are the running cost. Anthropic's toolset adds about 4,500 input tokens per request, and each screenshot costs roughly 1,000 to 1,800 input tokens.[1] Our arithmetic: 50 screenshots put 50,000 to 90,000 image tokens into the conversation, which makes pruning old ones a context-management problem.
How computer use is measured: OSWorld and its successor
OSWorld (2024)
OSWorld, first posted in April 2024, has 369 tasks "defined and executed on Ubuntu", drawn from real web and desktop applications, file operations and multi-application workflows.[5] Each task has an instruction, "a setup configuration" and "a manually crafted evaluation script to check if the task is successfully executed".[5] Success means the machine ends in the right state, checked by code, not that the agent's final message sounds plausible. Thirty tasks (8.1%) are infeasible by design.[5]
Humans "can accomplish over 72.36% of the tasks", while "the best model achieves only 12.24% success": GPT-4 with accessibility-tree input, within a 15-step limit.[5] The annotators were "computer science major college students who possess basic software usage skills" who had not seen the tasks before.[5] Among 550 failed GPT-4V examples, more than 75% involved inaccurate mouse clicks, and in a 28-task test agents degraded when application windows were moved, shrunk or joined by irrelevant ones.[5]
OSWorld-Verified and OSWorld 2.0
On 2025-07-28 the project announced OSWorld-Verified, fixing community-reported examples and separating verified from self-reported leaderboard results.[6] The leaderboard did not load when we read it, so we quote no entries.
OSWorld 2.0, posted on 28 June 2026, responds to saturation. Its authors write that "the same frontier agents are saturated at 79–83% binary accuracy on OSWorld 1.0", note that Claude Opus 4.8 reaches 83.5% on OSWorld-Verified, and argue that such accuracy "overstates real progress".[7] The new set has 108 long-horizon workflows with a median human time of about 1.6 hours; Claude Opus 4.7 at maximum thinking averages 318 tool calls per task, against about 30 in OSWorld 1.0.[7] The primary metric is binary completion at 500 steps, alongside a checkpoint-based partial score.[7]
In the authors' runs, the best configuration (Claude Opus 4.8 with maximum thinking and batched tool calls) completes 20.6% of tasks, with a 54.8% partial score, while GPT-5.5 "is far more token-efficient yet plateaus near 13%".[7] Their diagnosis: "These failures are not about basic GUI control or coding." Agents "lose track of constraints, miss information that arrives mid-task, guess rather than ask the user, and skip verification".[7] The paper reports human completion time, not a human success rate.[7]
WebArena (2023), the browser-only counterpart, scores functional correctness on "fully functional websites from four common domains"; its best GPT-4-based agent reached 14.41%, against 78.24% for humans.[9]
Why the scores are hard to compare
Our reading is that a bare "OSWorld score" hides several settings:
- Version: the 369 original tasks, the Verified fixes, or the 108 OSWorld 2.0 workflows.[5][6][7]
- Budget and harness: 15 steps in the 2024 paper against 500 in OSWorld 2.0, plus thinking effort and batched calls.[5][7]
- Observation: screenshot, accessibility tree or set-of-marks.[5]
- Metric: one OSWorld 2.0 run is 20.6% binary or 54.8% partial.[7]
- Subset: the OSWorld 2.0 leaderboard offers "Full set" and "Offline set" views without defining them.[8]
- Who ran it: verified or self-reported.[6]
Launch tables we have covered label the row "OSWorld Verified" (Meta Muse Spark), "OSWorld-2.0 (offline, partial score)" (Gemini 4 Argon) and "OSWorld 2.1 (offline subset)" (Claude Haiku 5.5); the OSWorld 2.0 project page we read does not mention a 2.1.[8] The 72.36% human figure belongs to the 2024 tasks and annotators, so a later agent score above it does not show that an agent uses a computer better than a person. How to read an AI benchmark claim sets out the general method.
Where computer use agents fail
Anthropic's limitations list is the most detailed of the three:[1]
- Latency: it "might be too slow compared to regular human-directed computer actions"; the page suggests work where speed isn't critical, such as automated software testing.
- Coordinates and tool choice: Claude "might make mistakes or hallucinate" when outputting coordinates or selecting tools.
- Awkward controls: scrolling may not take effect in some applications, dropdowns and scrollbars "might be tricky", and complex spreadsheet operations "might still require multiple attempts".
- False success: "Claude sometimes acts as if its actions succeeded without explicitly checking their results." The page suggests asking for a screenshot check after each step.
Google warns that "Computer Use may contain errors and security vulnerabilities" and advises against it for "actions where serious errors cannot be corrected".[4] OpenAI says to "check the actual outcome instead of relying only on the model's final answer".[3] Our reading of the two OSWorld papers is that the main failure has moved from missed clicks in 2024 to lost constraints and skipped verification on long tasks.[5][7]
The security problem: the screen is untrusted input
Everything on screen is input, and any of it can try to instruct the agent. Anthropic: "In some circumstances, Claude follows commands found in content even when they conflict with your instructions", including "instructions on webpages or contained in images".[1] OpenAI: "Treat screen content as untrusted", adding that "Typing sensitive information into a form counts as transmission."[3]
Anthropic's classifiers scan screenshots and steer the model to check whether a flagged instruction really came from you, but "The precautions listed earlier remain important even with these classifiers in place."[1] Google's prompt injection detection is off by default, and "Your application should always implement safety decision handling regardless of configured overrides."[4] The controls you build yourself are largely shared:
| Control | Anthropic | OpenAI | |
|---|---|---|---|
| Isolated environment | A dedicated VM or container with minimal privileges[1] | "Use an isolated browser or VM"[3] | "Run your agent in a sandboxed VM or container"[4] |
| Restrict reachable sites | An allowlist of domains[1] | An allow list of sites and actions[3] | A blocklist to start; "a more restrictive allowlist is even more secure"[4] |
| Clean, credential-free state | Keep sensitive data such as account logins from the model[1] | A fresh browser profile; credentials out of page text and screenshots[3] | "Start from a known, clean state for each new task if possible"[4] |
| Human confirmation | For consequential actions such as financial transactions or agreeing to terms of service[1] | For purchases, data transmission, destructive changes and other hard-to-reverse actions[3] | Whenever the safety decision requires it[4] |
| Bound and log the run | Cap iterations; log every action[1] | Step, time or cost limits, with cancellation[3] | Log prompts, screenshots, actions and safety responses[4] |
For prompt injection and least privilege in agents generally, see how AI agents work and how to secure them.
Computer use, tool calls or a browser script?
This table is our reading of the documentation above, not a vendor recommendation.
| Situation | Prefer | Why (our reading) |
|---|---|---|
| The application has an API or an MCP server | Tool calls | Typed arguments you can validate and no screenshots to pay for; see what MCP is |
| A fixed sequence of steps in a web app you control | A browser automation script | Deterministic, fast and testable, with no model in the loop at run time |
| Web tasks whose pages vary or are built with JavaScript | A browser-level tool | Anthropic points to browser use "when the task stays inside webpages", working through page structure as well as screenshots[2] |
| A desktop or legacy application with no API | Computer use, in a sandbox | The case these tools exist for, where speed is not critical[1] |
| Payments, messages, deletions, accepting terms | Any of the above, plus human confirmation | All three vendors ask for confirmation on consequential actions[1][3][4] |
Whichever you choose, evaluate on your own tasks; a leaderboard score is not your workload.
What this article did not do
We read the sources listed here on 4 November 2026, though not every section of the longer documents. We ran no agent and reproduced no result, so we make no claim about which model operates a screen best. Tool names and model lists change; check the linked pages. More on the agents and safety topic pages.
