# computer use

Also called: computer use tool, computer-using agent, CUA, GUI agent, computer_toolset_20260801

Computer use is a tool that lets a language model operate a graphical interface: the model looks at a screenshot and replies with mouse and keyboard actions such as click, type and scroll. The model never touches the machine. Your application runs each action in a browser, virtual machine or container you control, captures a new screenshot, and sends it back so the model can pick its next step. Anthropic and OpenAI both ship it as a tool in their APIs.

## What does computer use mean?

Product pages use the phrase loosely for any agent that gets work done on a computer, including agents that only call APIs or run shell commands. In API documentation it means something narrower: a tool whose calls are screen-level actions with pixel coordinates, and whose results are mostly screenshots. Anthropic calls it the computer use tool and ships it as the computer_toolset_20260801 toolset. OpenAI's Responses API exposes it as a tool of type computer.

Computer use is a form of tool calling with a fixed, provider-defined schema. You do not describe the actions yourself. The provider defines them, the model was trained on them, and your job is to execute them faithfully and report back what the screen looks like now.

## How does the computer use loop work?

On Anthropic's Messages API you add {"type": "computer_toolset_20260801"} to tools, with no beta header. That one entry gives Claude 17 member tools, among them screenshot, left_click, type, scroll, key and zoom. When Claude wants to act, the response has stop_reason: "tool_use" and one or more tool_use blocks whose name is the member tool and which carry "toolset_name": "computer". You run each block in order, then send one tool_result per block, matched by tool_use_id and echoing the toolset name. Only screenshot and zoom results need an image; a short text such as OK is enough for the rest.

OpenAI's loop has the same shape with different names. You pass tools: [{"type": "computer"}], the model returns a computer_call with a call_id and a list of actions (click, double_click, drag, move, scroll, keypress, type, wait or screenshot), and you reply with a computer_call_output carrying a computer_screenshot image and the same call_id. The loop ends when the model stops asking for actions and answers in text, so the step and cost limits are yours to set.

## What changes in your code

You write the executor. The API returns intentions. Something on your side, such as a desktop VM or a Playwright-driven browser, has to turn left_click at [512, 742] into a real click and capture the result.Coordinates live in screenshot space. Anthropic's reference says every coordinate is in the pixel space of the screenshots you return. The API does not downscale for you and rejects an oversized screenshot, so resize it yourself, keep the scale factor, and scale Claude's coordinates back up before you click. Claude Opus 4.7 and later accept up to 2576 pixels on the long edge and 4784 visual tokens; earlier models accept 1568 pixels and about 1.15 megapixels.Batches fail as a unit. Claude can return several actions in one turn. Run them in order and stop at the first failure; mark the failed one is_error: true, and answer each later one with the exact text Not executed: an earlier computer action in this turn failed. A turn that leaves any tool_use block unanswered is rejected with invalid_request_error.Tool versions are tied to models. Older integrations use computer_20251124 behind a beta header, which took display_width_px and display_height_px. The toolset version rejects those fields, and Claude 5.5 and later models on the Claude API and Google Cloud reject the older versions.

## Why is computer use slow and risky?

Every step costs a model call plus a screenshot round trip, and Anthropic's own limitations list says computer use may be too slow for human-paced interaction. It suggests work where speed is not critical, such as background information gathering and automated software testing. The same list warns that the model can make mistakes or hallucinate when it outputs coordinates or picks tools, and that reliability drops in niche applications.

The larger risk is prompt injection. Anything on screen enters the context, and Anthropic's docs say instructions on webpages or in images can override yours. Anthropic runs classifiers over screenshots that flag likely injections and steer Claude to check with you. Both providers ask for the same controls on your side: run in an isolated VM, container or browser, keep credentials and sensitive data out of reach, allowlist the sites it can visit, and have a human confirm purchases, data transmission and other hard-to-reverse actions. OpenAI adds a step, time or cost limit, and a check of the actual outcome instead of trusting the model's final answer.

The OSWorld benchmark measured the gap in 2024. On its 369 real desktop and web tasks, humans completed over 72.36 percent, while the best model at publication reached 12.24 percent, struggling mostly with GUI grounding and operational knowledge.

## When should you use something else?

If the system you want to drive has an API, call it through tool calling or an MCP server. One structured call skips the screenshot round trips, and its arguments are easier to audit than a trail of clicks. For tasks that stay inside webpages, Anthropic points to its browser use tool, which reads and acts on the page itself and needs no desktop. For GPT-6 Astra, OpenAI recommends code execution, where the model writes a PyAutoGUI or Playwright script that your environment runs, and keeps the computer tool as a supported alternative. Reach for screen-level computer use when the interface is the only way in, such as a legacy desktop app or a UI you are testing.

## One turn of Anthropic's computer use loop

POST https://api.anthropic.com/v1/messages
{
  "model": "claude-opus-5-5",
  "max_tokens": 1024,
  "tools": [{"type": "computer_toolset_20260801"}],
  "messages": [
    {"role": "user", "content": "Save a picture of a cat to my desktop."}
  ]
}

# A later turn, after Claude has seen a screenshot of the desktop:
{
  "role": "assistant",
  "content": [
    {"type": "text",
     "text": "I'll open the web browser to find a picture of a cat."},
    {"type": "tool_use", "id": "toolu_01WkoTUvSHDzTBu2xnGk8Ep8",
     "name": "left_click", "toolset_name": "computer",
     "input": {"coordinate": [512, 742]}},
    {"type": "tool_use", "id": "toolu_017nJn3RgSCkTMwuZDb4uUov",
     "name": "screenshot", "toolset_name": "computer",
     "input": {}}
  ],
  "stop_reason": "tool_use"
}

# Your code clicks at (512, 742) in the VM, captures the screen,
# and answers both calls in the next user message:
{
  "role": "user",
  "content": [
    {"type": "tool_result", "tool_use_id": "toolu_01WkoTUvSHDzTBu2xnGk8Ep8",
     "toolset_name": "computer",
     "content": [{"type": "text", "text": "OK"}]},
    {"type": "tool_result", "tool_use_id": "toolu_017nJn3RgSCkTMwuZDb4uUov",
     "toolset_name": "computer",
     "content": [{"type": "image",
                  "source": {"type": "base64", "media_type": "image/png",
                             "data": "iVBORw0KGgo..."}}]}
  ]
}

The request and response shapes follow Anthropic's computer use tool reference, and the IDs and coordinates come from its example response. The click and the screenshot arrive in one turn, so your code runs them in order. The tool_use input has no action field: the block's name is the action. Each tool_result echoes toolset_name, and only the screenshot result carries an image.

## Frequently asked

Q: Is computer use the same as tool calling?
A: It is a kind of tool calling. Actions arrive as tool calls and the screen comes back as a tool result, so the loop is the one you already run. The differences are that the provider defines the schema, the inputs are pixel coordinates and keystrokes, and most results are images. You still write the code that executes each action.

Q: Does the model control my actual computer?
A: Only if your executor runs the actions there. The model returns requests, and your application decides where they run. Anthropic recommends a dedicated virtual machine or container with minimal privileges, no access to sensitive data such as account logins, and an allowlist of domains.

Q: Why does the model click in the wrong place?
A: Check scaling first. Coordinates are in the pixel space of the screenshot you sent, so if you downscaled the image, you must scale each coordinate back up before clicking. Anthropic's docs also list coordinate mistakes and hallucinations as a known limitation, so log the screenshot next to each action when you debug.

Q: Is computer use the same as an agent?
A: No. An agent is the loop that decides and acts; computer use is one tool that loop can hold. Anthropic's quick start declares the computer use toolset alongside its bash and text editor tools, and the model picks whichever fits the step.

## Sources

- Claude API docs: Computer use tool: https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool
  the computer_toolset_20260801 toolset and its 17 member tools, the tool_use and tool_result shapes with toolset_name, batch failure handling, screenshot scaling limits, security precautions, injection classifiers and the limitations list
- OpenAI API docs: Computer use: https://developers.openai.com/api/docs/guides/tools-computer-use
  the Responses API computer tool, computer_call and computer_call_output, the action list, the code execution alternative for GPT-6 Astra, and the run-safely controls
- Claude API docs: Browser use tool: https://platform.claude.com/docs/en/agents-and-tools/tool-use/browser-use-tool
  the page-level alternative Anthropic recommends for tasks that stay inside webpages
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (Xie et al., 2024): https://arxiv.org/abs/2404.07972
  369 real computer tasks with execution-based scoring; humans above 72.36 percent and the best model at publication at 12.24 percent

## Related terms

- agent: https://ai.developertools.io/agent/index.md
- tool calling: https://ai.developertools.io/tool-calling/index.md
- prompt injection: https://ai.developertools.io/prompt-injection/index.md
- harness: https://ai.developertools.io/harness/index.md
- guardrail: https://ai.developertools.io/guardrail/index.md
- latency: https://ai.developertools.io/latency/index.md

Canonical page: https://ai.developertools.io/computer-use/
Edited by Derek Wonk. Last updated 2026-10-19.