Computer / browser use

ConceptWhat it is

Computer use, also called browser use, is an agent pattern where the model operates ordinary software the way a person does, reading the screen and issuing mouse and keyboard actions instead of calling a structured API. The agent receives a rendered screenshot or an accessibility / DOM tree, reasons about what it sees, and emits low-level actions like click at these coordinates, type this text, scroll, or press Enter. A controller executes those actions against a real browser or desktop, the screen changes, and the loop repeats.

It exists to reach the large surface of systems that have no usable API: legacy enterprise desktop apps, internal web tools behind logins, vendor portals, and anything a human can do in a browser but a developer cannot script cleanly. Because the interface is the pixels and widgets themselves, the same agent can in principle work across any application, at the cost of the brittleness and slowness that come from acting through a UI rather than a contract.

How it worksThe mechanics

A goal is given in natural language; the agent captures the current state as a screenshot and/or structured accessibility tree, the model plans a single next action, and a driver, a browser-automation library or OS input layer, executes it as a click, keystroke, scroll, or navigation. The environment updates, the agent re-observes the new state, and it repeats this observe, plan, act loop step by step, often re-grounding on each frame because element positions can move. The loop ends when the model judges the goal met, a stopping condition or step budget is hit, or a verification check confirms the outcome.

At a glanceSee it

The grounding problem — how a high-level intent like click the refund button becomes an actual on-screen coordinate, through raw pixels, a UI tree node, or a numbered set-of-marks overlay.

A safety gate the bare loop omits — screen text is screened for injected instructions, and consequential actions like purchases are held for human confirmation before they commit.

When to use itWhere it fits

• The target system has no API, SDK, or stable integration point and exposes only a human-facing UI.
• You must automate across many heterogeneous web apps or legacy desktop tools without building a connector for each.
• Tasks are exploratory or long-tail, filling forms, navigating portals, or gathering data, where bespoke scripting is not worth it.
• A human can supervise and the workflow tolerates occasional retries and slower runs.

When NOT to use itLimits & anti-patterns

• A clean API or direct data access exists, call it instead; it is faster, cheaper, and easier to verify.
• The task is high-volume or latency-sensitive, where per-step screenshots and model calls are too slow and costly.
• Errors are unacceptable or irreversible, such as payments or production changes, without strong guardrails and confirmation.
• The UI changes constantly or has strong anti-automation defenses like CAPTCHAs and bot detection.

Trade-offsAdvantages & costs

ExampleIn the real world

An operations team needs to reconcile invoices from a supplier portal that offers no export API. A computer-use agent is given the goal "download last month's paid invoices and enter their totals into the finance sheet." It opens the portal in a controlled browser, logs in with supplied credentials, reads the screenshot to locate the date filter, clicks and types the range, waits for the table to render, opens each invoice, and reads the total from the detail view. For each one it switches to the spreadsheet tab and types the value into the next row. A human reviews a short summary before the sheet is saved, and any invoice the agent could not read confidently is flagged rather than guessed.

ToolsHow to implement it

• Anthropic's Claude computer use, a capability where the model takes a screenshot and returns mouse and keyboard actions to drive a virtual desktop or browser.
• OpenAI's Operator and its underlying Computer-Using Agent (CUA), which drive web pages by reading the rendered view and acting on it.
• browser-use, an open-source library that pairs an LLM with Playwright so agents can navigate and act on web pages.
• Playwright and Selenium, the underlying browser-automation drivers that actually execute the clicks, typing, and navigation.

Cost & effortWhat it takes

This is one of the most expensive agent patterns per task. Each step sends a fresh screenshot, a large image input, plus accumulated context, and non-trivial tasks run dozens to hundreds of steps, so token and call counts climb quickly and latency is measured in minutes rather than seconds. Vision-capable models and a sandboxed browser or VM add infrastructure cost. Engineering effort shifts from building integrations to building reliability: retries, step budgets, verification checks, human review, and monitoring. Budget for high variable inference cost and ongoing maintenance as target UIs drift, and reserve the pattern for cases where the absence of an API rules out cheaper automation.

RelatedGo deeper