Computer use

Computer-use AI reads a screenshot, chooses an action, sees the new screen, and repeats. It helps when a client portal or old desktop system has no useful API. It also adds risk because the model must interpret each field, pop-up, permission, and layout change. In July 2026, OSWorld 2.0 tested 108 long computer workflows that took skilled people a median of about 1.6 hours. The best tested setup completed 20.6% under the benchmark’s strict 500-step measure. A service firm should start with one bounded transaction, a known starting screen, a result it can verify, and a recovery path for interface changes. Use direct system access when it exists. Use computer control when the screen is the only practical route and the action can be checked before it causes harm.

How does computer-use AI work?

Computer use is a loop between a model and a controlled browser or desktop. The application shows the model a screenshot, receives a proposed action, executes that action, and returns a fresh screenshot.

Google’s computer-use documentation describes the sequence as a goal and screenshot, a suggested interface action, execution, an updated screen, and another model request. OpenAI documents the same action loop for clicks, typing, scrolling, and screenshot requests. The loop ends when the task finishes, fails, reaches a stop rule, or needs a person’s decision.

Suppose a firm must copy an approved project status into a client portal with no API. The computer-use system can open the right account, find the project, enter the status, and check the confirmation screen. The completed update matters more than the number of clicks.

A chatbot that tells a coordinator which buttons to press is different. The person still operates the software and owns every handoff. Computer use gives the system the controls. A good connection becomes more useful, and a wrong instruction can act more quickly.

How is computer use different from an API and browser automation?

The three methods reach software through different interfaces. Pick the most direct stable method the system supports.

Method	How it reaches the software	Best fit	Main weakness
API	Sends named data and actions directly	Stable, supported transactions	The needed action may not exist
Fixed browser automation	Follows written selectors and rules	Repetitive screens with predictable paths	Layout or selector changes break the script
Computer-use AI	Interprets screenshots and chooses UI actions	Variable screens or software with no useful API	More ambiguity, delay, and security risk

An API call can say, in effect, “update project 417 to approved.” The software knows the project field, the allowed status values, and whether the update succeeded. Computer use must find the same project on screen, decide which control is the status field, choose the right option, and read the result.

Fixed browser automation sits between the two. It can be fast and dependable when a page stays consistent. It does not need a model to reinterpret the screen at every step. Computer use earns its extra cost when the interface varies enough to make a fixed script brittle, while the task remains narrow enough to verify.

This is one way to connect AI to the systems where the work already lives. It is a fallback connection for software built only for people. A reliable API is the better path when it exposes the required action.

What makes computer use fail in production?

Computer use carries hidden state from one screen to the next. A pop-up may cover a button. A session may expire. A client account may open in the wrong tab. A slow page may look complete before the final field loads. The next click can fit the screenshot and still be wrong for the transaction.

Long workflows multiply those chances. The July 13, 2026 revision of OSWorld 2.0 tests 108 end-to-end computer tasks. Human users take a median of about 1.6 hours per task. One tested setup averaged 318 tool calls, compared with about 30 in the first OSWorld benchmark. Under the 500-step binary-completion measure, the best tested configuration completed 20.6% and received a 54.8% partial score.

The paper reports failures that matter more than a ranking. Agents lost track of constraints, missed information that arrived during the task, guessed when they should have asked, and skipped verification. A model may click accurately for twenty steps and still submit the wrong account’s record on step twenty-one.

Speed creates another limit. OSWorld-Human, submitted June 19, 2025, evaluated 16 agents and found that the strongest systems took 1.4 to 2.7 times more steps than necessary. Later steps could take three times longer than early steps because planning and reflection dominated the delay. A task that saves labor but ties up a browser for forty minutes may still be useful. Measure that cost instead of assuming screen control is free.

These benchmarks do not predict the success rate of a five-step portal update. They show why a short demo cannot approve a long client workflow. Test the exact software, account state, exceptions, and final result that will exist in production.

Which computer-use actions need human approval?

Approval should happen immediately before an action that is hard to reverse. Sending a client message, submitting a government form, moving money, deleting a record, changing permissions, accepting terms, and exposing sensitive data should not inherit permission from a broad instruction given ten screens earlier.

OpenAI recommends an isolated browser or virtual machine, an allowlist of domains and actions, and a person in the loop for purchases, authenticated flows, destructive actions, and other hard-to-reverse steps. Anthropic’s computer-use security guidance adds low-privilege environments, limited sensitive data, and confirmation for actions with meaningful consequences.

The screen itself is untrusted input. A webpage, email, document, or image can contain instructions aimed at the agent instead of the user. Anthropic warns that on-screen instructions can conflict with the user’s request. OpenAI tells implementers to treat page content as untrusted and to count only direct user instructions as permission.

Give the worker a separate account with the minimum access required. Limit which sites and actions it can reach. Keep a record of screenshots, proposed actions, approvals, and final results. A guardrail blocks a forbidden path before it runs. A human-in-the-loop step owns a decision the system should not make alone.

The autonomy test is simple: can the firm check the result, and can it undo the action? Check and undo the action before granting more access. A system may prepare a payment or client email without receiving permission to send it.

How should a service firm test computer use?

Start with one transaction that has a clear starting state and a visible finish. “Update the approved project status in this portal” can be tested. “Handle the client account” hides several decisions inside one instruction.

Build the first test set from real permitted screens. Include the normal path, an expired session, a pop-up, a missing field, a changed label, a slow load, the wrong account already open, and a task that should stop for clarification. Record the expected final state for every case.

Score the result in the software. A clean click history means little if the record did not change or changed twice. Track full completion, exception rate, human review minutes, retries, wrong-account attempts, and actions stopped by a safety rule. Keep partial completion separate from accepted completion.

Write the recovery path before increasing volume. Decide how many times the system can retry, what evidence proves success, when it must ask a person, and how a person resumes without repeating completed work. Keep a manual route for outages and changed interfaces.

Name one person who owns the process end to end. Document the process first, including account selection, field rules, approval points, evidence, and exceptions. Computer use will follow the process that exists, including the workaround everyone forgot to mention.

If the job spans several systems, the wider control layer is orchestration. It decides when computer use runs, what context it receives, where the result goes, and which failure hands the work to a person. The computer-use step remains one tool inside that workflow.

Where does computer use sit on the Delivery Model Ladder?

Computer use belongs at Stage 2, Augmented, on the Delivery Model Ladder when a named workflow starts from a defined condition, opens permitted software, completes a bounded transaction, verifies the final state, routes exceptions, and records the result. A person may still approve the final external action.

At Stage 1, Enhanced, a person asks an assistant to update one screen, watches the actions, and carries the result into the next step. The model has the keyboard and mouse, while the person still owns the workflow.

Screen control alone does not create an AI employee. The worker needs standing access, a defined job, a trigger, a review standard, and responsibility for a result. Computer use supplies one access method when the firm’s software offers nothing better.

When should a firm skip computer use?

Use an API when it exposes the needed action reliably. Keep a person at the controls when a mistake cannot be checked or reversed. Skip computer use when a fixed script or ordinary manual action handles a rare task more safely.

Computer use earns its place when the screen is the only practical route, the task is bounded, and the firm can check the final state. Start with one transaction and keep a manual route for the day the interface changes.