Computer use

Computer-use AI operates software through the screen, looking at a screenshot, choosing what to click or type, then looking again to see what happened. It can help with a client portal or old desktop system that has no useful API.

But the model has to interpret the fields, pop-ups, permissions, and layout changes that a direct connection would avoid. So I’d start with one limited task, a known starting screen, a result you can check, and a way for a person to take over.

If the software offers reliable direct access to the action you need, use that. Computer control is worth considering when the screen is the practical route and you can check the action before it causes harm.

How does computer-use AI work?

The model works through an application that controls the browser or desktop. That application sends a screenshot, receives the model’s proposed action, runs it, and sends a fresh screenshot so the model can choose the next step.

Google’s computer-use documentation describes the sequence as a goal and screenshot, a suggested interface action, execution, an updated screen, and another model request. OpenAI documents the same action loop for clicks, typing, scrolling, and screenshot requests. The loop ends when the task finishes, fails, reaches a stop rule, or needs a person’s decision.

Suppose a firm must copy an approved project status into a client portal with no API. The computer-use system can open the right account, find the project, enter the status, and check the confirmation screen. The completed update matters more than the number of clicks.

That’s different from a chatbot telling a coordinator which buttons to press, because the person still operates the software in that case. With computer use, the system has the controls, so a wrong instruction can become a real action before someone spots it.

How is computer use different from an API and browser automation?

They reach the software in different ways, and I’d choose the most direct stable one it supports. An API sends a named request straight to the system, though it may not offer the particular action you need.

Fixed browser automation follows written rules for finding and using controls on a page. That’s useful for repeated work on stable screens, but changes to the page can break those rules.

Computer-use AI interprets screenshots and chooses the next action. It can deal with some variation a fixed script can’t, but that interpretation adds uncertainty, delay, and security risk.

An API call can say, in effect, “update project 417 to approved.” The software knows the project field, the allowed status values, and whether the update succeeded. Computer use must find the same project on screen, decide which control is the status field, choose the right option, and read the result.

Fixed browser automation can be fast and dependable when a page stays consistent, because it doesn’t ask a model to interpret every screen. Computer use earns its extra cost when the screen varies enough to break a fixed script, while the task is still limited enough to check.

This is one way to connect AI to the systems where the work already lives. It is a fallback connection for software built only for people. A reliable API is the better path when it exposes the required action.

What makes computer use fail in production?

What you see in one screenshot doesn’t tell you everything about the software. A pop-up can cover a button, a session can expire, the wrong client’s account can be open in another tab, or a slow page can look ready before its final field loads.

The next click can look right for that screenshot and still be wrong for the job.

Long workflows give those mistakes more places to happen. The July 13, 2026 revision of OSWorld 2.0 tests 108 end-to-end computer tasks that took skilled human users a median of about 1.6 hours each.

One tested setup averaged 318 tool calls, compared with about 30 in the first OSWorld benchmark. Under the 500-step measure that counts a task only if it fully finishes, the best tested configuration completed 20.6%, while its partial-credit score was 54.8%.

Those are different measures, so partial progress shouldn’t be counted as a completed job.

The paper reports failures that matter more than a ranking. Agents lost track of constraints, missed information that arrived during the task, guessed when they should have asked, and skipped verification. A model may click accurately for twenty steps and still submit the wrong account’s record on step twenty-one.

There’s also the time it takes. OSWorld-Human, submitted June 19, 2025, evaluated 16 agents and found that the strongest systems took 1.4 to 2.7 times more steps than necessary.

Later steps could take three times longer than early ones because planning and reflection accounted for most of the delay. A task that saves labor but ties up a browser for forty minutes may still help, but you need to count that time rather than assume screen control is free.

These benchmarks do not predict the success rate of a five-step portal update. They show why a short demo cannot approve a long client workflow. Test the exact software, account state, exceptions, and final result that will exist in production.

Which computer-use actions need human approval?

Approval should happen immediately before an action that is hard to reverse. Sending a client message, submitting a government form, moving money, deleting a record, changing permissions, accepting terms, and exposing sensitive data should not inherit permission from a broad instruction given ten screens earlier.

OpenAI recommends an isolated browser or virtual machine, an allowlist of domains and actions, and a person in the loop for purchases, authenticated flows, destructive actions, and other hard-to-reverse steps. Anthropic’s computer-use security guidance adds low-privilege environments, limited sensitive data, and confirmation for actions with meaningful consequences.

A webpage, email, document, or image can contain instructions aimed at the agent, and Anthropic warns that these can conflict with what the user asked for. OpenAI tells builders to treat page content as untrusted and count only direct user instructions as permission, so text on a page shouldn’t be allowed to grant the agent more authority.

Give the worker a separate account with only the access it needs, limit its sites and actions, and keep a record of screenshots, proposed actions, approvals, and final results.

A guardrail blocks a forbidden action before it runs, while a human-in-the-loop step gives a person a decision the system shouldn’t make alone.

Before granting more access, ask whether the firm can check the result and undo the action. A system can prepare a payment or client email without also having permission to send it.

How should a business test computer use?

Start with one transaction that has a clear starting state and a visible finish. “Update the approved project status in this portal” can be tested. “Handle the client account” hides several decisions inside one instruction.

Build the first test set from real permitted screens. Include the normal path, an expired session, a pop-up, a missing field, a changed label, a slow load, the wrong account already open, and a task that should stop for clarification. Record the expected final state for every case.

Check the result in the software, because a clean click history doesn’t help if the record didn’t change or changed twice. Track full completion, how often the system needs help, review minutes, retries, wrong-account attempts, and actions stopped by a safety rule.

Keep partial completion separate from work you’ve accepted as finished.

Write the recovery path before increasing volume. Decide how many times the system can retry, what evidence proves success, when it must ask a person, and how a person resumes without repeating completed work. Keep a manual route for outages and changed interfaces.

Name one person who owns the process end to end. Document the process first, including account selection, field rules, approval points, evidence, and exceptions. Computer use will follow the process that exists, including the workaround everyone forgot to mention.

If the job spans several systems, the wider control layer is orchestration. It decides when computer use runs, what context it receives, where the result goes, and which failure hands the work to a person. The computer-use step remains one tool inside that workflow.

Where does computer use sit on the Delivery Model Ladder?

Computer use belongs at Stage 2, Augmented, on the Delivery Model Ladder when a named workflow starts from a defined condition, opens permitted software, completes a bounded transaction, verifies the final state, routes exceptions, and records the result. A person may still approve the final external action.

At Stage 1, Enhanced, a person asks an assistant to update one screen, watches the actions, and carries the result into the next step. The model has the keyboard and mouse, while the person still owns the workflow.

Screen control alone does not create an AI employee. The worker needs standing access, a defined job, a trigger, a review standard, and responsibility for a result. Computer use supplies one access method when the firm’s software offers nothing better.

When should a firm skip computer use?

Use an API when it exposes the needed action reliably. Keep a person at the controls when a mistake cannot be checked or reversed. Skip computer use when a fixed script or ordinary manual action handles a rare task more safely.

Start with one limited task where the screen is the practical route and you can check what changed. Keep a manual way to finish it when the interface changes or the system stops.