Computer use: AI agents that operate a computer like a person – when it makes sense for your company

In vendors’ materials, this capability is called “computer use”. The agent is given a task described in plain language, for example “enter the orders from this spreadsheet into the warehouse system”. It then takes a screenshot, assesses what it sees and makes its next move: a click, typing some text, scrolling the page.

After each move, the agent looks at the screen again and decides what to do next. It does not need access to the program’s internals or a prior recording of the steps. If a button moves somewhere else, it will usually find it anyway.

This flexibility comes at a price. Each step requires an AI model to analyse an image, so it takes longer than with a classic robot and costs more. The agent can also occasionally make a mistake that a robot following a fixed script would not.

Companies have three ways of getting a program to move data between systems by itself. Integration connects programs directly via an API – a “socket” for data exchange provided by the vendor. An RPA (robotic process automation) robot replays recorded clicks according to a fixed script. An AI agent clicks on its own, interpreting what it sees.

The comparison in the table below shows that these solutions complement each other. In many companies, a combination gives the best result: integration wherever possible, a robot for stable screens and an AI agent for unusual cases.

It works best for tasks that an employee does today by copying data between programs, and which are not worth integrating.

Take a hypothetical wholesaler with 70 employees. Orders from some customers come in through those customers’ purchasing portals; each one looks different and changes from time to time. Every day, an employee logs in to five portals and retypes the orders into the sales system. An RPA robot would break with every change to a portal, and integrating with five customers would be too expensive, so here an AI agent has a real advantage. Other typical uses are listed below.

The cost of an agent is calculated in steps, not tasks. In Copilot Studio, Microsoft’s tool for building AI agents, each step consumes 5 Copilot credits with a standard model and 15 with a premium model. A pack of 25,000 credits costs USD 200 per month, or USD 0.008 per credit.

Suppose entering one order into the old system takes 25 steps. With a standard model, that is 125 credits, or about USD 1. At 400 orders a month, the agent will use 50,000 credits, which is two packs costing USD 400 in total.

For comparison: if an employee spends 6 minutes on one order, 400 orders take 40 hours a month. At PLN 70 an hour, that comes to PLN 2,800, while USD 400 for the agent is about PLN 1,600 at an assumed exchange rate of PLN 4 to the dollar. However, the agent’s cost must also include the computer it runs on, the set-up and the time of the person who checks the results and handles exceptions.

This calculation is sensitive to the number of steps. If screens are hard to read or the instructions are unclear, the agent may need several times more steps than you expect. So always measure the cost in a trial, not on paper.

An agent that operates a computer can do anything the account it works under allows: send an email, delete a file, approve an order. That is a completely different scale of risk from an assistant that only answers questions.

A particular threat is prompt injection. A malicious website or document can contain hidden text that the agent will read as a new instruction. Anthropic warns explicitly that instructions on web pages or in images can override the user’s instructions.

Vendors recommend a similar set of safeguards, summarised below. Treat it as a minimum, not a menu to choose from.

The agent works on screenshots, and these are sent to the provider’s AI model. If customer or employee data, or prices, are visible on screen, the provider processes them. This requires a data processing agreement (DPA) and a check of the region in which the data is processed.

In Copilot Studio, an administrator has to deliberately switch on models from companies other than Microsoft, such as Anthropic, for a given environment. Before making that decision, check that provider’s data processing terms. Wherever possible, limit what the agent sees: a separate account that displays only the fields needed, and test data during trials.

The first trial should answer three questions: does the agent do the task correctly, how many steps does it really need and how much work is left for a person? Only work on real cases gives the answers, not a demonstration. The plan could look like this: