macOS computer use 1.8x faster, 85% cheaper than cua-driver alone

I have Claude Code operate macOS apps in the background: fill a form, pick an option from a pop-up, run a calculation. With the MCP server I built on top of cua-driver, beans-picker, a typical task in my benchmark ran 1.8x faster and cost 85% less than with cua-driver alone (median 28.1 s vs 49.7 s, $0.060 vs $0.388). The app was never brought to the front in either case, so I could keep working on the same Mac while the agent ran.

This post explains where the time and money went, the idea that removed most of it, and what my benchmark does not prove. The idea applies even if you never install beans-picker.

Each step re-reads the whole window

cua-driver is a good driver. It clicks, types and presses keys in a macOS app without stealing focus. The cost comes from how an agent uses it.

To act, the model calls get_window_state. That returns the window's full accessibility tree and a screenshot. The model reads all of it, finds the element it wants, and clicks by index. After the click it reads the whole tree again to see what happened. A five-step task means five or more full reads, and every read stays in the conversation.

In my runs this was the bulk of the bill. The median cua-driver-only run used 646k Claude tokens, cache included, for tasks like "type this name into the Name field" or "compute 15% of 80 in Calculator".

The model does not need the whole tree to click a button. It needs to know which buttons exist and which one matches the step it has in mind.

The agent chooses from a short list of real controls

beans-picker splits the work into three parts.

• The caller thinks. Claude Code or Codex breaks the task down, picks the next step and decides what to do when something goes wrong.
• Jev picks the element. Jev is a model from TypeSafe that answers choice questions. beans-picker builds a list of candidate actions from the accessibility tree and asks Jev which one matches the step. Jev only picks from that list and never writes text, so it cannot invent a target.
• cua-driver acts. The click, typing or key press goes through cua-driver, in the background.

The server itself never calls a generative model. The model's reading is replaced by a short candidate list and a summary of what changed.

The idea is not tied to Jev. If your agent spends most of its tokens reading UI trees, try giving it a short list of actions that actually exist on the window and let it choose one. The list is much smaller than the tree, and every entry is something real.

How one act call runs

A call to act goes through five stages.

1. Snapshot. cua-driver returns the window's accessibility tree. beans-picker also computes a signature of what matters: the title, field values, visible text and other windows.
2. Candidates. The tree becomes a list of actions: clicks, toggles, text entry, pop-up choices, menu commands sent as their keyboard shortcut, a step up or down on a slider, a page down on a table, Return and Escape, and so on.
3. Jev picks. Jev answers two questions over the same candidates, one that allows "none of these" and one that forces a choice. beans-picker acts only when the leader is clear: a probability of at least 0.8, or at least 0.5 when both questions agree and the leader has twice the runner-up's probability. Otherwise it returns ambiguous with the top candidates, and the caller decides.
4. cua-driver acts. The server refuses anything that needs the foreground, such as bring_to_front. It also watches the frontmost app during the call and stops with foreground_violation if the target app comes to the front.
5. Effect check. beans-picker takes fresh snapshots until the effect shows, up to 5 by default. Only then does it return done.

Text is checked by exact equality in the targeted field. A trailing space that got lost, or text that landed in the search field instead of the note body, comes back as mismatch, not done. Actions that may not be undoable, like delete, close, send and quit, return needs_confirmation until the caller passes allowDestructive: true.

What the calls look like

There are three tools: observe lists candidates, act performs one action and checks it, and extract reads a value. A typical act call from the agent:

{
  "app": "Calculator",
  "instruction": "press the percent key"
}

You can pass text to enter text exactly as given, or then to chain up to 12 more steps that stop at the first one that is not done. If an earlier call returned ambiguous, you call again with the candidateId you want.

The result tells the agent what ran and what changed. Shortened, it looks like this:

{
  "status": "done",
  "action": { "...": "the chosen candidate and its cua-driver route" },
  "change": { "...": "a summary of what changed on the window" }
}

status is one of eight values. The agent-side skill in the repo tells the model what to do with each one.

The benchmark: 1.8x faster and 85% cheaper than cua-driver alone

I ran 8 tasks, 3 times each, in two conditions: Claude Code headless with Sonnet 5 and only cua-driver, or the same setup with only beans-picker. That is 48 runs on 2026-09-24. Five tasks run on a small AppKit window made for the bench, such as a name with leading and trailing spaces or clearing a search field next to a "Delete note" button. Three run on Calculator, such as (48 + 16) / 8.

Success was decided by a separate script that compared the final state with the expected values by exact equality. The agent's own "success" line was only used to count false claims.

The beans-picker cost includes Jev. Jev is billed separately at $0.042 per 1M input tokens, and all 265 Jev calls in the benchmark added about $0.026 in total.

Both conditions nearly always succeeded, so I am not claiming a better success rate. The difference is time and cost. Most of it comes from not sending the whole tree and a screenshot to the model at every step.

What this benchmark does not show

• It does not show fewer false success claims. Neither condition claimed success on a failed run.
• It did not confirm exact text. The helper that reads a field's exact text had no Accessibility permission during the run. 18 text actions came back unverified; the judge later found all of them exactly right.
• It is not faster on every task. On the two simplest tasks, beans-picker was a second or so slower. A Jev call plus fresh snapshots costs more than one direct click.
• It is a small sample. 3 repetitions per task, one model, one person's Mac. I wrote the tasks and the fixture app myself, and the only baseline was Claude Code with cua-driver.

Trying it

You need macOS, Python 3.12+, cua-driver at ~/.local/bin/cua-driver with its Accessibility and Screen Recording permissions, a Jev API key, and the Xcode Command Line Tools.

uv tool install beans-picker
mkdir -p ~/.config/beans-picker && echo 'JEV_API_KEY=...' > ~/.config/beans-picker/.env.local
claude mcp add beans-picker -- "$(command -v beans-picker)"
beans-picker grant-ax   # optional: lets act confirm exact text

After grant-ax, turn on beans-picker axtext under System Settings > Privacy & Security > Accessibility. Without it, text actions return unverified instead of done. For Codex, add the same absolute path as an MCP server in ~/.codex/config.toml.

Window text leaves your machine only when a step needs Jev. The repo's SECURITY.md lists exactly what is sent.

This is a personal hobby project. I am not affiliated with Cua/trycua or TypeSafe; cua-driver and Jev are separate projects that beans-picker talks to.

Repo: https://github.com/mimo-3/beans-picker

I wrote this with help from an AI assistant and checked every number against the benchmark records.

If you find an app or a control that beans-picker gets wrong, please open an issue or tell me in the comments.