How AI Agents Can See Desktop Apps Without Screenshots

AI agents don't need screenshots to see a desktop app. agent-desktop reads the accessibility tree instead. Here is how it works, and what I got when I asked an agent about my Obsidian notes.

How AI Agents Can See Desktop Apps Without Screenshots

Many AI agents that control a computer work from screenshots. They take a picture of the screen, guess where the button is, and click.

agent-desktop works from the accessibility tree instead. This is the same information screen readers use to read apps out loud.

It's an open-source CLI from Vercel Labs, under the Apache-2.0 license. It runs on macOS, Windows and Linux.

It has no AI inside. It reads the app and returns what it finds as JSON. The agent you connect to it, like Claude Code, decides what to do.

In this article, I'll set it up with Claude Code and look at what it returns for a real app. Then I'll ask Claude a simple question about my Obsidian notes.

I'll use Obsidian as the test. It's a desktop app, and I want to see what an agent can learn from its window alone.

Setting it up

Install it with npm:

npm install -g agent-desktop

Then check that it can reach the accessibility system:

agent-desktop status

Output of agent-desktop status

If accessibility shows as granted, you're ready.

agent-desktop has an MCP server built in. This command adds it to Claude Code:

claude mcp add agent-desktop -- agent-desktop mcp

Adding the MCP server to Claude Code

Claude Code can now use agent-desktop's commands as tools. You can also run the same commands from the shell, which is how I show the output below.

There is one more step on Linux. An app only shows up if it registers with the accessibility system. Many desktop apps do this on their own, but Slack is built on Electron, and it needs a flag:

slack --force-renderer-accessibility

Without it, Slack showed me only its menu bar. With it, I got the full window.

Obsidian was a little different, because mine runs as a Flatpak. I gave it access to the accessibility bus once, and started it with the same flag:

flatpak override --user --talk-name=org.a11y.Bus md.obsidian.Obsidian
flatpak run md.obsidian.Obsidian --force-renderer-accessibility

After this, Obsidian appears in agent-desktop list-apps. On my machine it's listed as xdg-dbus-proxy, and not as obsidian.

list-apps showing the app

What is the accessibility tree?

Every app tells the operating system what's in its window: buttons, text, menus. A screen reader uses this to read the app out loud.

It's a tree. A window has groups, groups have buttons and text, and each item has a role, a name and a state.

I made a small note in Obsidian with four tasks, and ticked two:

The Obsidian note with two tasks ticked

This is the command that reads the window:

agent-desktop snapshot --app xdg-dbus-proxy --compact --max-depth 40

And this is one checkbox from the result:

{
  "role": "checkbox",
  "ref_id": "@skyk1hwakht4e:e233",
  "states": ["checked"],
  "available_actions": ["Toggle", "RightClick", "Click", "SetFocus"]
}

The snapshot output in the terminal

What each part means:

  • role says what the element is. This one is a checkbox.
  • states says what it is doing right now. This box is checked.
  • available_actions lists what you can do to it.
  • ref_id is its address. We'll look at that next.

The task text is a separate item next to the checkbox. Claude matches them up.

How agent-desktop works

There are four layers between Claude and the app:

The four layers and the loop

  • Claude Code decides what to do.
  • agent-desktop turns that into commands, and returns the app's tree as JSON.
  • The accessibility API is how the operating system talks to apps. It is AT-SPI on Linux, AX on macOS and UI Automation on Windows.
  • The app is Slack, Obsidian or anything else.

Claude points at things with refs, and not with coordinates. Every element you can use gets a ref, like @skyk1hwakht4e:e233.

The first part is the snapshot ID. The second part is the number of the element in that snapshot.

So the agent says "use e233 from this snapshot", and not "click at x=412, y=630". A ref is only valid for the snapshot it came from.

agent-desktop has no loop of its own. The loop lives in Claude.

Each command does one small thing and prints one JSON result. Claude runs them in four steps.

1. Look. Claude takes a snapshot of the app.

agent-desktop snapshot --app <app> -i --compact

The result is the tree, with a ref on every element Claude can use.

2. Decide. Claude reads the tree and picks what matters.

This step happens inside Claude. agent-desktop only supplies the facts: names, roles, states and refs. If the question is "which tasks are open?", Claude looks at the checkboxes. If the job is "press Save", it looks for the button named Save.

3. Act. Claude uses a ref to click, toggle or type.

agent-desktop click @<snapshot>:e<N>

By default, agent-desktop does this through the accessibility API. It doesn't move your mouse or take over your keyboard. You can keep working while it runs.

The result says what happened. This is a click I sent to a Help button in Slack:

"steps": [{ "mechanism": "semantic_api", "outcome": "succeeded" }],
"disposition": { "delivery": "delivered_unverified", "retry": "unsafe" }

semantic_api means it used the accessibility action, not a fake mouse click. delivered_unverified means the click was sent, but agent-desktop can't promise it did anything.

So Claude can't just assume it worked. It checks.

4. Check. Claude takes another snapshot and compares.

Did the box get ticked? Did a new window open? Claude decides from the new tree, not from the result of the click.

If the screen changes and Claude uses an old ref, the tool returns an error like STALE_REF. Claude then looks again, and picks a new ref.

The result also says whether a retry is safe. Claude should only repeat an action when it says safe. Otherwise a click could happen twice.

In the demo below, I only use Look and Decide. That's already enough to answer real questions about an app.

A full snapshot of a big app can get large, so there are flags to keep it small:

  • -i keeps only the items you can interact with.
  • --skeleton gives a shallow overview, with a count of what's hidden below each branch.
  • --root <ref> reads only one part of the tree.
  • --compact removes empty wrapper levels.

On Slack, a skeleton snapshot was about 2.6 KB. The interactive one was about 19 KB.

One more thing: snapshots stop at a depth of 10 by default. On Slack, that hid every message. Adding --max-depth 40 fixed it.

Trying it on my Obsidian note

I asked Claude one question: which tasks in my Obsidian note are still open?

I didn't install a plugin or export anything. Claude just read the window.

This is only a demo. Notes like these are plain files, so you could read them in other ways. The point is what an agent can learn from the window alone.

I opened a fresh Claude Code session with the MCP server connected, and typed:

Use agent-desktop to read the Obsidian window (the app is named xdg-dbus-proxy). Which tasks in the "agent-desktop demo" note are still open?

Claude called agent-desktop 3 times and answered in 26 seconds. It reads the window with a command like this:

agent-desktop snapshot --app xdg-dbus-proxy --compact --max-depth 40

The snapshot of my window had about 260 refs. Four of them were checkboxes.

Claude read the tree. Two checkboxes had "states": ["checked"]. The other two had no state.

It matched each checkbox with the text next to it, and answered:

Two tasks are still open: Book the team lunch and Update the README.

Claude's answer

Before I ticked anything, all four boxes had no state. After I ticked two by hand, the same command returned checked on two of them.

So Claude saw the real state of the app, not a guess from a picture.

agent-desktop vs other ways

A common way to let an AI agent use a desktop app is with screenshots. Anthropic's computer use tool is one example. It gives Claude screenshots, plus mouse and keyboard control. After a batch of actions, Claude usually takes another screenshot to see what happened.

It works on any app you can see on screen. That's a big plus.

The downside is that the agent works from pixels. It has to read text out of an image and work out where to click.

Some tools mix both. They take screenshots and also use the accessibility system for clicking or typing. agent-desktop goes the other way: the accessibility tree is how it sees the app.

The same Obsidian note, seen as pixels and as named elements

agent-desktop gets the same information straight from the app. The text is exact, and a checkbox comes with its state.

Screenshot agent agent-desktop
What it reads An image of the screen The accessibility tree, as JSON
How it points Coordinates Refs
Works on Any app you can see Apps that expose an accessibility tree
Your mouse and keyboard Used by the agent Left alone by default

If an app doesn't expose a tree, screenshots are still the only way in.

You might expect the output to be smaller than a screenshot, but not always. My full Obsidian snapshot was about 41 KB of JSON. That's what the flags are for. Claude can read the outline first, then open one part.

Claude Code on its own works with your files and your terminal. It can't see a desktop app unless you give it a tool. Browser tools cover web pages, and agent-desktop covers the desktop apps next to them, like Slack, Obsidian or an internal company tool.

And if the app has an API, use the API. It's faster and more reliable than reading a screen. agent-desktop is for the apps that don't have one.

So, is it worth trying?

I think so, if you have an app that Claude can't reach any other way. Here is where I'd use it:

  • Apps with no API. Internal tools, older desktop software, and other apps that don't offer one. Claude can read what's on screen without a plugin.
  • Checking an app after a change. Take a snapshot and check that the right items and states are there. It could help when testing desktop apps.
  • Asking questions about what's open. "What's in this window?" or "Which items are marked done?" works in any app you can make visible.

What I liked most is how plain it is. Every command prints JSON, so you can run it yourself and see exactly what Claude sees. There's no hidden model, and no guessing from pictures.

On Linux, Slack and Obsidian needed a couple of extra steps. After that, reading them worked well.