Oleksandr Kuzmenko, EditorPublished 6 min readAI-assisted · editor-reviewed
TL;DR
- Enables UI automation via OS accessibility trees instead of heavy vision models and screenshots.
- Ships a native Model Context Protocol server exposing 57 operational UI control actions.
- Executes headless by default on macOS 13+, Ubuntu 24.04+, and Windows.
Key facts
- Total Commands
- 61 command names, 57 operational
- Supported OS
- macOS 13+, Ubuntu 24.04+, Windows
- Execution Mode
- Headless by default (no pointer movement)
- Protocol Support
- Model Context Protocol over stdio
Architecture Without Screen Pixels
Unlike vision-based computer use models, agent-desktop queries platform-specific accessibility frameworks: Cocoa Accessibility on macOS, UI Automation on Windows, and AT-SPI on Linux. It exposes the active UI state as a structured JSON snapshot containing element roles, names, states, and available actions.
Actions target a generated ref_id (e.g. @s8f3k2p9:e8). By executing directly against accessibility handles, the system operates headless without hijacking the system mouse pointer or requiring visual rendering.
Model Context Protocol Integration
The repository provides standard Model Context Protocol support out of the box via agent-desktop mcp. Once added to your agent's MCP configuration, 57 operational commands become accessible as tools (desktop_snapshot, desktop_click, desktop_type, desktop_session_start).
{
"mcpServers": {
"agent-desktop": {
"command": "agent-desktop",
"args": ["mcp"]
}
}
}Core Commands and Safety Handles
The tool ships 61 command names, including:
- Navigation:
snapshot,find,get,list-surfaces,list-windows - Interaction:
click,double-click,type,set-value,scroll,toggle - Safety: Returns
STALE_REForAMBIGUOUS_TARGETif the interface tree mutates between actions. - Piped Input:
typeandset-valuesupport--stdinto keep secrets and sensitive tokens out of process execution logs.
Try it in 2 minutes
brew install agent-desktop
agent-desktop permissions --guide
agent-desktop snapshot --app Finder -iWhen to use
- Controlling native desktop tools like Finder, Slack, or system settings from terminal AI agents.
- Building lightweight local desktop agents without sending constant desktop screenshots to multimodal APIs.
When NOT to use
- Canvas-based or OpenGL/WebGL games and custom graphic apps lacking accessibility labels.
- Remote web browser automation where standard Playwright or Puppeteer APIs are already available.
What to do today
- Install agent-desktop using Homebrew on macOS: `brew install agent-desktop`.
- Add the agent-desktop MCP server to Claude Code or Cursor settings.
- Test `agent-desktop snapshot --app Finder -i` to verify local accessibility permissions.
Step names are taken from What to do today.
Step names are taken from What to do today. Install agent-desktop using Homebrew on macOS: `brew install agent-desktop`.. Add the agent-desktop MCP server to Claude Code or Cursor settings.. Test `agent-desktop snapshot --app Finder -i` to verify local accessibility permissions.
- Install agent-desktop using Homebrew on macOS: `brew install agent-desktop`.
- Add the agent-desktop MCP server to Claude Code or Cursor settings.
- Test `agent-desktop snapshot --app Finder -i` to verify local accessibility permissions.
