[FEATURE]: Computer Use / Desktop Control tool (screenshot + mouse/keyboard + browser automation) #20490

Feature hasn't been suggested before.

• I have verified this feature I'm about to request hasn't been suggested before.

Re: duplicate flag — #3495 was a newbie question about DIY workarounds (closed). #6992 covers browser-only automation. This issue is specifically about full desktop computer use (any GUI app, not just browser) and the per-OS/display-server requirements that make it non-trivial. Not a duplicate.

What I'm requesting

Claude Code shipped full desktop computer use (screenshot + mouse + keyboard control) in March 2026. OpenCode has nothing equivalent. I'd like to see it added.

The core loop is: agent takes a screenshot, sees what's on screen, clicks/types, takes another screenshot, repeats. This lets it work with any GUI app — not just things that have a CLI or API.

For browser-only automation, see #6992. This request is about the broader desktop case: Electron apps, native GUI tools, visual debugging, anything that lives outside the terminal.

Per-platform requirements

Linux — X11

DISPLAY set, no WAYLAND_DISPLAY. Use scrot for screenshots, xdotool for mouse/keyboard input, xclip for clipboard. Works headlessly with Xvfb for CI.

Linux — Wayland

WAYLAND_DISPLAY set. xdotool does not work here — completely different toolchain required:

• Screenshots: grim on wlroots compositors (Sway, Hyprland, river); XDG Desktop Portal D-Bus call on GNOME; spectacle on KDE. Use XDG_CURRENT_DESKTOP to pick the right one.
• Mouse/keyboard: ydotool with the ydotoold daemon running. Requires user in the input group and the uinput kernel module loaded. If the daemon isn't running, fail with a clear error telling the user what to fix — not a silent hang.
• Clipboard: wl-copy / wl-paste (wl-clipboard package).
• Headless/CI: Use cage or weston --headless as a virtual Wayland compositor.

Linux — XWayland (mixed session)

Both DISPLAY and WAYLAND_DISPLAY are set. X11 tools (xdotool, scrot) work for XWayland windows but not for native Wayland windows. This is a common situation on distros that run XWayland compatibility alongside a Wayland session and needs its own detection path.

macOS

• Screenshots: screencapture -x <file> built-in; -R x,y,w,h for region capture.
• Mouse/keyboard: cliclick (available via Homebrew) or CoreGraphics via Node FFI.
• Clipboard: pbcopy / pbpaste built-in.
• Accessibility permission: macOS blocks input injection unless the running process (Terminal, the OpenCode binary, etc.) has been granted Accessibility access in System Settings → Privacy & Security → Accessibility. The implementation must detect when this permission is missing and show a clear error with instructions on how to grant it — otherwise the user just sees mysterious failures.

Windows

• Screenshots: PowerShell with System.Windows.Forms / System.Drawing, or nircmd.
• Mouse/keyboard: PowerShell SendKeys or Win32 mouse_event / keybd_event via a small helper. AutoHotkey is another option if available.
• Clipboard: PowerShell Get-Clipboard / Set-Clipboard built-in.
• UAC caveat: Input injection is blocked into windows running at a higher elevation than the injecting process. This is a known Windows limitation and should be documented clearly rather than worked around silently.

General requirements

• Auto-detect the environment and route to the correct backend — don't make the user configure this manually
• When a required tool is missing, surface a specific, actionable error (what to install, what permission to grant) rather than crashing or hanging silently
• Opt-in via config (computerUse.enabled: false by default)
• Input actions (mouse/keyboard) should require DangerFullAccess permission; screenshot-only should be lower friction
• All actions should be logged to the session event stream for auditability

Related

• [FEATURE]: Browser automation like Claude in Chrome #6992 — browser automation specifically (Playwright/Claude in Chrome) — overlaps with the browser tier of this
• [FEATURE]: possible to extend to computer use? #3495 — closed, was a question not a feature spec
• [FEATURE]: Sub-agent tool-set isolation by role (Explore / Plan / Verification scopes) #20483 — sub-agent tool isolation (computer use should be excluded from read-only agent scopes)
• Lessons from claude-code-main? #20475 — learnings from claude-code source