Repository navigation
design: remote GUI evaluation — performant desktop for the human client, cheaper answer for agents #5924
Description
design: remote GUI evaluation — a more performant desktop for the human client, and a cheaper answer for agents
Context: RustDesk is the incumbent remote GUI. Measured 2026-11-04 (WS-C, #5775): a displayed session burns 26–45% of a core — it renders via llvmpipe software-GL and encodes in software on our Linux/AMD path — it is launched by hand (~/tools/rustdesk-gl/launch.sh) and has no supervisor (#5775 carries the idle auto-exit ruling). Drew asks: are there more performant GUIs for the client side (a human viewing/controlling a machine), or better ones for agent use?
Environment constraints that score everything below: NixOS laptop with a GPU; sandbox VM with no GPU (4 vCPU, oversubscribed — see #5701); tailnet-only access; self-hosted; declarative packaging matters (nixpkgs presence checked 2026-11-04 via by-name tree).
Part A1 — human client side
| Candidate | CPU, software-render case | CPU, hardware case | Latency posture | Wayland / X11 | Nix packaging | Self-host / tailnet | Unattended |
|---|---|---|---|---|---|---|---|
| RustDesk (incumbent) | High — measured 26–45% of a core displayed; VP9 (libvpx) default; Linux hw-encode via FFmpeg hwcodec exists but is the uneven path (solid NVENC/QSV, weak AMD — field reports + our measurement agree it fell back to software) | Good where NVENC/QSV engage; VAAPI hit-or-miss | Decent on LAN; remote quality depends on relay vs direct | Wayland experimental (portal capture); login screen needs X11 | rustdesk, rustdesk-server in nixpkgs; server module exists |
Self-host hbbs/hbbr; fleet splice designed in #5694 | Yes — permanent password; strongest suit. Covers the sddm greeter, which nothing else here does |
| Sunshine + Moonlight | x264 fallback = last resort, same-or-worse bill than VP9. Do not run it in software mode | Best in field: VAAPI/NVENC/QSV first-class on all three vendors; host CPU ~nil while encoding; 10–25 ms glass-to-glass on LAN-class links | Lowest of the set (built for game streaming) | X11 solid; Wayland via KMS capture (works, rough edges on some compositors); cannot serve the login/greeter screen | sunshine + services.sunshine module; moonlight-qt client |
Fully self-hosted, direct UDP — tailnet-perfect | Service runs pre-login but has no greeter; pair with a greeter-capable fallback |
| KasmVNC | Very low for desktop content (damage-based RFB + WebP, not a video pipeline); poor at full-motion/video | No HW-encode story (not a video pipeline) | Fine for admin/desktop work | X11 only — it is its own Xvnc; cannot share a Wayland session | Not in nixpkgs (upstream .deb/Docker, GPL-2.0) — packaging cost | Self-hosted, browser client, tailnet-friendly | Yes — own virtual session, no console tie |
| Selkies 2.0 (2026-09) | Vendor claim: 1080p60 software encode <100% CPU (x264/vpx) — unmeasured, treat as a target not a fact | VAAPI/NVENC incl. zero-copy paths | WebRTC-class, low | X11 + headless-Wayland backends in 2.0 | Not in nixpkgs | Self-hosted, browser client, single-port WebSocket/WebRTC | Yes for virtual sessions; upstream maintenance health was flagged ("in need of maintainers") pre-2.0 — watch item |
| wayvnc + neatvnc | Low (damage-aware capture, tight/zlib encodings, optional H.264) | H.264 with GPU assist where the compositor hands over dmabuf | Low on LAN | Wayland, wlroots only (sway/labwc/Hyprland — not GNOME/KDE) | wayvnc in nixpkgs |
Self-hosted; also runs against headless sway (WLR_BACKENDS=headless) |
Yes, incl. fully headless sessions |
| TigerVNC | Low for static X11 desktops (Tight encoding) | None | Moderate | X11 only | In nixpkgs | Self-hosted | Yes (own X session via Xvnc); better as the standard viewer than a new server bet |
| X2Go | Low (NX compression) | None | Good on thin links | X11 only, aging | Client only; server not in nixpkgs | Self-hosted | Yes, but upstream is in maintenance twilight — skip |
| NoMachine (proprietary) | Low–moderate, efficient NX | H.264 HW paths on some platforms | Low | X11, partial Wayland | Not packaged (vendor binaries) | Self-host server, but vendor licensing/account surface | Yes — capable; costs the self-host purity |
| AnyDesk (proprietary) | Low–moderate | Some | Low | X11/Wayland varies | In nixpkgs (unfree) | Cloud-mediated by default | Yes; proprietary + account surface |
Chrome Remote Desktop (incumbent: amd-crd container) |
Moderate (VP8/VP9 inside a container on a GPU-less box) | Client-side HW decode only | Variable — Google relay in the path | X11 virtual session | Container image, not a NixOS citizen | No — Google account + Google relays; breaks the tailnet-only/self-host posture | Yes (PIN). Verdict: keep as break-glass only, never expand |
| Apache Guacamole | Adds a gateway hop; the backend VNC/RDP still pays the full encode bill | n/a | +1 hop | Whatever the backend supports | guacamole-server in nixpkgs |
Self-hosted browser access layer | n/a — a gateway, not an encoder; only enters if "browser access with no client install" becomes a hard requirement |
Reading of the table
- The software-render case has no winner: on a GPU-less box every video pipeline pays roughly the same x264/VP9 bill, and llvmpipe rendering stacks on top. The only real answers for the sandbox are don't stream or stream rarely, on demand.
- The hardware case has a clear winner: Sunshine, because hardware encoding is its design center on all three GPU vendors, while RustDesk's Linux hardware path is the part that demonstrably fell back to software here.
- RustDesk's moat is real but narrow: greeter/login-screen access and unattended permanence. Sunshine structurally cannot do the greeter. This argues for a split, not a swap.
Recommendation — human side
- Laptop (has GPU): adopt Sunshine + Moonlight as the primary interactive remote desktop, declared via
services.sunshine; Moonlight clients on Drew's devices. Acceptance spike before committing: confirm from Sunshine's log that a hardware encoder actually engaged on the laptop's GPU (thesoftwareencoder line is the silent-failure mode), and measure host CPU during a displayed 1080p session against RustDesk's 26–45%-of-a-core baseline. - RustDesk is demoted, not deleted: greeter/fallback duty + the design: remote desktop through the fleet brokers (RustDesk transport, secrets, pre-warming, virtual desktops) #5694 fleet-splice role, with Standing-load inventory 2026-11-04 (WS-C): residents measured — daemon dominates, RustDesk on-demand only, conn-watch stopped, sysstat keep #5775's idle auto-exit applied. It stops being the default interactive path.
- Sandbox (no GPU): no standing GUI streaming at all. The
amd-crdcontainer stays as break-glass. If a recurring sandbox-GUI need ever appears, spike KasmVNC or Selkies on-demand (both need packaging work — that cost is the gate) rather than running anything resident. - Guacamole, X2Go, NoMachine, AnyDesk: not adopted (gateway-not-encoder / unmaintained / proprietary-account surface, per the table).
Part A2 — agent side: stop eating pixels
The hierarchy, cheapest truth first:
- Structured APIs / CLI (fleet verbs, gh, files, Supabase) — zero pixels, exact answers. Most "GUI" agent tasks are actually this.
- CDP against the Helium browser (already live: 127.0.0.1:19222 on the laptop) — DOM-level truth, deterministic input, screenshots on demand.
- AT-SPI accessibility trees for native apps — semantic widgets/labels, no rendering, no encode.
- Screenshot on demand from whatever session exists — a single-frame encode costs milliseconds; a 30–60 fps stream costs the 26–45% core measured above. Pixels are for visual verification, taken as stills.
- A persistent virtual desktop — only when an app genuinely demands a real display/session.
Where a virtual desktop is needed, run it per-agent and headless: Xvfb (X11 apps) or headless sway (WLR_BACKENDS=headless, Wayland apps) — order ~10–30 MB RAM and ~zero idle CPU per session, versus a continuously encoded real desktop. Attach wayvnc to the headless sway session only when a human or agent truly needs a live view; otherwise the session is driven via CDP/AT-SPI/xdotool-class input and read via stills.
Recommendation — agent side: standardize on CDP → AT-SPI → stills, with Xvfb/headless-sway as the display of last resort. No agent consumes a standing video stream, ever; that single rule is worth more than any encoder choice in the table.
Links: #5694 (RustDesk fleet splice — this issue narrows its role, does not replace it), #5775 (idle auto-exit + standing-load rulings).
Acceptance criteria
- Candidates are scored against the stated constraints: GPU NixOS laptop (human client), GPU-less 4-vCPU sandbox (llvmpipe today), agent-driven use, and supervision/launch requirements.
- The RustDesk incumbent's measured cost (26–45% of a core, software GL + software encode) is the baseline every candidate is compared against.
- The evaluation gives a separate answer for the human client and for agent use — they may differ — each with expected CPU cost and evidence quality (measured vs claimed).
- It ends with a concrete next step: a named pilot (with what to measure) or an explicit stay-with-RustDesk recommendation.
Activity
developing-today commentedon Nov 5, 2026
CODE-1601
dezren39 commentedon Nov 5, 2026
Intake review
Related: Context: #5775 (WS-C measurements; carries the idle auto-exit ruling for the incumbent).
Still-live check: searched merged PRs referencing this issue at 2026-11-05T04:31:57Z — none found; the issue stands as filed.
Notes / risks: Vendor performance claims for GPU-less Linux are unreliable — prefer candidates that can be piloted and measured on the actual sandbox before any commitment.
Suggested approach: Shortlist from the constraint list first (Wayland/X, GPU-less encode path, headless/agent control), then pilot the strongest candidate side-by-side with RustDesk under the #5775 measurement method.
Acceptance criteria were appended to the issue body and labels normalised by the fleet monitor's intake pass.
Merge pull request #6088 from developing-today/feat/5924-sunshine-moo…
dezren39 commentedon Nov 5, 2026
Implemented — PR #6088, runbook at
doc/2026-11-04_remote-gui-access-runbook.md. Activation is staged-ready: everything up to the switch is done and verified; the switch itself waits for the desktop session to be free (it was actively in use tonight, and a switch can restart it).Flake (host
amd)nixos/hosts/amd-sunshine.nix:services.sunshineuser service,capture = "kms"(this host runs Hyprland/wlroots under SDDM),encoder = "vaapi",capSysAdmin,openFirewall = false— reachable over the trustedtailscale0interface only. Moonlight-Qt installed as client. Hardware path verified on the machine first: Radeon 780M,vainfoshows VAAPI H.264 + HEVC EncSlice.nixos/environment/default.nix: Helium now launches with--remote-debugging-port=19222(loopback) on every path — the installed Helium previously exposed no CDP port at all.scaleElectron2xwas generalized to do it; other apps unchanged.The encoder failure mode is worse than "silent" — and is now loud. In the pinned Sunshine source (
src/video.cpp,probe_encoders), a configured encoder that fails validation is logged and the probe falls through to software. The pin is a preference, not a guarantee. The guarantee issunshine-encoder-check(user oneshot + hourly timer): it fails intosystemctl --user --failedon a rejected pin, no working encoder, or a software winner (Found H.264 encoder: … [software]/Creating encoder [software]), using the source's exact log strings. Verdict logic tested against synthetic journals — all four cases (vaapi OK / software FAIL / pin-rejected FAIL / no-evidence inconclusive) behave correctly.Fleet-proxy verdicts (full study in runbook §4): streaming stays tailnet-direct — it's UDP (47998–48010) and a TCP CONNECT proxy can't carry it; WireGuard already does it best. CDP rides the CONNECT proxy (:3130) +
tailscale serve— proven live tonight. RustDesk stays on its dedicated SSH forward until the #5694 splice exists (the broker is request/response-shaped; it carries no byte streams). SSH stays onlaptop-ssh.sh. No live routing changed.CDP smoke test (passed): sandbox → runtime CONNECT proxy (:3130) →
100.99.113.103→ temporarytailscale servemapping → throwaway Helium on a scratch port:/json/version✓,/json/list✓, WebSocketRuntime.evaluate("1+1") → 2✓. Scaffolding removed; the live desktop browser was left untouched. Use the tailnet IP literal for CDP — Chrome rejects non-localhost DNS Host headers.Remaining = activation checklist (runbook §5), in order, when the desktop is free and no switch is running: switch from the primary → create the
tailscale serve19222 mapping → quit + relaunch Helium → Sunshine web-UI credentials → pair Moonlight → runsunshine-encoder-checkduring a real stream. That last step is the spike criterion from this issue, and it is the one verification that cannot be done before activation.Build: the full
nixosConfigurations.amdtoplevel closure builds clean on the merged tree, and the artifacts are in the product —sunshine.servicein the system user units (ExecStart = capability wrapper + renderedsunshine.confwithcapture=kms,encoder=vaapi), andsunshine-encoder-check.service+.timerin the home-manager generation.feat(browsers): structural CDP for Chromium and Brave on loopback (co…
dezren39 commentedon Nov 5, 2026
CDP-everywhere audit + fix (2026-11-05, extends this issue's runbook)
Drew asked whether every fleet browser always enables CDP at any boot. Audit of every host in the flake plus the darwin host, with fixes landed. Port convention (all loopback-only; remote path stays loopback +
tailscale serve/ CONNECT proxy per the runbook): 19222 Helium · 19223 Brave/Chrome · 19224 Chromium. No config anywhere binds CDP to a non-loopback address.NixOS hosts
amd+nixos(shared desktop profile; VM variants inherit) — PR #6123, merged 61d5424amd: process carries the flags, listener on 127.0.0.1:19222.chromiumCdp, a bin-only wrapper on 19224. Bin-only becausewrapElectronWithFlagsprovably fails on this package (it symlinksshare/applicationsinto chromium-unwrapped; the helper's desktop rewrite hits the read-only target) and no rewrite is needed (desktop file uses bareExec=chromiumvia PATH). Derivation built + wrapper inspected.braveCdpon 19223, binaries wrapped + desktop entries repointed. Derivation built + inspected.--remote-debugging-port(mcpx launches the system chromium by path) still win.--remote-debugging-porton current Firefox starts the WebDriver BiDi remote agent, not CDP. Gap, needs a decision if BiDi-based control is wanted.amdfrom a non-Nix install — a remote-desktop service, not a launchable browser, no CDP surface. No flatpak browsers; no browser autostart/systemd launch paths exist (launch paths are the wrapped binaries + their desktop entries).Darwin host MGM9JJ4V3R — dezren39/darwin-nix PR #1512, merged b2b2bc1f
google-chrome-cdppackage: a Google Chrome CDP app (linked into /Applications/Nix Apps on switch) +google-chrome-cdpCLI wrapper, both execing the cask binary with the flags on 19223. Residual gap by construction: launching the stock Chrome.app directly bypasses the wrappers. Firefox-family casks: same BiDi note as above.Activation
Nothing was switched or restarted anywhere. The NixOS changes ride the next system switch (standing switch order), the darwin change the next darwin-rebuild switch. Full toplevel builds of the NixOS change were starved by unrelated fleet builds (laptop load ~40) at merge time; the two new leaf derivations — the only new build inputs — are built and inspected, and the changed modules evaluate for both hosts.
dezren39 commentedon Nov 5, 2026
WebDriver BiDi follow-up — Firefox family enabled on all hosts (2026-11-05)
Companion to the CDP summary above. Drew's order (2026-11-05 01:52 CST): "sure enable bidi on all." The Firefox family has no CDP;
--remote-debugging-portstarts its WebDriver BiDi remote agent.Fleet port table (final, loopback-only everywhere): 19222 Helium CDP · 19223 Brave/Chrome CDP · 19224 Chromium CDP · 19225 Firefox BiDi · 19226 Zen BiDi · 19227 Zen Twilight BiDi (darwin only).
NixOS (host amd + VM variants) — PR #6146, merged
18f72361:symlinkJoinwrapper (firefoxBidi, port 19225) innixos/environmentsystemPackages and as home-managerprograms.firefox.package; rawfirefoxremoved from the user/user-for-vm/backup package lists (the wrapped system package serves all users). Desktop file uses bareExec=firefox→ PATH resolution covers launcher launches.zen-beta): same wrapper shape (zenBidi, port 19226), replacing the raw zen-browser package in systemPackages.--remote-debugging-portonly. The agent binds 127.0.0.1 itself (no address flag exists in the Firefox agent); no--remote-allow-origins— Firefox's default policy verified live on 159: foreign Host → 400, Origin-bearing upgrade → 400, localhost/no-Origin → 101.Darwin (MGM9JJ4V3R) — darwin-nix PR #1515, merged
c0884199:firefoxFamilyBidipackage in the #1512 launcher pattern —Firefox BiDi/Zen BiDi/Zen Twilight BiDiapps +firefox-bidi/zen-bidi/zen-twilight-bidiCLIs on ports 19225/19226/19227. Zen and Twilight share a profile dir/bundle id, so only one runs at a time and the distinct ports never collide in practice. Mac not contacted; activation rides Drew's next darwin-rebuild.Verification: NixOS wrappers built + scripts inspected; amd/nixos/vm_amd configs evaluate; HM package =
firefox-159.0-bidi. Live BiDi proof with isolated throwaway profiles through the built wrapper binaries:WebDriver BiDi listening on ws://127.0.0.1:19225(Firefox) and:19226(Zen), each answering a WebSocketsession.statusround trip — at path/session(101 +result.ready=true);/is only the httpd status page in current Firefox, which cost one debugging round. Darwin: config evaluates from Linux; installPhase simulated, generated launchers/Info.plists inspected byte-for-byte. No switch run anywhere; no running browser restarted. NixOS activation rides the same pending switch as the CDP/Sunshine work.Remaining gaps: darwin stock-app bypass (Dock/Finder/
open -alaunches of Firefox.app/Zen.app/Twilight.app skip the launchers — same as Chrome); Zen Twilight has no NixOS install (darwin only); Firefox-family on any future host needs the wrapper/launcher added there.dezren39 commentedon Nov 5, 2026
BiDi follow-up note (2026-11-05): the Firefox/Zen BiDi enablement from #6146 briefly broke main's Home Manager evaluation (#6156 — the bare
symlinkJoinwrapper handed toprograms.firefox.packagelacked theoverrideattribute home-manager's firefox module probes and calls). Fixed in PR #6173 (merged1a069c3ff2) by forwarding the underlying package'soverridethrough the wrapper — the BiDi feature survived the repair intact: same wrappers, same flags, same ports (Firefox 19225, Zen 19226), and the environment wrapper outPaths are byte-identical to the #6146 builds. The module-composed HM firefox package was verified end-to-end (BiDi flag on the exec line; the module's own policies baked into the wrapped package it installs).dezren39 commentedon Nov 5, 2026
Sunshine activation spike — complete (2026-11-05 ~08:35 CST)
Switches: primary restored to main first (it had been left on
bolt/issue-6182-streaming-fallback+ untracked claimant files; both preserved untouched). Three switches: gen 537 (main f86e532 — OpenClaw #6151/#6162 and BiDi #6146/#6173 landed), gen 538 (PR #6241), gen 539 (PR #6246, rc=0 clean). OpenClaw gateway verified healthy after every switch (unit active,/healthz={"ok":true,"status":"live"}); the #6167 post-activation proofs are still owed separately.Why Sunshine never started:
sunshine.serviceis a user unit (WantedBy=graphical-session.target); the running graphical session predated the deploy, so nothing pulled it in. No config change needed —systemctl --user start sunshinebrought it up active (running since 08:31:44 CST).Encoder verdict (the point of the pin):
sunshine-encoder-checkrun 08:32:42 CST, verbatim:OK: VAAPI hardware encoder confirmed in Sunshine's log.(Result=success). Sunshine's own probe agrees: capture=kms,Creating encoder [h264_vaapi](plus hevc_vaapi / av1_vaapi),vaapi vendor: Mesa Gallium driver 26.2.6 for AMD Radeon 780M Graphics (radeonsi, phoenix, ACO). No software encoder anywhere. The hourly check timer now has a running Sunshine to verdict.CDP over the tailnet: Helium CDP was already live on loopback with the #6123 wrapper flags. Added the missing serve mapping (serve state is imperative by design, never in the flake):
tailscale serve --bg --tcp 19222 tcp://127.0.0.1:19222, tailnet only; no serve added for 19223/19224 (runbook names only 19222). Proven end-to-end from the sandbox via the egress CONNECT proxy (hatch-egress-proxy:3130) against the IP literal:http://100.99.113.103:19222/json/versionand/json/listanswer (Chrome/154.0.8037.97). No Helium relaunch: its flags are verifiably live, so a restart would only disturb the open session.Quota drop-in: LEFT IN PLACE. The switched system opencode is 2.0.30+7774bf0, which predates fork PR #1722 — its subcommand list has no
quota(the fork-built binary at ~/git/opencode-builds/quota-13ab3dc5/ does). Removing the gatherer's QUOTA_SUBCOMMAND drop-in now would break the gatherer; it should come out when a system build carrying the quota subcommand ships.RustDesk assert detour (found, fixed, deployed): the #6140 oneshot failed its first two real activations (gens 537/538, exit 1 after 60 instant retries). Root cause, reproduced in the service's exact scrubbed environment:
runuserpreserves the caller environment, and from a system service that PATH is the unit PATH — the innerenv … sh -cdied withenv: 'sh': No such file or directorybefore fleet ever ran, with the fallback'sbwsunreachable behind it. Fixes: PR #6241 (unitpath = [ pkgs.bws ]) and PR #6246 (PATH pinned to bash+coreutils+bws store bins on the runuser env call itself). Gen 539: the assert ran to Result=success in ~1s and re-asserted the vault password (Done!).STOPPED at Moonlight pairing — Drew's act:
https://100.99.113.103:47990(Sunshine web UI; self-signed cert — accept the warning). The first visit asks you to CREATE the admin username/password (stored hashed under ~/.config/sunshine; none exists yet).100.99.113.103manually if it doesn't auto-discover.No client is paired yet, so no real-stream encoder test was possible; the startup probe + encoder check are the hardware-encoder evidence until a stream runs. Sunshine's ports are listening; openFirewall=false stands — reachable over the tailnet via the trusted tailscale0 interface. Do not open the firewall.
Process note: tool-output fabrication hit this lane twice (a fake "switch failed rc=2" background delivery and a fake PR number). Every consequential state above was re-verified with an independent second call.
dezren39 commentedon Nov 5, 2026
Sunshine admin account created — 2026-11-05
The spike left "create the Sunshine admin account" as a Drew step; Drew directed Muse to do it and vault the credential (main chat, 2026-11-05 ~09:30 CST).
drew, strong generated password (32 chars).sunshine --creds drew <new password>, service restarted. Stored username verified asdrewin sunshine_state.json afterwards.GET /api/configwith the new credentials returns HTTP 200; without credentials HTTP 401. Sunshine user service active.sunshine.amd-6-admin(id 9465a41d-c0cd-4948-bfb3-b4f90100fe1e) — JSON value with username/password/url. Read-back verified twice (on the laptop at creation, and again from the sandbox): username and password fingerprint match what was set.Sunshine (amd-6) admin(Dev vault, id qkbkymtcjludwsksaphe4p37ta), username + password fields; verified present viaop item get. Filed from the Bitwarden-retrieved value after a fingerprint match, so both vaults provably hold the same credential.