design: bake the SSH recovery path into the fleet binary — orchestrate system ssh now, native russh deferred #5697

design: bake the SSH recovery path into the fleet binary

Links: epic #5412 · laptop-agent design #5343
Drew's end state: no shell scripts, no Python scripts — just the fleet
binary. This issue covers the recovery SSH path only (the daemon-down
escape hatch). The everyday path's Python is the sibling item: the
fleet laptop-agent client domain (see Ordering, below).

Context — the recovery path today (three of our files)

~/tools/laptop-ssh.sh (the shim) delegates exec-style calls to the
laptop-agent, but shapes the agent cannot express fall back to real SSH.
The fallback classes, from the shim's own header:

• LAPTOP_SSH_REAL=1 (explicit escape hatch / daemon-down recovery)
• no arguments (interactive session) or a leading ssh option
  (-L/-R/-W/-N/-t/-p/-l/-i/-o … — forwards and TTY allocation)
• stdin carrying data for the remote command (sudo -S feeds,
  cat > file pushes) — the daemon has no stdin channel

The fallback is ~/tools/laptop-ssh-real.sh, which execs system ssh:

exec ssh \
  -o "ProxyCommand=python3 $HOME/tools/laptop-proxy-connect.py" \
  -o "StrictHostKeyChecking=yes" \
  -o "UserKnownHostsFile=$HOME/.ssh/known_hosts_laptop" \
  -o "IdentitiesOnly=yes" -i "$HOME/.ssh/id_ed25519" \
  -o "ConnectTimeout=60" -o "ConnectionAttempts=3" \
  -o "ServerAliveInterval=15" -o "ServerAliveCountMax=8" \
  -o "ControlMaster=auto" -o "ControlPath=$HOME/.ssh/cm-%r@%h:%p" \
  -o "ControlPersist=3600" \
  "user@100.99.113.103" "$@"

…and ~/tools/laptop-proxy-connect.py is the HTTP CONNECT relay to the
runtime proxy (:3130), reading the proxy credential from HTTPS_PROXY
and sending it only inside the CONNECT request (the #948 fix; the pinned
host key in ~/.ssh/known_hosts_laptop is a single line:
100.99.113.103 ssh-ed25519 …).

Findings — most of this is already in the binary

1. The ProxyCommand already exists as Rust.
   pkgs/fleet/crates/fleet-proxy/src/stdio.rs is documented as
   "fleet proxy connect --stdio — the ProxyCommand relay:
   laptop-proxy-connect.py's job, fleet-shaped" — it opens the CONNECT
   tunnel (tunnel.rs: credential from HTTPS_PROXY, inside the
   request only, never argv, never logged) and splices stdin/stdout
   onto it. Defaults already line up with the SSH target:
   client.rs run_connect uses config.target_host() for --host
   (fleet-config DEFAULT_TARGET_HOST = "100.99.113.103") and
   args.port.unwrap_or(22). It is already SSH-tested the hard way:
   the hand-rolled pumps exist because splice(2) froze the SSH
   handshake mid-stream (fleet step 3: proxy domains — hello/capability frame, PATH shim regeneration, fleet init #5431), and the no-join teardown is the fleet-proxy(stdio): relay never exits after remote close while stdin stays open — up.join() waits on a stdin-blocked thread #5518
   fix. So the Python file is replaceable today by
   ProxyCommand=fleet proxy connect --stdio.
2. The codebase is deliberately synchronous and small.
   fleet-proxy's dependency list is fleet-registry, fleet-config,
   fleet-proxy-proto, serde, serde_json, libc — no async
   runtime, no crypto stack. The workspace release profile is a size
   contract (design note §2: connect variant target ≤1.5 MiB, hard
   budget ≤2 MiB, opt-level = "z", LTO, stripped). Any SSH-in-Rust
   route is measured against that contract.
3. System ssh is present and is an OS component: /usr/bin/ssh,
   OpenSSH 9.6p1 on this sandbox; NixOS provides it on the laptop side
   of life too. It is platform — like the kernel and libc the binary
   already relies on — not one of our scripts.

Route (i): native Rust SSH via russh — what "possible" honestly means

russh is the mainstream async Rust SSH library (client + server;
downstream clients ship PTY sessions, forwarding and known_hosts
checking on it). Checked against our exact uses (docs.rs, 2026-11-04):

Plus the build cost from Finding 2: russh pulls Tokio + a crypto
backend and a large transitive tree into a workspace whose connect
variant has a hard 2 MiB budget and a no-async design.

Verdict on (i): possible — genuinely — for the exec/stdin classes,
which are most recovery use. Not a drop-in for the full fallback
surface (interactive, forwards, arbitrary options, warm multiplexing)
without reimplementing a meaningful slice of the OpenSSH client.
The recovery path is the worst place in the system for parity gaps:
it is, by definition, the thing we reach for when the primary path is
already broken.

Route (ii): the binary orchestrates system ssh (recovery only)

A new verb (proposed: fleet proxy ssh -- [ssh args…], in the proxy
domain because it is stdio + tunnel config) that:

• builds exactly the argv laptop-ssh-real.sh builds today, except
  ProxyCommand=<this exe> proxy connect --stdio (path via
  std::env::current_exe — the pattern client.rs's spawn_serve
  already uses), with the same pinned UserKnownHostsFile, key,
  timeouts, ServerAlive and ControlMaster settings, target
  user@ + config.target_host();
• then exec()s ssh (process replacement, as the shell script's
  exec does): stdin/stdout/stderr, TTY, signals and exit code are
  OpenSSH's own, untouched — including every fallback class, by
  construction, and arbitrary user -o args pass through after ours.

What it buys: the entire #948-hardened, battle-tested-since-2026-10-26
semantics with zero reimplementation; warm ControlMaster included.
What it costs: a runtime dependency on an external ssh binary.

Is that within the spirit of Drew's end state? Yes. The end state
names our shell scripts and our Python scripts. Route (ii) deletes
all three from the path (laptop-ssh-real.sh, the Python ProxyCommand,
and — with the sibling item — the shim itself). OpenSSH is an OS
component shipped and maintained with the platform, in the same
category as the kernel, libc and the setsid the binary already calls.
"No external process at all" is a stronger requirement than the one
stated, and route (i) shows its price.

One honest caveat: recovery would depend on fleet proxy connect --stdio working. That dependency already exists in substance — the
CONNECT relay is the only road to the laptop — and the Rust relay is
smaller, frozen-er and better tested (#5431, #5518) than the Python it
replaces. Document the manual last-resort (system ssh with a hand-typed
CONNECT ProxyCommand, from this issue) in the runbook, not as a script
on PATH.

Recommendation — hybrid, staged

1. Now: route (ii). Implement fleet proxy ssh orchestrating system
   ssh with self-ProxyCommand. This achieves the stated end state for
   the recovery path in one small, low-risk step.
2. Deferred: route (i), exec-only scope, as an optional native mode
   of the same verb, triggered only by: a target host with no OpenSSH
   client, a concrete route-(ii) failure attributable to OpenSSH, or an
   explicit future requirement of zero external processes for a specific
   deployment. If built, its scope is exec/stdin/exit-code + pinned-key
   verification via check_known_hosts_path; interactive and forwards
   stay on system ssh until parity is proven, not asserted.

Ordering dependency

The laptop-agent client domain in the binary is still a stub:
in pkgs/fleet/crates/fleet/src/main.rs the domain falls through to
the capability stub (exit 69) for real verbs, and the shim's header says
its delegate flips from ~/tools/laptop-agent.py to
fleet laptop-agent when that domain lands. Sequence:

1. fleet laptop-agent client domain lands → removes the everyday
   Python (laptop-agent.py).
2. This item's fleet proxy ssh lands → removes the recovery
   shell + Python (laptop-ssh-real.sh, laptop-proxy-connect.py).
3. Only then can laptop-ssh.sh itself be deleted: its exec classes
   point at (1), its fallback classes at (2). Doing (2) first is fine;
   deleting the shim before both exist is not.

Next steps

1. Implement fleet proxy ssh per route (ii): fixed option block
   mirroring laptop-ssh-real.sh, self-ProxyCommand, argv passthrough,
   HTTPS_PROXY-unset error identical to today's.
2. Parity tests before any cutover: wrong-host-key negative test
   (security(aggregator): fleet-feed push exposes the proxy credential in process argv and disables SSH host-key verification #948), stdin-data test (sudo -S shape), bash -s script test,
   interactive -t smoke, -L forward smoke, second-call warm
   ControlMaster timing.
3. Flip the shim's fallback exec to the new verb; soak; then retire
   laptop-ssh-real.sh and laptop-proxy-connect.py.
4. Keep the russh evaluation above as the starting point if a stage-2
   trigger ever fires — do not re-litigate from scratch.