Design: fleet proxy get/put — file transfer over the agent channel #5759

Design: fleet proxy get / fleet proxy put — file transfer over the agent channel

Drew (2026-11-04): "Maybe we need a tool to make it easier to get and put files onto the laptop, like fleet proxy get, fleet proxy put, or something — basically SCP. I think you already built a more performant version of that using HTTP, and maybe this is just papering over that concept. Whatever you think is best."

Extends #5343 (laptop-agent; the daemon's Rust port is phase 3 there). Epic: #5412. Sibling design: #5697 (fleet proxy ssh — same "give the proxy domain the laptop verbs users actually reach for" logic).

Inventory: what file movement exists today (verified 2026-11-04)

There is no file transfer. Anywhere. The "more performant HTTP version" Drew remembers is the agent's exec transport — warm pooled CONNECT tunnels + the session broker (~0.4 s warm vs ~1.8 s cold SSH) — and it carries commands and JSON results only:

• Daemon (~/laptop-agent/laptop_agent.py on the laptop, 347 lines, laptop-agent.service): routes are exactly GET /health, GET /jobs, GET /jobs/<id>, POST /exec, POST /jobs, POST /jobs/<id>/kill. Every body is JSON (_body() parses JSON; _send() replies JSON). No file/blob route, no streaming body path.
• Rust fleet proxy: verbs are serve, connect, init only. The broker wire (fleet-proxy-proto) is 8-byte-BE-length-prefixed JSON frames, 64 MiB ceiling — a JSON channel end to end.
• How files actually move today:
  • SSH stdin pushes, deliberately left on the shim's real-SSH fallback: fetch-charge-inputs.py (cat > file.tmp && mv … over SSH stdin) and turso-atlas/scripts/atlas-backup.py. The daemon has no stdin channel, so these never migrated (shim inventory, design: laptop-agent — resident execution agent on the laptop (replace per-command SSH spawns) #5343).
  • Base64 re-expression through /exec for anything binary that must cross as a command result/argument — manual, per-call, no integrity or atomicity story.
  • Ad-hoc cat/mkdir -p shell-outs over the agent for text.

So the get/put verbs are not papering over an existing file channel — they create the channel the two named scripts have been waiting for.

Options

0. Client-only verb over /exec (chunked base64 appends via shell commands). Works against today's daemon with zero server changes. Rejected: it institutionalizes the hack inside the binary that is supposed to be the clean end state — a shell process per chunk on the laptop, quoting hazards on every path, no atomic commit, no resume except client bookkeeping, and it dies anyway when the daemon goes Rust.
1. Daemon file RPC over the existing channel (recommended). New daemon routes speaking the same framed-JSON/HTTP-JSON as /exec; the client chunks. Zero transport changes: rides the pooled broker tunnels, token auth, trip logging, and the proto-0 legacy path unchanged. The same routes are implemented by the Rust daemon role (fleet serve laptop-agent, design: laptop-agent — resident execution agent on the laptop (replace per-command SSH spawns) #5343 phase 3) when it lands, so the protocol outlives the Python daemon.
2. Raw octet-stream routes over a direct CONNECT tunnel (true streaming, no base64). The tunnel code exists (fleet-proxy tunnel.rs/stdio.rs), but it bypasses the broker's pooling/framing and needs the daemon to accept non-JSON bodies. Deferred as a profiling-driven optimization for multi-GB transfers (atlas dumps); v1's ~33% base64 tax on a tailnet path is not the bottleneck that connection setup was.

Protocol (v1, option 1)

All routes on the daemon (port 4799), token-authed like every route, JSON envelopes:

• POST /files/stat {path, sha256?: bool} → {exists, type, size, mtime, sha256?} — hashing is opt-in per call (hashing a 1.5 GB dump costs seconds; callers that need verification ask once).
• POST /files/begin {path, size, sha256, mode?} → opens a transfer temp sibling named deterministically from (path, sha256) — .<name>.fleet-part-<sha256[..12]> — so an interrupted client resumes the same transfer by re-issuing begin. Refuses when free space < size + margin.
• POST /files/write {path, sha256, offset, data_b64} → appends/verifies at offset in the temp; returns {size} (bytes durably written). Idempotent at a given offset (rewrite of the same offset with the same bytes is a no-op success) so client retries are safe.
• POST /files/commit {path, sha256} → daemon hashes the temp; on match: fsync, atomic rename over the target, apply declared mode; on mismatch: {error: "sha256 mismatch", actual} and the temp stays for inspection/resume. POST /files/abort removes the temp.
• POST /files/read {path, offset, length} → {data_b64, eof, size, mtime}. The get flow is stat (with sha256: true) → read chunks into a local temp → verify whole-file sha256 → rename locally. Symmetric failure semantics both directions.

Chunk size: 2 MiB raw (≈2.67 MiB base64 — far under the 64 MiB frame ceiling, small enough that a lost chunk is cheap). File size is unbounded by the protocol. Offsets are byte counts; resume = stat the deterministic temp, continue from its size.

Path handling: daemon normalizes, rejects NUL and non-UTF8; no confinement beyond the daemon user's own permissions (the daemon already executes arbitrary commands as that user — file routes are not a privilege change, and the design says so rather than pretending a sandbox).

Client verbs (Drew's naming, proxy domain)

fleet proxy put [--mode OCTAL] [--no-resume] <local-path | -> <laptop-path>
fleet proxy get [--mode OCTAL] [--no-resume] <laptop-path> <local-path | ->
fleet proxy put -r <local-dir> <laptop-dir>     # recursive: client-side walk,
fleet proxy get -r <laptop-dir> <local-dir>     # per-file transfers

• - = stdin/stdout streaming: no resume, no temp (streamed chunks in order), sha256 still computed and printed to stderr at completion.
• Progress and the final sha256 go to stderr; stdout carries data only for get -.
• Directory recursion is per-file (each file resumable/atomic independently); empty directories are not preserved in v1 (noted, not hidden); symlinks are not followed by default.
• Mode: put sends the local file's mode in begin unless --mode overrides; default 0644 when the source has no mode (stdin). Ownership is always the daemon user (the daemon is unprivileged — no chown exists to offer). Timestamps are not preserved in v1.
• Exit codes follow the workspace convention: 2 usage, 3 transport, 69 capability-unavailable.

Registry / capability placement

• Subcommands get/put join the proxy domain in fleet-registry (registry data, like serve/connect/init).
• Capability: the verbs are proxy-client surface, so the natural home is proxy-connect — which is compiled into every variant, including the D7-size-gated connect builds (≤ 2 MiB). Chunking + sha256 in std-only Rust is small, but this must be measured at implementation: if the connect-static gate breaks, the verbs move behind a proxy-files feature present in full (and absent from connect), and connect builds answer the established exit-69 stub. Either outcome is a normal registry decision, not a redesign.
• Implementation home: the fleet-proxy crate's client half, beside connect — it already owns broker negotiation, the legacy proto-0 fallback, and tunnel construction. The daemon client types shared with the laptop-agent domain should be factored rather than duplicated when the second consumer appears.

Daemon side: Python now, Rust by construction

The Python daemon is to-be-deleted surface (#5343), so the question is whether interim routes there are worth it. Yes: the patch is ~100 additive lines on a 347-line file, the protocol is the durable artifact (the Rust role implements the same routes), and the two scripts blocked on a file channel (fetch-charge-inputs.py, atlas-backup.py) are live production paths, not hypotheticals. Their re-pointing off SSH stdin is a follow-up under their own owners once the verbs ship.

Testing / build

• Workspace unit tests: chunk math, offset idempotency, resume-from-temp, sha256 mismatch refusal, path normalization.
• smoke.sh: hermetic broker round-trip — put + get a multi-chunk file through a real broker against a fake daemon; byte-identical round trip; interrupted put resumed to completion.
• Ships in the existing pkgs/fleet Nix package (all variants) — no new build machinery; deployed copies flip by the established pattern.

Non-goals (v1)

No delta/rsync algorithm, no compression, no directory sync semantics (no delete-mirroring), no multi-file archives, no timestamp preservation, no cross-host (non-laptop) targets — the verbs speak to the daemon the proxy already reaches.