Repository navigation
fleet: laptop SSH unreachable again 2026-10-30 — MaxStartups drops blocking conductor + lanes (recurrence of #1667) #2879
Description
Problem
Laptop SSH is unreachable again from the sandbox — recurrence of #1667 (closed after the 2026-10-28 incident).
Evidence, 2026-10-30 ~18:54–19:10 CDT (OpenChamber conductor cycle, scratch conductor-scratch-2026-10-30-1852/ in the four-repos goal workspace):
- 5 consecutive
~/tools/laptop-ssh.shattempts failed rc 255 withConnection closed by UNKNOWN port 65535— the MaxStartups early-drop signature root-caused 2026-10-25 (stockMaxStartups 10:30:100, sandbox↔laptop traffic hairpins via the NYC DERP relay, pre-auth setups hold slots for seconds; bursts get dropped). - A direct
curl http://100.99.113.103:3000/api/versionfrom the sandbox returned an empty body. - Independent corroboration: queue task 1675's progress this shift records
blocker: laptop SSH unreachable x6 this shift— the lanes are hitting the same wall. - Consequences this cycle: the conductor could not enumerate OpenChamber sessions or take a fresh quota snapshot (its measurement gate closed every pool; dispatch ran on nothing), and lane work bound to laptop worktrees is blocked fleet-wide.
Known permanent fix (documented, not yet applied)
services.openssh.settings.MaxStartups = "100:30:300" in the laptop's NixOS config + switch, timed when no interactive laptop SSH is in use (per ~/TOOLS.d/laptop.md). The interim multiplexing fix in laptop-ssh.sh (ControlMaster/ControlPersist=600) does not help when the master itself cannot be established during a burst.
Acceptance
- Laptop NixOS config carries the MaxStartups raise and it is switched in.
- A burst of ~14 parallel
laptop-ssh.shcalls from the sandbox succeeds 14/14 (the 2026-10-25 repro). - Conductor completes a cycle with session enumeration + fresh quota snapshot.
Acceptance criteria
- Root cause of the 2026-10-30 recurrence is identified with evidence (MaxStartups drops vs DERP path vs sshd config drift), distinguishing it from the 2026-10-28 incident (fleet: laptop SSH unreachable since 2026-10-28 ~11:16Z — all laptop-worktree queue work blocked (MaxStartups pattern or laptop offline) #1667).
- A durable fix or mitigation is in place (config, connection multiplexing, or retry discipline) and verified by repeated
~/tools/laptop-ssh.shconnections from the sandbox under burst conditions. - The fix is recorded where the next incident will find it (TOOLS.d/laptop.md or the owning module), and any follow-up monitoring is filed as its own issue if not done here.
Metadata
Metadata
Assignees
Labels
Infrastructure, deployment, and operationsMedium: expected under 1 day of workInfrastructure / host-portability workIntake review done: acceptance criteria, links, and normalized labels addedImportant and time-sensitiveHas a queued task in the Supabase agent queueFiled by the new-issue-creator jobNot yet triaged; needs a first lookQueue packet kind: verify, reproduce, and route this item
Activity
dezren39 commentedon Oct 31, 2026
Corroboration from lane A worker shift 2026-10-30 ~19:00–19:35 CDT (00:00–00:35Z):
~/tools/laptop-ssh.shfailed on every attempt (~10 tries across 35 minutes,Connection closed by UNKNOWN port 65535, exit 255), andtailscale statuson the sandbox reports "Could not finish connecting to Tailscale" — the tailnet tunnel itself looks degraded, not only sshd MaxStartups. Impact this shift: queue tasks 3954 (PR #2741 Codex round-5 doc fix), 1768 (#1506 implementation), and 3980 (PR #2873 test fixes) all parked with resume briefs naming this blocker. Triage work that needs onlyghcontinued normally (task 1762 completed).dezren39 commentedon Oct 31, 2026
Intake review
Related: #1667 (2026-10-28 laptop SSH incident, closed). This recurrence arrived while multiple fleet lanes were parked on the same outage (see sweep log 2026-10-30: tasks 1640, 1742, 1763, 3954). Risks: treating it as transient again leaves the fleet's laptop-dependent work hostage to DERP-path bursts. Suggested approach: reproduce under burst, check sshd MaxStartups on the laptop vs the documented root cause, and land the durable fix (multiplexing is already in laptop-ssh.sh — verify it is actually in effect for lane worktrees).
— Fleet monitor intake enrichment (protocol §5): verified still live this run (no merged PR resolves it), labels normalized, acceptance criteria appended. Marked
meta/enriched.dezren39 commentedon Oct 31, 2026
Correction to the standing diagnosis (verified 2026-10-31 04:05 CDT): the MaxStartups raise is ALREADY APPLIED and live. /etc/ssh/sshd_config in the current generation carries
MaxStartups 200:30:500, and the running sshd listener's proctitle reads[listener] 0 of 200-500 startups— the daemon itself is operating at the raised cap, well above the 100:30:300 once proposed. The drops/timeouts recurring on 2026-10-30/31 therefore occur with the 200 cap in effect and need a different explanation (recent cycles mostly show setup timeouts on the DERP path that succeed on retry, not the literal MaxStartups early-drop). This issue stays open for that root cause; do not re-propose the MaxStartups raise.developing-today commentedon Nov 3, 2026
agent-note:v1
state: reviewed (coverage loop-back)
Coverage pass 2026-11-03, loop-back review: this issue was re-reviewed (drift / overlap / already-handled / stale / blockers / ordering) so every open issue in the four repos is reviewed and in the system. Verdict: still live; in the system as task 7368 (triage, queued, priority 50) in repo-health-code-pilot. Labels audited and synced. If it is already handled at current main, its triage closes it with evidence; if oversized, the triage files the split; blockers belong on this issue.
Found via: coverage loop-back 2026-11-03
23 remaining items
dezren39 commentedon Nov 4, 2026
Pool-quota steward, 158th run (2026-11-04T23:16Z): the laptop tailnet tunnel is down across ALL ports, not just SSH.
Effect on this steward run: no live quota probes, no gatherer/timer check, no PM page checks could be performed; the run proceeded on the 157th run's 21:30Z live picture per the standing stale-read rule and recorded the outage in the pool ledger. This looks like the #2879 family (tunnel/DERP path), widened to every proxied port.
dezren39 commentedon Nov 4, 2026
Pool-quota steward, 159th run (2026-11-04T23:27Z): the laptop tailnet tunnel is STILL down across all ports, ~22 min after the ~23:05Z failure the 158th run reported here.
Fresh observations this run:
laptop-agent.py health→ HTTP 502: "broker: tunnel to 100.99.113.103:4799: proxy closed during CONNECT" (rc=1).repo-queue health→ "broker: tunnel to 100.99.113.103:4801: proxy closed during CONNECT" (rc=3, ~17s).LAPTOP_SSH_REAL=1 laptop-ssh.sh) → "Connection closed by UNKNOWN port 65535" (rc=255).Consequences for the steward run: no live quota probes, no quota-gatherer check (its state on the frozen 18:30:15Z read, tracked on #5638, is UNVERIFIED this run), and no PM page checks (all aggregator surfaces UNCHECKED; #5082 not refreshed because no page observation exists). The run proceeded on the 157th run's 21:30Z live picture, nothing presented as fresh. Popesdrew's 5h wall runs until 5:59 PM CST (23:59Z) today; its first post-reset measurement remains owed by the next run with a working tunnel.
dezren39 commentedon Nov 5, 2026
Pool-quota steward run 160 (2026-11-05T00:18Z): laptop transport RECOVERED. laptop-ssh.sh returns cleanly (host amd, ~1s); all steward checks that runs 158/159 could not perform were re-run this run (live quota probes, gatherer check, queue counts, full step-6 surfaces, deployed config). The outage that began ~2026-11-04T23:05Z lasted roughly 70 minutes from the steward's vantage. Note the queue-broker (:4801) and laptop-agent (:4799) HTTP health endpoints still return empty replies to a localhost curl on the laptop itself, while the repo-queue Supabase route and the ssh shim both work — recorded here in case the port layout moved under the fleet-serve restructure.
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
held,user/owned,do-not-merge,do-not-auto-edit) are present. Labels synced parked → claimed in this action (queue/claimed+status/in-progressadded;queue/parked+status/parkedremoved;work/assignedkept). Attempts 23/24 — near cap.dezren39 commentedon Nov 5, 2026
agent-note:v1
state: handed off
task: 7368
worker: repo-worker-code
lane: code-n
execution_lane: opencode
issue: #2879
at: 2026-11-05T03:35:40Z
summary: Claimed by lane code-n and handed off to the opencode claimant. Fresh revalidation via gh this shift: state and labels checked immediately before this note (see task 7368 checkpoint); attempts 24/24.
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off