fleet infra: laptop /home at 100% (5.6M free) — cargo verification fails with os error 28

Fleet infra — laptop (amd-6) /home is 100% full; local build/test verification is broken.

Evidence (2026-11-01 ~06:28Z, queue task 5073 shift, via ~/tools/laptop-ssh.sh):

df -h /home/user: /dev/nvme0n1p2 3.3T 3.2T 5.6M 100% /home
cargo test --lib petname in a fleet worktree (~/git/fleet/5073/pkgs/id) failed across many crates with No space left on device (os error 28) writing to target/debug/deps — a disk failure, not a test failure. PR fix(id): petname skeleton — capital-I class and 8->b, labels() skeleton collisions (#2069) #3360 was merged on other verified evidence (Codex re-review clean); this issue is about the verification capability, not that PR.

Impact: every fleet lane that verifies on the laptop (the standing rule: builds/tests run on the laptop or CI, never the sandbox) can silently lose its local verification path; failures look like compile errors unless the os error 28 line is spotted.

Acceptance:

Identify the top space consumers under /home (du report attached to this issue).
Free space safely — candidates named explicitly before any removal; no recursive/wildcard deletes of shared trees (standing delete guardrail); Drew approves anything not obviously disposable (stale target/ dirs in dead worktrees, caches).
Confirm with a fresh df (<90%) and one successful targeted cargo test in a fleet worktree.
If a recurring consumer is found (e.g. unbounded target/ growth across ~/git/fleet worktrees), add a bounded cleanup or alert so this is caught before 100%.

Update 2026-11-01 ~06:40Z (lane D, task 2663 shift): fresh df -h /home on the laptop now shows /dev/nvme0n1p2 3.3T 2.0T 1.2T 63% — ~1.2T free, recovered from 5.6M free at 06:28Z. Cause of the recovery is unknown to this lane (no cleanup was run by this shift). The 100%-full observation and the os error 28 cargo failures at 06:28Z were real; the remaining acceptance is the root-cause / recurring-consumer item — what filled it and what consumed/freed ~1.2T — so the next fleet infra pass can prevent a recurrence.

🤖 Agent note · update

Agent: queue worker (repo-worker-code), lane A
Session / ID: queue task 4925, repo-queue-worker-shift lane A 2026-11-01 01:04 CDT
Found via: hit the same disk-full during task 4925 verification, then observed recovery
Update: At 06:23Z a write in the fleet worktree failed with ENOSPC and df showed /home 100% (0 free), matching this issue. At ~06:25Z the laptop's ~/.local/share/Trash (measured 1.2T minutes earlier) was emptied — not by this lane — and df -h /home at 06:25Z showed 63% used / 1.2T free. Laptop builds/tests then ran fine (bun suite executed 06:27–06:30Z). The immediate blockage has cleared; the underlying cause (Trash growing to 1.2T) remains worth addressing, so leaving the issue open for Drew's call on trash retention/purge policy.

agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 40 (priority/normal). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03

🤖 Agent note · handed off

Agent: fleet monitor
Session / ID: task 13577 (repo-fleet-monitor run 2026-11-08 16:12 CST)
Found via: monitor reconciliation
Task: Claim evaporated — task 13577 is queued and unclaimed (no live lease) while the issue still carried claimed labels. Labels reset to the truth: queue/queued + status/needs-triage. Checkpoint stage at evaporation: triage_verification_running.
Expectation: s — a lane re-claims task 13577 and posts a fresh pickup note.
Links: task 13577

🤖 Agent note · state: picked up

Agent: queue worker (repo-worker-code), lane code-d
Session / ID: task 13577
Found via: queue claim (execution lane: direct)
Task: Resume, not restart (runbook section 10 rule 4): task 13577 was already triaged in full by lane code-l earlier today — findings in comment 6727594479 (outcome triaged; acute disk-full condition gone; remainder tracked by issue Smart janitor: recurring laptop disk/worktree hygiene (auto-delete stale target dirs, auto-push-and-archive aging worktrees) #6884 and issue infra: laptop /home at 100% — fleet worktree cargo targets fill the disk, builds die with ENOSPC #6797; no new task enqueued) — but lane code-l lost the claim at the complete verb and the fleet monitor re-queued the task (note at 2026-11-08T22:36Z). This lane re-verifies the state fresh and executes only the remaining closing verbs.
Expectation: s — fresh df verification, then complete task 13577 with outcome triaged.
Links: issue fleet infra: laptop /home at 100% (5.6M free) — cargo verification fails with os error 28 #3372; issue Smart janitor: recurring laptop disk/worktree hygiene (auto-delete stale target dirs, auto-push-and-archive aging worktrees) #6884; issue infra: laptop /home at 100% — fleet worktree cargo targets fill the disk, builds die with ENOSPC #6797

🤖 Agent note · state: done

Agent: queue worker (repo-worker-code), lane code-d
Session / ID: task 13577
Found via: queue claim (execution lane: direct)
Task: Task 13577 completed with outcome triaged. The triage findings stand as recorded by lane code-l in comment 6727594479: the acute disk-full condition on issue fleet infra: laptop /home at 100% (5.6M free) — cargo verification fails with os error 28 #3372 is gone, bounded cleanup landed in merged PR infra: cap fleet Cargo target storage with a safe janitor (#3448) #4175, and the remaining operational questions are already tracked by issue Smart janitor: recurring laptop disk/worktree hygiene (auto-delete stale target dirs, auto-push-and-archive aging worktrees) #6884 and issue infra: laptop /home at 100% — fleet worktree cargo targets fill the disk, builds die with ENOSPC #6797 — no new task enqueued. Fresh re-verification by this lane at 2026-11-08T23:10Z: laptop df /home = 600G free, 82% used (bar: under 90%), issue fleet infra: laptop /home at 100% (5.6M free) — cargo verification fails with os error 28 #3372 OPEN with no hard-stop labels. Issue fleet infra: laptop /home at 100% (5.6M free) — cargo verification fails with os error 28 #3372 stays open as the infra tracker; its queue/work/status lifecycle labels are cleared with this completion.
Expectation: done — no further action on task 13577.
Links: issue fleet infra: laptop /home at 100% (5.6M free) — cargo verification fails with os error 28 #3372; issue Smart janitor: recurring laptop disk/worktree hygiene (auto-delete stale target dirs, auto-push-and-archive aging worktrees) #6884; issue infra: laptop /home at 100% — fleet worktree cargo targets fill the disk, builds die with ENOSPC #6797; PR infra: cap fleet Cargo target storage with a safe janitor (#3448) #4175

Navigation Menu

fleet infra: laptop /home at 100% (5.6M free) — cargo verification fails with os error 28 #3372

58 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

fleet infra: laptop /home at 100% (5.6M free) — cargo verification fails with os error 28 #3372

Description

Activity

dezren39 commented on Nov 1, 2026

dezren39 commented on Nov 1, 2026

developing-today commented on Nov 3, 2026

58 remaining items

dezren39 commented on Nov 8, 2026

dezren39 commented on Nov 8, 2026

dezren39 commented on Nov 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions