Repository navigation
github-backup: estate mirror system (laptop mirrors + dedicated App + Worker webhooks + weekly sweep) #3215
Description
System of record: the docs in this repo — infra/github-backup/ (README + docs/RUNBOOK.md). This issue summarizes and tracks rollout; the repo teaches.
Related: #3205 (per-repo hygiene census / standard task).
What it is
Bare-mirror backup of Drew's GitHub estate on the laptop (~/backups/github/<owner>/<repo>.git), decided 2026-10-31:
- Scope: Tier C — all non-forks + forks <500 MB across 17 owners + force-includes (818 mirrors); nixpkgs family stored once via git alternates pool; 7 passive giants excluded by ruling (upstream is their backup); gaps recorded (dezren39/edsebooks HTTP 451, vertexinc SAML).
- Freshness: dedicated GitHub App hatch-repo-backup (metadata:read + contents:read; push + repository events; its own rate-limit pool) → Cloudflare Worker receiver (HMAC-verified, KV queue) → laptop drainer every 5 min (15-min per-repo coalescing).
- Floors: daily fetch of all mirrors; weekly reconciliation sweep (new repos auto-mirrored per config tier rules; new giant forks land as pending-Drew-review).
- Secrets: Bitwarden by name only (
github-app.backup-*), never in repo/tree/chat.
Rollout status
- System code + config + units + Worker + runbook — PR github-backup: estate mirror system (scripts, units, Worker, runbook) #3214
- PR github-backup: estate mirror system (scripts, units, Worker, runbook) #3214 merged; units installed on laptop from ~/git/code
- Initial clone fleet complete (818) + MANIFEST/FAMILIES verified
- Worker deployed (wrangler; KV namespace id into wrangler.toml)
- App created (one human click — manifest flow in RUNBOOK §3) + webhook secret set both sides
- App installed on dezren39 + admin orgs (install-report verified)
- End-to-end webhook proof (ping + one real push fetch)
- First weekly reconcile report reviewed
Restore/DR procedures: RUNBOOK §8–§9.
Acceptance criteria
- Every rollout checklist item above is complete: PR github-backup: estate mirror system (scripts, units, Worker, runbook) #3214 merged, units installed on the laptop from ~/git/code, the initial clone fleet (818 mirrors) verified against MANIFEST/FAMILIES, the Worker deployed, the hatch-repo-backup App created and installed, and an end-to-end webhook proof (ping + one real push fetch) recorded.
- The first weekly reconcile report has been reviewed and any gaps it names are filed or recorded here.
- Restore/DR procedures (RUNBOOK §8–§9) have been verified against the live mirrors.
Metadata
Metadata
Assignees
Labels
Large: multi-day or open-ended; expect staged check-insInfrastructure / host-portability workIntake review done: acceptance criteria, links, and normalized labels addedNice to have; do when convenientQueue worker lifecycle stateFiled directly by a humanQueue worker lifecycle stateQueue packet kind: verify, reproduce, and route this itemQueue worker lifecycle state
Activity
dezren39 commentedon Oct 31, 2026
Rollout update (2026-10-31 16:05 CDT):
{"events": 1, "fetched": ["NixOS/nixpart"]}, exit 0. Secrets sourced from Bitwarden by name via the wrapper.github-app.backup-*) + Worker secret store.python3 bin/app_setup.py convert <code>→ set the webhook secret on the App (value in Bitwarden: github-app.backup-webhook-secret) → install on dezren39 + admin orgs →app_setup.py install-report. Until then the drainer simply finds an empty queue; daily fetch + weekly sweep already keep the tree current.dezren39 commentedon Nov 1, 2026
Status update (2026-10-31, verification close-out + Drew's naming/deploy directives):
Clone fleet: complete and verified — 818/818 mirrors on the laptop (29 GB tree), MANIFEST regenerated (818 mirrored, 0 missing), fsck sample 5/5 clean, working checkout cloned from an alternates member, fleet HEADs consistent (nix exact; hatch-personal/code are ancestor-stale by 35/44 live commits, closed by the daily fetch). Two retry passes with the system's own
clone_all.pycleared all 10 pass-1 failures (all network mid-transfer drops). Checklist boxes for PR merge, units installed, and clone fleet are therefore now true.Naming (Drew, 2026-10-31): the public name is
developing-today-backup. The GitHub App was never created under any name (API 404 for both slugs; no app-credential secrets in Bitwarden), so it will simply be created under the new name — the slug is available. The Worker WAS live ashatch-repo-backup-webhook; PR #3280 renamed it repo-side and redeployed it as developing-today-backup (live, version taggedgit-307a2501ca65), the laptop drainer was repointed via Bitwardengithub-app.backup-worker-urland verified on its next tick, and the old Worker was deleted (old URL 404s).Deploy method (Drew, 2026-10-31): before #3280 the Worker deployed by bare wrangler commands from the runbook. It now follows the repo deployment standard's shape:
just deploy-developing-today-backup→nix run .#developing-today-backup-deploy→ secrets from Bitwarden/SOPS → version upload (git-SHA tag) → separate deployment at 100%, withgithub-backup-units/github-backup-fetch/github-backup-reconcilerecipes alongside. Tool gap recorded in #3280: the deploy script drives wrangler's versions commands, not the standard's cf CLI, because this Worker is wrangler.toml-shaped and dependency-free while cf's flow requires the TypeScript cf/config format.Still open: the App creation click sequence (RUNBOOK §3, now for
developing-today-backup), App installs, the first genuine GitHub webhook delivery proof, and defects #3219 / #3229 (verified still present in deployed + main code).dezren39 commentedon Nov 1, 2026
🤖 Design-architect pass 29 (2026-10-31) — routing note: the eight defect filings against this system from its first day (#3219, #3229, #3233, #3238, #3240, #3250, #3257, #3270) are unified under design #3282, "a mirror is what it can prove — one validity, outcome and alarm contract for the github-backup estate mirror" (design doc: files/designs/code-github-backup-mirror-contract.md in the keep-the-four-repos-healthy goal workspace). The eight are adopted as #3282's phases (no parallels); a closing audit (#3283, gated) will post completion evidence here. One rollout interaction to expect: when Phase 1 (#3238's resolvability predicate) lands, the first reconcile/fetch afterwards will recount any bare-skeleton mirrors as missing and re-clone them — while the initial 818-mirror clone is still in flight that recount is the correction working, not a regression. This umbrella's rollout checkboxes are unaffected.
dezren39 commentedon Nov 1, 2026
Intake review
Still live: rollout is mid-flight — PR #3214 is the system PR named in the checklist, and merged PR #3280 (github-backup rename/deploy standard) is a follow-on, not the rollout completing. No merged PR closes this tracker.
Related: #3205 (per-repo hygiene census / standard task); PR #3214 (system code + config + units + Worker + runbook); PR #3280 (follow-on rename/deploy).
Notes / risks: the remaining checklist is mostly laptop-side state (units installed, 818-mirror fleet, Worker deployed, App installed, webhook proof) — verifiable only from the laptop and the Cloudflare dashboard, not from this repo. The App creation click and org installs are the human-gated steps.
Suggested approach: work the checklist in order; record the end-to-end webhook proof and the first weekly reconcile review as comments here.
Labels applied: kind/infrastructure, source/human, status/needs-triage, effort/l, meta/enriched (priority/low already present). Acceptance criteria appended to the body, drawn from the rollout checklist.
dezren39 commentedon Nov 1, 2026
App-creation attempt via RustDesk (2026-10-31 evening): blocked before any GitHub submission — the App does not exist yet.
What happened: the sandbox-side RustDesk client (1.4.5, nix build) cannot render its Flutter UI in this sandbox. Every GL configuration fails at GTK GLArea init — verbatim:
Failed to initialize GLArea: No available configurations for the given pixel format(X11) /No GL implementation is available(Wayland). Verified with an independent system-libGL probe: the sandbox's Xvfb (with GLX enabled) advertises GLX 1.4 with 80 fbconfigs and the Ubuntu Mesa client works against it — but every nix-linked GL client (the RustDesk closure, nix glxinfo) fails identically on all servers tried (system Xvfb, TigerVNC Xvnc, nixpkgs Xvfb ±iglx — the last two don't advertise GLX at all). Root cause: the nix Mesa/GLVND stack cannot create GL contexts in this sandbox (the RustDesk closure ships libglvnd with no usable vendor path here); system Mesa can't be mixed into nix-libc processes (GLIBC_PRIVATE symbol clash). A headless wlroots/cage + EGL attempt hit the same wall. No RustDesk session to the laptop was ever established, so no browser step ran and nothing was submitted to GitHub.Prepared and in place for a retry (e.g. from a client with a working GL stack, or Drew at the laptop):
dezren39 commentedon Nov 1, 2026
GitHub App created + installed + webhook proof — 2026-10-31 (CDP session on laptop)
App:
developing-today-backup, App ID 5562183, owner @dezren39. Created via the manifest flow (manifest verified byte-exact: metadata+contents read-only, events push+repository, webhookhttps://developing-today-backup.developing-today.workers.dev/github). Sudo cleared via GitHub Mobile approval (match number relayed live).Credentials — recovered from a convert crash (bug filed by this comment)
app_setup.py convertexchanged the code fine but crashed storing the private key:bws_setpasses the value as a bare CLI argument and the Bitwarden CLI's parser rejects values beginning with-(the PEM header) —error: unexpected argument '-----BEGIN RSA PRIVATE KEY-----'. app-id / client-id / client-secret were stored before the crash; the conversion PEM was unrecoverable (GitHub shows it once).Recovery, completed same session:
SHA256:j5HZDaG+2pWAuVbNdTwuNE4M8WpazhQqXcRZCNvD8lc=.github-app.backup-private-keyusing a--argument separator (probe secret round-tripped byte-exact first); stored value sha256 matches the downloaded file (43f2eb26…); download shredded.SHA256:8qYjT+…); exactly one key pair remains.app_setup.py: makebws_setinsert--before the value (or usebws secret edit/createvalue-safe form) soconvertcan't crash on PEM storage again.install-reportwas run as an equivalent stdlib+openssl RS256 mint →GET /app/installationsreturned 200.Webhook secret
app_setup.py webhook-secretrun (fresh value in Bitwarden), the same value set on the App (General → Webhook → Change secret; save confirmed), andjust deploy-developing-today-backupredeployed the Worker (versionacbb41a4, taggit-efc4e182d400) re-setting Worker secrets from Bitwarden./healthok.Visibility change — decision flagged
The manifest's
public: falsemakes the App installable only on the owner account (GitHub routes org install attempts back to the personal installation; the Install App chooser listed only dezren39). To complete the planned org installs the App was made public (Advanced → Make public). Consequence to note: any GitHub account can now install it on their own repos; such deliveries are signed with the App's webhook secret, so the Worker would accept and queue foreign repos. Recommend the drainer enforce the owner allowlist (the reconcile plan is already owner-scoped). Flip back to private is one click in Advanced if ever desired — but org installs require it to stay public.Installations — 16/16, all "All repositories" (verified via App JWT)
dezren39 177385379 · 3d0d 177388121 · Cloistered 177388175 · chip-rom 177388265 · chiplox 177388292 · Clustered-Computing 177388628 · IDEXX-UTDesign 177388652 · SpongeScribe 177388683 · developing-today 177388707 · developing-today-fork 177388738 · developing-today-forks 177388761 · dezren3 177388791 · dezren39-dev 177388812 · drewry-pope 177388833 · sleep-atomic 177388857 · twitter-base 177388879.
Sweep-only (no admin): NixOS, vertexinc — covered by reconcile polling as designed.
Webhook proof (timestamps CDT, 2026-10-31)
2bd5d85e-d5a4-11f1-82c5-8d7b8483596f(dezren39/openclaw-personal) 22:28:25 → Response 202 from the Worker (signature verified with the synced secret).Laptop helper servers (form :8766 / capture :8765) shut down. Session notes: the laptop network flapped repeatedly tonight (ERR_NETWORK_CHANGED) — several page reads and one install confirmation were lost to it, all recovered by retry; drewry-pope's first install attempt actually completed despite the error page (installation 177388833).
dezren39 commentedon Nov 1, 2026
Update (2026-10-31 23:12 CDT): the drainer owner allowlist flagged as owed at App creation is done — #3326, PR #3328 (merged
72e347609), deployed to the laptop, and proven live with a hand-injected foreign webhook (DRAIN-DROPlogged, no clone attempted, drain Result=success). The public-App foreign-install exposure is closed at the drainer; details on #3326.9 remaining items
dezren39 commentedon Nov 3, 2026
🤖 Agent note · handed off (task 8202)
triage-verified(checkpoint + findings on task 8202). Remaining steps only:~/hatch/state/queue-8202-drain-timer-issue.md(github-backup-drain.timer monotonic-only schedule lost its anchor in the Fold script-installed laptop user units into the flake (github-backup, worktree-reaper, quota exports, dashboard-backup, oc-password-sync) #3720 home-manager migration;NextElapseUSecMonotonic=infinity, last trigger 2026-11-01T22:07:10Z, webhook drain path dead ~2 days; scoped fix:OnCalendar=*:0/5, daemon-reload/restart, assert finite NextElapse). Dedupe against open issues first.triaged.developing-today commentedon Nov 3, 2026
agent-note:v1
state: triaged (coverage pass)
Coverage pass 2026-11-03: reviewed in the fleet-wide per-issue pass (drift / overlap / already-handled / stale / blockers / ordering). Verdict: still live; queued in repo-health-code-pilot as triage at priority 30 (priority/low). If this is already handled at current main, the triage closes it with evidence; if it needs splitting, the triage files the split.
Found via: coverage pass 2026-11-03
dezren39 commentedon Nov 3, 2026
agent-note:v1
state: handed_off
task: task 8202 (triage, repo-health-code-pilot, lane code-l)
worker: repo-worker-code
issue: #3215
Execution lane assigned: opencode. Handed off to the laptop claimant. Resume context (from the prior attempt's recorded findings): triage evidence is gathered — umbrella still live, 818 mirrors verified, one new defect found (github-backup-drain.timer monotonic anchor lost in the #3720 home-manager migration; drain path dead since 2026-11-01). Remaining steps: file the prepared drain-timer defect issue (~/hatch/state/queue-8202-drain-timer-issue.md), post the triage evidence comment here, complete task 8202 as triaged.
dezren39 commentedon Nov 3, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 3, 2026
Launched queue task 8202 for execution in the bound OpenCode session
ses_e5e5cb53c62ffeGclCLkRvFj72.dezren39 commentedon Nov 3, 2026
Triage at current main — 2026-11-03
Still live; do not close. Verified issue state OPEN via GitHub and compared repository
main(dcca6cd510866556a00dedb2c92b7d21cf817e2b). PR #3214 and follow-on #3280 are merged; the dedicateddeveloping-today-backupApp is installed on 16/16 targets and its ping/push→Worker→laptop-drain proof was recorded above. Laptop verification recorded 818 mirrors, 0 missing, and the 2026-11-02 reconcile report (one inventory error:dezren39SAML). The current acceptance criteria are not complete.New live blocker: standing webhook drain is not running
The laptop’s
github-backup-drain.timerlast triggered 2026-11-01 22:07:10 CST; at the 2026-11-03 verification it showedNextElapseUSecMonotonic=infinity,Trigger: n/a, and an empty drain-service journal. This coincides with the home-manager unit migration in #3720. The evidence and scoped remediation are filed as #5157: replace the monotonic-onlyOnBootSec+OnUnitActiveSecschedule with a calendar schedule (OnCalendar=*:0/5), reload/restart the user timer, verify a finite next elapse and a successful drain tick, and add a runbook/test assertion that timers retain a finite next elapse. Search found no existing issue for this distinct timer regression; it is not covered by the phases of #3282.Order: address #5157 before treating webhook freshness as continuously operational or closing this umbrella. No code-level dependency blocks that scoped fix; completion evidence requires the laptop unit update/reload and observed drain tick. Then finish the remaining umbrella acceptance: review the first weekly reconcile report and file/record any gaps; verify restore/DR procedures in RUNBOOK §§8–9 against live mirrors; and complete closing audit #3283, which is gated on the #3282 phases. The inventory error and the fact that the 818-mirror count alone is not restore/DR proof remain visible until those checks are done.
dezren39 commentedon Nov 3, 2026
Correction / precision on #5157: the live failure evidence remains
NextElapseUSecMonotonic=infinity, no next realtime trigger, and no drain-service journal entry. The reported last service trigger (22:07) precedes timer reactivation (about 22:16) by only nine minutes, so anOnUnitActiveSecanchor would ordinarily be expected. I updated #5157 to treat the home-manager migration as a temporal association, not a proven cause, and to leave the exact systemd transition for confirmation during implementation. The calendar-trigger fix remains scoped because it supplies a future schedule independent of that anchor.dezren39 commentedon Nov 3, 2026
Current-main recheck before triage completion: remote
mainadvanced to910213507772c55c6fee21085fcc651fb4acfe05(merge PR #5159) while this pass was in progress. I fetched the timer file and confirmed it is unchanged:OnBootSec=3min,OnUnitActiveSec=5min,Persistent=true, noOnCalendar=. Thus #5157 remains live at the latest main; the earlier commit SHA in my first triage note was the main tip at the time of that check.