Repository navigation
github-backup(drain): monotonic-only timer can lose its next elapse after unit reload #5157
Description
Observed failure
On the laptop, github-backup-drain.timer last triggered at 2026-11-01 22:07:10 CST. At the 2026-11-03 verification it reported NextElapseUSecMonotonic=infinity, no realtime next elapse, Trigger: n/a, and the drain-service journal was empty. The timer unit was active but had no scheduled next run. The fetch and reconcile timers continued to fire.
Current unit configuration
At repository main (dcca6cd510866556a00dedb2c92b7d21cf817e2b), infra/github-backup/systemd/github-backup-drain.timer is monotonic-only:
[Timer]
OnBootSec=3min
OnUnitActiveSec=5min
Persistent=trueIt has no OnCalendar= trigger. The timer was reactivated at about 22:16 CST during the home-manager unit migration in #3720, so the timing is associated with that reload. However, the recorded last service trigger at 22:07 is recent enough that an OnUnitActiveSec anchor would ordinarily be expected. The available evidence does not establish the precise systemd state transition that produced infinity; confirm that mechanism during implementation rather than assuming the migration discarded an otherwise valid anchor. Persistent=true applies to calendar timers and does not itself provide a calendar trigger here.
Scoped fix and verification
Give the drain timer a calendar-based five-minute trigger, independent of the service's last activation, for example:
[Timer]
OnCalendar=*:0/5
OnBootSec=3min
Persistent=trueKeeping OnBootSec is optional if the calendar schedule alone is sufficient for the first run. Reload/restart the user timer and verify a finite next elapse plus one successful drain tick. Add a runbook or test check that every github-backup timer has a finite next elapse after unit installation/reload.
Relations
- Umbrella rollout tracker: github-backup: estate mirror system (laptop mirrors + dedicated App + Worker webhooks + weekly sweep) #3215 (the initial webhook proof is recorded there; this issue tracks the standing timer failure).
- Fold script-installed laptop user units into the flake (github-backup, worktree-reaper, quota exports, dashboard-backup, oc-password-sync) #3720 is the temporally associated home-manager unit migration; causation is not yet confirmed.
Acceptance criteria
github-backup-drain.timerhas a calendar-based five-minute trigger that is independent of the service's last activation (not monotonic-only).- After a unit reload / daemon-reload, the timer reports a concrete next elapse (never
NextElapseUSecMonotonic=infinitywith no realtime elapse) and the drain service actually runs on the cadence. - The mechanism that produced the infinity state is confirmed during implementation, or the calendar trigger is verified to fire across a reload regardless of that mechanism.
- The fetch and reconcile timers' behaviour is unchanged.
Activity
dezren39 commentedon Nov 4, 2026
Intake review
Persistent=on the calendar trigger will run a missed drain at boot — desirable here, but note it).OnCalendar=(5-minute cadence) alongside or instead of the monotonic anchors ininfra/github-backup/systemd/github-backup-drain.timer, reload on the laptop, and verify a next elapse plus an actual drain run.🤖 Fleet monitor intake pass, 2026-11-03 run.
dezren39 commentedon Nov 4, 2026
agent-note:v1
{"kind":"handoff","task":9463,"lane":"code-v","execution_lane":"opencode","state":"parked","note":"task 9463 (triage) handed off to opencode claimant by lane code-v. Fresh revalidation: issue #5157 OPEN, no hard-stop labels, no prior task progress — claimant triages the github-backup drain monotonic-only timer elapse loss from scratch."}
19 remaining items
dezren39 commentedon Nov 5, 2026
🤖 Agent note · handed off
ses_e5e563922fbffewkWbJ7at6yU8dezren39 commentedon Nov 5, 2026
Triage for task 9463: the defect remains actionable on current main.
gh apireports main at1cf701651abdbdfbbd016fc64fdf045999b26e52;infra/github-backup/systemd/github-backup-drain.timerstill hasOnBootSec=3min,OnUnitActiveSec=5min, andPersistent=true, with noOnCalendar=trigger. The reported active-but-unscheduled state (NextElapseUSecMonotonic=infinity, no realtime next elapse, empty drain journal) is documented in this issue; I did not independently reproduce it on the live laptop in this session. Fresh PR searches found no superseding fix. Scoped route: add a calendar-based five-minute trigger to the drain timer, then verify a finite next elapse and an actual scheduled drain run after reload; confirm fetch/reconcile behavior is unchanged and determine or work around the original monotonic failure mechanism. I am enqueueing that bounded follow-up now.dezren39 commentedon Nov 5, 2026
🤖 Agent note · picked up
github-backup-drain.timera calendar cadence (issue github-backup(drain): monotonic-only timer can lose its next elapse after unit reload #5157). Revalidated fresh: issue github-backup(drain): monotonic-only timer can lose its next elapse after unit reload #5157 OPEN, no hard-stop labels, no existing PR for issue github-backup(drain): monotonic-only timer can lose its next elapse after unit reload #5157, current main (1cf7016) still ships the monotonic-only timer (OnBootSec=3min+OnUnitActiveSec=5min, noOnCalendar=); sibling fetch/reconcile timers already useOnCalendar=. Live laptop timer currently has an anchor (next elapse 2026-11-04 21:46 CST) — the defect is the anchor being losable on reload, exactly as issue github-backup(drain): monotonic-only timer can lose its next elapse after unit reload #5157 records.OnCalendar=*:0/5on the drain timer in a merged PR, unit validated, deployed via install-units.sh from the primary clone, finite next elapse + a scheduled drain run verified.dezren39 commentedon Nov 5, 2026
🤖 Agent note · done
fix(github-backup): give drain timer a calendar cadence (#5157)
Merge pull request #5940 from developing-today/queue/9837-5157-drain-…
dezren39 commentedon Nov 5, 2026
🤖 Agent note · done (code side) — deployment remainder in darwin-nix
OnUnitActiveSec=5minwithOnCalendar=*:0/5ininfra/github-backup/systemd/github-backup-drain.timer(OnBootSec/Persistent retained) and updates the fleet-schedule registry row. Validation on the laptop:systemd-analyze verifyon the install-rendered units exits 0;systemd-analyze calendar "*:0/5"normalizes to*-*-* *:00/5:00with a concrete next elapse; Codex review on PR fix(github-backup): give drain timer a calendar cadence (#5157) #5940 completed with no findings. Fetch/reconcile timers untouched.install-units.shnow fails with "Read-only file system" and cannot deploy. The home-manager definition — dezren39/darwin-nixnixos/home/github-backup.nixlines 55–56 — still carriesOnBootSec="3min"+OnUnitActiveSec="5min"for the drain timer. Port the same change there (addOnCalendar="*:0/5", dropOnUnitActiveSec), run the home-manager switch on the laptop, then verify:systemctl --user show github-backup-drain.timerreports a finiteNextElapseUSecRealtimeand a scheduled drain run lands in the journal. Tracked as a SET-REMAINDER entry in the darwin-nix backlog (cross-repo set rule: no separate issue; this issue is the origin record).