Repository navigation
chore(deps): nixpkgs update blocked 2026-10-30 #2715
Description
chore(deps): nixpkgs update blocked 2026-10-30
Scheduled nixpkgs-update-amd run, 2026-10-30. The update was attempted on branch chore/nixpkgs-update-2026-10-30 (left pushed, no PR — the update does not build). Worktree was on the laptop (~/git/fleet/nixpkgs-update-2026-10-30, removed after the run).
What moved
just update-nixpkgs updated all 4 nixpkgs inputs, 0 failed:
nixpkgs-master:c61a74e1e2c8c75d774d7883927d3d21cb7d4e90→086873f31aea9365cb77f2df5620aadd1ac74accnixpkgs-stable:cfdb2236cff2a6adaace5a4ef37ea73152d710e9→086873f31aea9365cb77f2df5620aadd1ac74accnixpkgs-unstable:6510408d147d8c5e6d76043e899e9251756f6a75→31ecbaca1bdefa45dffc409f0250bd8039ea0010nixpkgs-25:e7439b6b14ad3cc35d05608ebca9bce01a25f5f8→31ecbaca1bdefa45dffc409f0250bd8039ea0010
Commits on the branch:
a5983341— chore(deps): nixpkgs update 2026-10-30 (flake.lock only)aec3998e— fix(deps): bump impermanence (see below)
Fixed along the way (eval)
First eval failed: the pinned impermanence rev 7b1d382f reads config.boot.initrd.systemd.enable at its nixos.nix:495, and nixos-unstable 31ecbaca removed that option together with the scripted initrd:
error: The option `boot.initrd.systemd.enable' can no longer be used since it's been removed. Scripted initrd has been removed.
Upstream impermanence already fixed exactly this (fa627f4c8b, 2026-10-21, "nixos: boot.initrd.systemd.enable was removed on unstable", guarding the read with options.boot.initrd.systemd.enable.visible or true). The branch bumps the impermanence input alone to fa627f4c8b8e92f58f58a73d99ee4e1304e64eb2. After that, eval passes:
nix eval .#nixosConfigurations.amd.config.system.build.toplevel.drvPath
→ "/nix/store/0yk2blbsgl9xpgqygcqq3zbzc5xd9cli-nixos-system-amd-26.11.20261027.31ecbac.drv"
(warnings only, incl. the gemini-cli removal notice.)
Failing step: build
nix build .#nixosConfigurations.amd.config.system.build.toplevel on the laptop fails at nodejs-24.22.1 (from nixpkgs-unstable — the root nixpkgs input follows it, and nix-openclaw follows root). The package is the upstream binary repack (node-v24.22.1-linux-x64.tar.xz); after auto-patchelf against the new gcc 16.2.0 libstdc++/libgcc, its install-check runs the patched node binary and it segfaults:
error: Cannot build '/nix/store/rbp7s9mpjqy2park6d340gj14bmlgc0j-nodejs-24.22.1.drv'.
Reason: builder failed with exit code 139.
> Running phase: installCheckPhase
> /nix/store/7svk92vqz7334ssj2m94s4kvbf5x7wk7-check-installed-node-runtime.sh: line 6: 460 Segmentation fault (core dumped) "$out/bin/node" "$NODE_RUNTIME_CHECK"
Cascade (all "1 dependency failed"): openclaw-2026.10.7 and its runtime plugins → home-manager-generation / user-environment / activation-script → toplevel. So the amd system cannot build with this pin set.
What was tried
- Reproduced in isolation:
nix build --no-link /nix/store/rbp7s9mpjqy2park6d340gj14bmlgc0j-nodejs-24.22.1.drv^outfails identically (same segfault line, exit 139) — deterministic, not a flake or OOM. - No in-repo mechanical fix identified: the config consumes plain
nodejs(nixos/environment/default.nix) and openclaw's nodejs comes from its own flake's follows. Swapping openclaw/the system to a different nodejs (source-built, another major) or pinningnixpkgs-unstableback wholesale is a judgment call, not a mechanical successor swap, so this run stopped per the nixpkgs-update skill instead of improvising one. - Side note: after the failure summaries, the build session wedged — the nix daemon held 15 zombie builder processes for ~8 minutes with no log progress; the (already failed) build client was killed. Toplevel was already unbuildable at that point; this did not change the outcome.
Next step (for Drew / a sweep)
Decide one of:
- Wait for / chase an upstream nixpkgs fix for the binary
nodejs_24segfault under the gcc-16 patchelf, then re-run the update from the branch above (the impermanence commit is needed either way). - Override the nodejs used by openclaw + the system environment to a source-built or different-major nodejs on this branch and prove the toplevel build.
- Pin
nixpkgs-unstableback to6510408don the branch and land the master/stable portion only, knowing the main input then stays put.
Acceptance: nix build .#nixosConfigurations.amd.config.system.build.toplevel completes on the laptop from this branch (or a successor branch), PR opened and merged; never switched by an agent.
Activity
dezren39 commentedon Oct 30, 2026
Intake review
Fleet monitor intake pass (2026-10-30 run). Labels normalised this pass:
kind/infrastructure,priority/normal,effort/m,status/blocked,user/review,meta/enriched. The body's acceptance line (toplevel builds on the laptop from the branch, PR opened and merged, never switched by an agent) is retained as the acceptance criteria (no duplicate section appended).Still live / blocked on a decision: the update branch
chore/nixpkgs-update-2026-10-30remains unmerged by design — the run stopped at a judgment call, per the nixpkgs-update skill, after the binarynodejs_24segfault under the gcc-16 patchelf made the toplevel unbuildable. The three options in the body are the decision:nodejs_24segfault, then re-run the update from the branch (the impermanence bump is needed either way).nixpkgs-unstableback to6510408don the branch and land the master/stable portion only.Monitor's read: option 3 is the lowest-risk path — it keeps the update cadence moving and lands the impermanence fix without improvising a nodejs override (option 2 is exactly the kind of mechanical-successor improvisation the skill warns against); option 1 leaves all four inputs stale for an upstream timeline nobody controls. Flagged
user/reviewfor Drew / Muse review in a chat, per the label flow — this comment is the intake flag, not the decision.Related: the branch commits
a5983341(flake.lock) andaec3998e(impermanence bump) are the reusable artefacts whichever option is chosen.fix(nixpkgs): restore regex-pcre build env after Haskell builder and …
dezren39 commentedon Nov 2, 2026
2026-11-02 attempt (scheduled
nixpkgs-update-amd) — still blocked; new branchAttempted on branch
chore/nixpkgs-update-2026-11-02(left pushed at02090364d, no PR — the update is not proven to build). Worktree was on the laptop (~/git/fleet/nixpkgs-update-2026-11-02, removed after the run). Base:origin/mainf12f5b7e4.What moved (unstable-only, per the 2026-10-31 "use unstable" directive)
just update-nixpkgs-unstable(the script'snixos-unstableref filter — plainjust update-nixpkgswould also move master):nixpkgs-unstable:6510408d147d->0bb2eedc20e4(2026-11-01)nixpkgs-25:e7439b6b14ad->0bb2eedc20e4(2026-11-01)nixpkgs-master(c61a74e1) andnixpkgs-stable(cfdb2236) untouched, verified in the lock diff.Commits on the branch:
4e5845da7— chore(deps): nixpkgs update 2026-11-02 (flake.lock only)1ade55b83— fix(deps): bump impermanence (cherry-pick ofaec3998e4from the 2026-10-30 branch; theboot.initrd.systemd.enableremoval error recurred verbatim at the new pin, upstream fixfa627f4still the right rev)02090364d— fix(nixpkgs): downgrade gtk+ broken problem to warn for gerbv (new breakage, below)New eval breakage: gtk2 problems-framework gate
After the impermanence fix, eval failed with:
Chain:
systemd.services.mandb->man-paths->gerbv-2.10.0(buildInputs includegtk2-x11).gerbvsits innixos/environment/default.nix(electronics section) and has no gtk3+ successor package. The new pin gatesbrokenthroughnixpkgs.config.problems.handlers, which overrides this repo's existingallowBroken = truepolicy (nixos/nixpkgs/default.nix). The branch adds a handler scoped to gtk+ only ("gtk+".broken = "warn"), preserving current system content; gtk2's actual removal by 27.05 will force the drop-gerbv decision then.Failing step this run: evaluation could not complete in the run window (laptop contention)
The third eval (after the gtk+ fix) never finished: for the whole run the laptop was saturated (load 146-200) by a concurrent toplevel build of the 2026-10-30 branch (
nix build .#nixosConfigurations.amd... --out-link /tmp/toplevel-2026-10-30, started 06:29 CST, still running at report time — that effort is presumably still chasing this same update from the older pins). The eval starved at thenixpkgs-patchedderivation for 17+ minutes (process alive but ~4% CPU) and was killed when this run's window closed. The acceptance build was not started: it cannot finish inside a run under this contention and must not be left detached (skill rule from the 2026-10-30 13h orphan incident).So: eval past the gtk+ fix is not yet proven, and the build is not attempted. Nothing about the new pins themselves is yet shown unbuildable.
Next step
Re-run from the pushed branch once the laptop is not carrying another toplevel build:
nix eval .#nixosConfigurations.amd.config.system.build.toplevel.drvPath, then the toplevel build. At0bb2eedcthe 2026-10-30 branch's other fixes (nodejs patchelf — #2727, activitywatch, awscli, cargo-llvm-cov — #4277) may or may not still be needed; the nodejs segfault was observed at31ecbaca, not at this rev. Acceptance unchanged: toplevel build completes on the laptop, PR opened and merged, never switched by an agent.dezren39 commentedon Nov 2, 2026
Status update (2026-11-02): the update's own breakage is fixed; PR opened, deliberately NOT merged — the only remaining red is #2696.
PR: #4519
What was fixed on
chore/nixpkgs-update-2026-10-30(each with a dedicated removal issue named in a comment at the override site):Build state: each fix above was verified against the exact derivation the toplevel graph uses (both activitywatch variants incl. aw-core's full suite, the pinned awscli chain, the hook-patched nodejs with installCheck, misfortune/regex-pcre, cargo-llvm-cov's binary). A string of timing suites (uvloop, anyio, psygnal, cherrypy, power-profiles-daemon, easyprocess, pytest-subprocess, httpcore, atuin, openvswitch, libmemcached, tinysparql) failed under the laptop's concurrent-build load and each passed solo — flakes, not update breakage.
The toplevel build as a whole has not run to completion: the graph holds 55,615 derivations with thin cache coverage on these pins (multi-day source-build volume at current load; ~32k still unrealised when the attempts were wound down), and mcpx's heavy derivation hangs rather than fails in its test stages (in-graph:
cargo testchildless at ~0 CPU for 27+ min, twice; solo conformance processes stall identically), so the toplevel can never assemble while #2696 stands. That is the only known remaining failure, it is pre-existing on main, and this update touches nothing in pkgs/mcpx. Once #2696 is fixed and a full toplevel build completes green, the PR can merge. This issue stays open until then. NOT switched at any point.dezren39 commentedon Nov 2, 2026
Follow-up on the root cause here (branch-HEAD pins landing before Hydra/cache had built them): #4553 tracks gating the
nixos-unstableinput updates inscripts/update-nixpkgs-inputs.pyon the newest Hydra-fully-built revision for the channel, fail-closed. That change is being implemented separately; this issue and PR #4519 are untouched by it.fix(scripts): gate nixpkgs-unstable updates on the Hydra-cached revision
3 remaining items
dezren39 commentedon Nov 3, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
dezren39 commentedon Nov 4, 2026
🤖 Agent note · handed off
opencode, so this lane hands off per runbook §1b — the opencode claimant launches the bound session. Issue chore(deps): nixpkgs update blocked 2026-10-30 #2715 carriesuser/review+status/blocked: per the runbook,user/reviewmeans the assigned worker reviews first — that reviewer is the bound opencode session; escalate touser/escalatedonly if the decision is genuinely Drew's, with the review attached.dezren39 commentedon Nov 4, 2026
Launched and bound queue task 7956 to OpenCode session ses_e5e5a81a7f3ffeDI0OMQdMT7B8 via opencode-route/workhorse (build). The task claim remains owned by repo-worker-code.
dezren39 commentedon Nov 4, 2026
Current-main triage (2026-11-04)
Verdict: still blocked; do not merge PR #4519 as-is. I verified GitHub
mainatce1724861f2b571aacd0635e248be84b92da81a1.The original four-input update path has materially changed on current main:
flake.nixremoves thenixpkgs-masterandnixpkgs-stableinputs; it follows the fleet pin fornixpkgs-unstable. Current lock resolves that fleet pin to nixpkgsf9887a8a434bfba06e20dfc77bc8e0628266af84, newer than PR chore(deps): nixpkgs update 2026-10-30 #4519's31ecbaca. The old master/stable changes and unstable pin in chore(deps): nixpkgs update 2026-10-30 #4519 are therefore not a current-main fix path.nixpkgs-25remains ate7439b6b14ad3cc35d05608ebca9bce01a25f5f8and is still consumed for ActivityWatch andmisfortune(nixos/environment/default.nix). Any remaining update must be scoped to this retained exception and revalidated against its consumers; do not blindly carry over the old branch's broader overrides.mergeable_state=dirty; its head is 10 commits ahead and 3,252 behind current main. Checks are green, but the branch is not mergeable and its description says the updated-pin full toplevel build was not completed.a0625480; that historical result does not validate chore(deps): nixpkgs update 2026-10-30 #4519's pin set.Current blocker and order: the existing PR cannot be used: first decide whether #2715 still intends to update the retained
nixpkgs-25exception now that unstable is fleet-pinned and master/stable were removed. If yes, create a successor from current main limited to that input/its affected consumers, then prove eval and the toplevel build (or CI equivalent), resolve current-main review/mergeability, and only then merge. If no, close #2715 as superseded with this evidence. I recommend the narrow successor only ifnixpkgs-25is intentionally due; no code change or merge was made in this triage.This is a genuine scope decision for Drew, so I am leaving the issue blocked and escalating for that decision.
dezren39 commentedon Nov 4, 2026
nixpkgs-update-amd run 2026-11-04 — BLOCKED (nixpkgs-25 / regex-pcre)
Scheduled update attempt from current main
ca8e03dbd. Branch:chore/nixpkgs-update-2026-11-04, left pushed atfaa9f2507(flake.lock commit only). No PR opened; nothing switched.What the update is now: since #5311 merged,
nixpkgs/nixpkgs-unstablefollow the fleet pin, and master/stable are gone — sojust update-nixpkgs-unstablemoves exactly one input:nixpkgs-25,e7439b6b14ad→1d060086d60c, pinned to the Hydra-gated channel revision (jobsetnixos:unstable, eval 1830995, latest finishedtestedbuild 352014846), not branch HEAD.Failing step:
nix eval .#nixosConfigurations.amd.config.system.build.toplevel.drvPathWhat was tried: a problems-handler alone does not fix it. Solo repro against the new rev (scratch import of
nixpkgs-25@1d060086withproblems.handlers."regex-pcre".broken = "warn", then buildinghaskellPackages.misfortune): evaluation passes, the build fails deterministically inregex-pcre-0.95.0.1's configure phase —So at this rev regex-pcre is marked broken because it genuinely does not build (the Haskell builder no longer puts pkg-config in its build env — the same mechanism the 2026-10-30 run hit; its fix was the vendored-pcre + pkg-config override tracked by #3906, which exists only on the unmerged PR #4519 branch). That override class is a design judgment, not a mechanical fix, so this run stops here per procedure.
Not blocked:
activitywatch(the other nixpkgs-25 consumer) evaluates cleanly at the new rev (activitywatch-0.13.2.drv). The sole blocker is misfortune → regex-pcre — i.e. the very package the nixpkgs-25 pin exception exists for (#3327).Decision for Drew (same fork today's triage flagged): (1) keep
nixpkgs-25frozen ate7439b6buntil upstream un-breaks regex-pcre or misfortune drops it — the scheduled update then no-ops on this input; (2) revive the #3906-style override (pkg-config + vendored pcre 8.45) on this branch and land it with a removal issue; (3) drop misfortune from the amd environment, after which nixpkgs-25 can likely be removed entirely (activitywatch already evals at the channel rev). Recommendation: (1) for now — the frozen pin costs nothing, and (2) re-creates an unmerged-override branch the fleet already carries once.dezren39 commentedon Nov 4, 2026
Drew's ruling 2026-11-04 — "fix it or pin misfortune"
Drew has ruled on the fork above: fix it or pin misfortune. This supersedes the stay-frozen stop for this blocker (option 1 is off the table unless both routes below fail).
Execution order:
regex-pcre) on the package set that actually feeds the amd toplevel — thenixpkgs-25raw legacyPackages segment innixos/environment/default.nix(hash-matched against the failing graph; the chore(deps): nixpkgs update 2026-10-30 #4519 instance sat on the master segment, which no longer exists post-nix: drop nixpkgs-master and nixpkgs-stable inputs (#3327) #5311). Dedicated removal issue filed first: chore(deps): remove the nixpkgs-25 regex-pcre pkg-config/pcre override once stock regex-pcre builds at the channel rev #5447.e7439b6bvia a separate frozen segment/input feeding only misfortune, whilenixpkgs-25moves to1d060086for the rest.Either route lands only with the full acceptance build (
nix build .#nixosConfigurations.amd.config.system.build.toplevelon the laptop) passing; the PR is merged per Drew's standing merge authority; nothing is switched. Outcome will be posted here when the run completes.fix(nixos): restore regex-pcre build env on the nixpkgs-25 segment
dezren39 commentedon Nov 4, 2026
Resolved 2026-11-04 — Route A landed (PR #5453, merged)
Drew's ruling executed. Route A (fix it) worked; Route B (pin misfortune) was not needed.
What landed (PR #5453, merge commit
44b2dd56e):nixpkgs-25e7439b6b->1d060086plus the #3906-class override on the set that feeds the toplevel -- thenixpkgs-25segment innixos/environment/default.nixis now imported asnixpkgs-25-fixedwith:config.problems.handlers."regex-pcre".broken = "warn"-- handlers live in the importing config; an.extendoverlay on the rawlegacyPackageswas verified to still refuse evaluation, so the segment is re-imported with config + overlays.pkg-configappended tonativeBuildInputs, vendored pcre 8.45 appended tobuildInputs. At1d060086the stock derivation had NO pkg-config innativeBuildInputsand an EMPTYbuildInputs(no pcre at all), so both halves are load-bearing.Placement proof (hash match): the committed tree's toplevel closure contains exactly the overridden regex-pcre drv
/nix/store/5xl002q8kvd30wvnhpr36pj64za3i9hv-regex-pcre-0.95.0.1.drvand misfortune drv/nix/store/gy36sgsqznvg17q017fvhwml9pw4wpmj-misfortune-0.1.2.1.drv; the stock regex-pcre drv is absent. Both were solo-realised and store-validated (regex-pcre outzq479fjj..., hashsha256:1lbl2qhrmpq16xdkpihyckv88qlcddwsxk4zimna2sxdxl8j99y3; misfortune out3d832xsl..., hashsha256:1p6rwjcz8f3q94ig0bpivwf2q1glh5pfg5kcl0b0yl3innh3qk31).Acceptance build:
nix build .#nixosConfigurations.amd.config.system.build.toplevelcompleted on the laptop from the branch's committed tree. Output/nix/store/jiz6p50by61r4dfyrbb4565a9w1pbjaa-nixos-system-amd-26.11.20261102.f9887a8-- store-valid, hashsha256:087qi62555v3155jdw59x50wyi5mk3kj8cwk5ap2brgrg28fzb9d. NOT switched. CI note: the PR's only Actions check stayed queued in the fleet runner congestion; merged under the procedure's congestion clause with the laptop build as acceptance (recorded on the PR).Follow-through: the override is temporary -- removal tracked in #5447 (condition + safe-removal test). The nixpkgs-update skill was updated in the same run (ruling recorded, import-with-config placement mechanics, dirty-tree drv note). Worktree removed; branch deleted.
dezren39 commentedon Nov 5, 2026
Resolution (Drew, main chat 2026-11-05): "if it's fixed then stop."
The freeze question is closed with no freeze. The break this issue tracked was fixed by Route A (PR #5453: pin moved e7439b6b → 1d060086 with the nixpkgs-25-fixed regex-pcre override, build-verified), which per Drew's 2026-11-04 ruling took the freeze option off the table. The pin stays on the normal update train; the override's removal remains tracked in #5447 on its existing premise.