Skip to content

fix(scripts): gate nixpkgs-unstable input updates on the Hydra channel (cached) revision #4553

Closed

Description

@dezren39

Problem

scripts/update-nixpkgs-inputs.py updates the nixpkgs inputs to branch HEAD (nix flake update --refresh). For the nixos-unstable ref set that is the wrong target: branch HEAD runs days ahead of what Hydra has built and what cache.nixos.org actually serves. An update can therefore pin a revision whose binaries do not exist in the cache yet, turning the next system build into a massive source build.

This is not hypothetical: the 2026-10-30 update pinned nixpkgs-unstable at 31ecbaca before Hydra had built it, and the amd toplevel became a multi-day source build (55k derivations, thin cache coverage). See #2715 and PR #4519 for the full fallout, including the temporary overrides that had to be bisected and pinned on top of that update.

Fix: gate the unstable ref set on the Hydra channel revision

For every input whose original.ref is nixos-unstable (currently nixpkgs-unstable and nixpkgs-25), the update target is no longer branch HEAD. It is the newest revision Hydra has fully built for the channel:

  1. GET https://hydra.nixos.org/job/nixos/unstable/tested/latest-finished (JSON, follow redirects; jobset nixos:unstable, renamed from nixos:trunk-combined) → take jobsetevals[0].
  2. GET https://hydra.nixos.org/eval/<id> (JSON) → target revision is jobsetevalinputs.nixpkgs.revision.
  3. Pin each gated input to that revision in flake.lock: rewrite its locked entry (rev + narHash + lastModified taken from a real nix fetch of that exact revision, e.g. nix flake metadata github:NixOS/nixpkgs/<rev>) while leaving its original entry untouched, so original.ref stays nixos-unstable and input discovery (this script and nix/nixpkgs-inputs.nix) keeps matching.
  4. If an input is already at the Hydra revision, the run is a clean no-op for it and says so.

Verified live 2026-11-02: latest-finished build 351844113 (finished, status 0), eval 1830926, revision 0bb2eedc20e4739f29e8045034f2828ad7054f8f — which is exactly the revision the 2026-11-02 scheduled update run converged on by hand.

Dead ends, do not use: channels.nixos.org/<channel>/git-rev returns 404; NixOS/nixpkgs-channels has been untouched since 2021.

Fail closed

If the Hydra lookups fail, time out, or return an unexpected shape (missing fields, a revision that is not a full 40-char commit hash), the script exits non-zero with a clear message and leaves flake.lock untouched. No update is better than an uncached update.

Scope

Acceptance criteria

  • For every input whose original.ref is nixos-unstable, scripts/update-nixpkgs-inputs.py pins the Hydra channel revision (latest-finished nixos:unstable tested eval's nixpkgs revision) instead of branch HEAD, rewriting only the locked entry (rev/narHash/lastModified from a real fetch) and leaving original untouched.
  • An input already at the Hydra revision is a clean no-op and the run says so.
  • On any Hydra lookup failure, timeout, or unexpected response shape, the script exits non-zero with a clear message and leaves flake.lock unchanged (fail closed).
  • Non-unstable inputs keep the existing branch-HEAD update behavior.

Activity

  1. added a commit that references this issue on Nov 2, 2026
    2d63376
  2. added
    kind/bugSomething isn't working (prefixed form of the classic bug label)
    source/humanFiled directly by a human
    priority/highImportant and time-sensitive
    effort/mMedium: expected under 1 day of work
    meta/enrichedIntake review done: acceptance criteria, links, and normalized labels added
    on Nov 2, 2026
  3. dezren39 commented on Nov 2, 2026

    @dezren39
    MemberAuthor

    Intake review

    Filed by Drew with the incident evidence (#2715, PR #4519) and a live-verified Hydra revision in the body. No open PR references this issue this run. Risks: Hydra API shape drift — the fail-closed AC covers it. Suggested approach: implement the gate inside the existing update script's per-input loop, keyed on original.ref, with a fixture test for the eval JSON parsing.

    — fleet monitor intake pass, 2026-11-02. Labels normalized (kind/source/status/priority/effort) and acceptance criteria appended to the body; original body preserved above the new section.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    effort/mMedium: expected under 1 day of workkind/bugSomething isn't working (prefixed form of the classic bug label)meta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedpriority/highImportant and time-sensitivesource/humanFiled directly by a humanstatus/needs-triageNot yet triaged; needs a first look

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions