Repository navigation
ci: track root causes and restore CI on latest and release/v11 #10027
Description
Purpose
This is the umbrella tracking issue for restoring CI and post-release integration on npm/cli's latest and release/v11 branches. It owns the overall failure inventory, coordinates fixes across npm/cli and upstream fixtures/actions, and tracks verification on both branches.
Track issues by root cause, not by error message, job, runtime, or branch. A child issue can contain multiple failures. Where the initiating cause is not yet established, retain an explicitly scoped investigation rather than presenting an infrastructure hypothesis as fact.
Investigated scope
The inventory below covers all 14 failed jobs in the four completed September 22, 2026 branch-head CI and Release runs:
| Branch | Release | Commit | Workflow | Failed jobs |
|---|---|---|---|---|
latest |
npm 12.1.0 | c039090578a5b21a1aa3aba9c96e199feaf1823a |
Release | 9 |
latest |
npm 12.1.0 | c039090578a5b21a1aa3aba9c96e199feaf1823a |
CI | 1 |
release/v11 |
npm 11.20.0 | d12b9434dd010b5fb7044c3cc149cdda317813f8 |
Release | 3 |
release/v11 |
npm 11.20.0 | d12b9434dd010b5fb7044c3cc149cdda317813f8 |
CI | 1 |
This is a complete inventory for those runs, not a claim that every historical failure has the same causes. The failed Release workflow status must not be equated with failed package publication: these failures occur in downstream integration.
Root-cause and incident owners
- ci: Node.js nightly integration fails while extracting source archive #10016
- ci: binary-split CITGM source download uses stale repository owner #10020
- ci: release integration selects Node 23 outside npm 12 supported engines #10024
- ci: CITGM Git-dependency fixtures lack npm 12 policy configuration #10026
- ci: CITGM relies on native-build option forwarding rejected by npm 12 #10023
- ci: live-registry smoke test fails on a socket reset with fetch retries disabled #10022
- ci: Cygwin provisioning fails before Windows shim tests run #10025
| Owner issue | Group | latest Release |
v11 Release | latest CI |
v11 CI |
|---|---|---|---|---|---|
| #10016 | Invalid Node nightly source response; publication-race hypothesis remains unproven | 1 | 1 | 0 | 0 |
| #10020 | Stale binary-split source repository metadata |
2 | 2 | 0 | 0 |
| #10024 | Stable integration matrix selects Node 23 outside npm 12's engines | 2 | 0 | 0 | 0 |
| #10026 | CITGM Git dependency is blocked by npm 12's default policy | 2 | 0 | 0 | 0 |
| #10023 | CITGM native-build option forwarding is rejected by npm 12 | 2 | 0 | 0 | 0 |
| #10022 | Live-registry smoke-test socket reset with retries disabled; initiating cause unresolved | 0 | 0 | 1 | 0 |
| #10025 | Cygwin provisioning fails before shim tests; installer cause unresolved | 0 | 0 | 0 | 1 |
| Total | 9 | 3 | 1 | 1 |
Complete failed-job inventory
| Branch/workflow | Failed job | Owner |
|---|---|---|
latest Release |
Build nightly | #10016 |
| v11 Release | Build nightly | #10016 |
latest Release |
Node 22: binary-split | #10020 |
latest Release |
Node 23: binary-split | #10020 |
| v11 Release | Node 22: binary-split | #10020 |
| v11 Release | Node 23: binary-split | #10020 |
latest Release |
Node 23: test nodejs | #10024 |
latest Release |
Node 23: test npm | #10024 |
latest Release |
Node 22: JSONStream | #10026 |
latest Release |
Node 23: JSONStream | #10026 |
latest Release |
Node 22: thread-sleep | #10023 |
latest Release |
Node 23: thread-sleep | #10023 |
latest CI |
Smoke Tests | #10022 |
| v11 CI | Windows Shims Tests | #10025 |
Grouping decisions and important qualifications
The Node 23 warning assertion, npm smoke stderr mismatch, EBADENGINE, subsequent unsatisfied registry mocks, and engine-dependent coverage failure belong under #10024. They should not produce separate tickets per assertion.
The two CITGM policy/configuration failures require different fixes. Allowing Git dependencies does not make --build-from-source a recognized npm CLI flag, and removing that flag does not permit Git dependencies. Both failures occur on supported Node 22, independently of #10024.
#10023 preserves the history from mapbox/node-pre-gyp#58: npm install --build-from-source was a documented and functional way to forward a native installer's configuration through npm. node-pre-gyp still implements source builds; npm 12 now rejects the unknown npm CLI flag before that installer can use it. The migration must verify the actual fixture's installer and preserve any intended source-build coverage, not simply remove the option and assume success.
#10016 and #10020 share the diagnostic weakness of streaming curl -sSL output into tar, but their upstream conditions are distinct. Better HTTP handling does not repair stale repository metadata or guarantee an archive exists. Both nightly jobs selected the same archive, but their logs do not preserve the HTTP response status/body, so a publication race is not a proven fact.
The socket reset in #10022 and the Cygwin setup failure in #10025 are separately owned investigations, not evidence of one shared outage. In particular, the Cygwin downgrade conflict and subsequent installer exit are both observed, but a causal relationship has not been established.
Cross-cutting follow-up owned by this tracker
- Preserve HTTP status and useful failure diagnostics for source downloads without feeding non-success responses directly into extraction.
- Correct
FINALEXIT=STEPEXITin the npm integration result aggregation and its template. It currently assigns a literal string and later emitsexit: STEPEXIT: numeric argument required. This masks the intended exit status after earlier test failures; it is not another primary cause of the 14 failed jobs. - Ensure upstream CITGM fixes reach the published version used by matrix generation. A change merged upstream but not consumed by npm/cli does not close the coverage gap.
- Apply branch-appropriate workflow/template changes to both release lines and document local equivalents for any new diagnostic or validation commands.
- Attach post-fix run evidence for both branches and confirm the affected tests actually execute rather than being skipped, suppressed, or bypassed.
Completion criteria
Close this tracker only when every child issue is resolved or has an explicitly justified disposition, the affected coverage is restored on both branches, and any remaining infrastructure uncertainty is documented with an owner and evidence. A rerun that happens to pass is not by itself proof of root cause, and making one error disappear is not proof that the rest of an integration job succeeded.
npm 12.2.0 follow-up inventory
The npm 12.2.0 Release workflow at c276cadf785c1018d82dd287fe7742567055efb9 reproduced the existing Node 23, JSONStream, binary-split, and thread-sleep failures and exposed two additional root causes:
- ci: nightly npm smoke test rejects expected engine warning #10064
- ci: nightly native CITGM fixtures ignore configured Node headers #10065
Workflow: https://github.com/npm/cli/actions/runs/36746332014
The ordinary branch CI workflow passed: https://github.com/npm/cli/actions/runs/36746330757
| Owner issue | Group | Failed jobs |
|---|---|---|
| #10024 | Stable Node 23 remains outside npm 12's supported engines | 2 |
| #10026 | JSONStream Git dependency remains blocked by npm 12 policy | 3 |
| #10020 | binary-split still consumes stale metadata because the merged CITGM change was not in the published version used by this run | 3 |
| #10023 | thread-sleep still supplies the rejected --build-from-source npm CLI option |
3 |
| #10064 | Nightly npm smoke test rejects the expected unsupported-engine warning | 1 |
| #10065 | leveldown and microtime do not consume the configured Node source directory on nightly | 2 |
| Total | 14 |
npm 12.2.0 failed-job additions
| Branch/workflow | Failed job | Owner |
|---|---|---|
latest Release |
Nightly: test npm | #10064 |
latest Release |
Nightly: leveldown | #10065 |
latest Release |
Nightly: microtime | #10065 |
The release/v11 branch CI for npm 11.21.0 also passed. Its separate release-please failure occurred before downstream integration and is tracked in #10066.
npm 11.21.0 release automation failure
The npm 11.21.0 Release workflow failed in template-oss-release-please before downstream release integration could run:
Workflow: https://github.com/npm/cli/actions/runs/36756679730
Failing job: https://github.com/npm/cli/actions/runs/36756679730/job/110028585934
The corresponding branch CI workflow passed: https://github.com/npm/cli/actions/runs/36756678607
Both changed packages were published to the npm registry, but their GitHub release records remain incomplete:
| Package | Registry publication | Missing Git tag | Missing GitHub Release |
|---|---|---|---|
npm@11.21.0 |
Published | v11.21.0 |
v11.21.0 |
@npmcli/config@10.14.0 |
Published | config-v10.14.0 |
config-v10.14.0 |
The confirmed fatal error was an unhandled GitHub HTTP 502 Bad Gateway response during the release-please step. The 11-of-13 component discovery message and the parser warning for the old non-Conventional Commit Fix typos in some files were not fatal.
The first remediation is to rerun the complete failed workflow so skipped dependent jobs are reevaluated. A successful rerun should create both tags and GitHub Releases and then execute Release Integration. If the 502 recurs, capture the request endpoint class, response status, and retry behavior before manually creating release records.
Upstream CITGM changes
Four npm/cli integration failures have corresponding changes in nodejs/citgm:
| npm/cli owner | CITGM pull request | State | Effect |
|---|---|---|---|
| #10020 | nodejs/citgm#1142 | Merged | Updates binary-split to use the current max-mapper/binary-split repository. |
| #10065 | nodejs/citgm#1122 | Merged | Removes the deprecated and archived leveldown fixture. This resolves only the leveldown half of #10065; microtime remains. |
| #10026 | nodejs/citgm#1158 | Open | Allows JSONStream's transitive Git test dependency in that fixture's npm 12 environment. |
| #10023 | nodejs/citgm#1159 | Open | Removes the obsolete --build-from-source argument and native tag from the now pure-JavaScript thread-sleep fixture. |
The merged binary-split and leveldown changes are included in the pending CITGM 10.0.4 release pull request, nodejs/citgm#1147. None of these changes should be treated as consumed by npm/cli until a CITGM release containing them is published, the npm/cli matrix resolves that version, and the affected package tests execute successfully.
Activity