Skip to content

node-test-commit-arm-fanned is failing git-rebase since 2026-07-25 #4408

Closed
Closed
node-test-commit-arm-fanned is failing git-rebase since 2026-07-25#4408

Description

@trivikr
trivikr
opened on Jul 27, 2026

https://ci.nodejs.org/job/node-test-commit-arm-fanned/

Last success

[git-rebase #132289](https://ci.nodejs.org/job/git-rebase/132289/) started.
[git-rebase #132289](https://ci.nodejs.org/job/git-rebase/132289/) completed. Result was SUCCESS
Triggering projects: [post-build-status-update](https://ci.nodejs.org/job/post-build-status-update/)
    >> Job status: [node-cross-compile] the 'build only if scm changes' feature is disabled.
Starting build job [node-cross-compile](https://ci.nodejs.org/job/node-cross-compile/) at 01:02:06.
Counting node-cross-compile. enabledIndex=1
node-cross-compile: Quiet period groovy=[], index=1 -> quietPeriodGroovy=null
Error calculating quiet time for index 1 and quietPeriodGroovy []: Cannot invoke "java.lang.Integer.intValue()" because "quietPeriod" is null; returning 0
quiet period for node-cross-compile is 0 seconds.Finished Build : [#59076](https://ci.nodejs.org/job/node-cross-compile/59076//) of Job : [node-cross-compile](https://ci.nodejs.org/job/node-cross-compile/) with status : [SUCCESS](https://ci.nodejs.org/job/node-cross-compile/59076//console) at 01:02:28
Variable Existence Condition: checking "NODEJS_MAJOR_VERSION" variable 
Numerical comparison: [27] >= Greater than or equal to [12]
Run condition [And] enabling perform for step [MultiJob Phase]
    >> Job status: [node-test-binary-armv7l] the 'build only if scm changes' feature is disabled.
Starting build job [node-test-binary-armv7l](https://ci.nodejs.org/job/node-test-binary-armv7l/) at 01:02:28.
Counting node-test-binary-armv7l. enabledIndex=1
node-test-binary-armv7l: Quiet period groovy=[], index=1 -> quietPeriodGroovy=null
Error calculating quiet time for index 1 and quietPeriodGroovy []: Cannot invoke "java.lang.Integer.intValue()" because "quietPeriod" is null; returning 0
quiet period for node-test-binary-armv7l is 0 seconds.Finished Build : [#22895](https://ci.nodejs.org/job/node-test-binary-armv7l/22895//) of Job : [node-test-binary-armv7l](https://ci.nodejs.org/job/node-test-binary-armv7l/) with status : [SUCCESS](https://ci.nodejs.org/job/node-test-binary-armv7l/22895//console) at 01:02:31
Collecting metadata...
Metadata collection done.
[Checks API] No suitable checks publisher found.
Notifying upstream projects of job completion
Triggering a new build of [git-delete-branches](https://ci.nodejs.org/job/git-delete-branches/)
Variable Existence Condition: checking "NODEJS_MAJOR_VERSION" variable 
Numerical comparison: [27] >= Greater than or equal to [12]
Run condition [And] enabling perform for step [Builder to mark whether executed]
Finished: SUCCESS

https://ci.nodejs.org/job/node-test-commit-arm-fanned/48548/console

First failure

[git-rebase #132291](https://ci.nodejs.org/job/git-rebase/132291/) started.
[git-rebase #132291](https://ci.nodejs.org/job/git-rebase/132291/) completed. Result was FAILURE
Build step 'Trigger/call builds on other projects' marked build as failure
Collecting metadata...
Metadata collection done.
[Checks API] No suitable checks publisher found.
Notifying upstream projects of job completion
Variable Existence Condition: checking "NODEJS_MAJOR_VERSION" variable 
Numerical comparison: [27] >= Greater than or equal to [12]
Run condition [And] enabling perform for step [Builder to mark whether executed]
Finished: FAILURE

https://ci.nodejs.org/job/node-test-commit-arm-fanned/48549/console

Activity

  1. trivikr commented on Jul 27, 2026

    @trivikr
    trivikr
    on Jul 27, 2026
    MemberAuthor

    The have been 60+ continuous failures at the time of comment

  2. sxa commented on Jul 27, 2026

    @sxa
    sxa
    on Jul 27, 2026
    Member

    Noting that this does not appear to be limited to a single version of the workspace on the mnx-1 machine:

    From https://ci.nodejs.org/job/git-rebase/132437/console (git-rebase path)

    14:11:39  > git rev-parse --resolve-git-dir /home/iojs/build/workspace/git-rebase/.git # timeout=10
[...]
14:12:10 + git checkout -B jenkins-node-test-commit-windows-fanned-e261f4b251be190ec3f7a7e44e768c39166cd0b7
14:12:10 Switched to and reset branch 'jenkins-node-test-commit-windows-fanned-e261f4b251be190ec3f7a7e44e768c39166cd0b7'
14:12:10 + grep -q '^67.158.54.159' /home/iojs/.ssh/known_hosts
14:12:10 + git push binary_tmp@67.158.54.159:binary_tmp.git +jenkins-node-test-commit-windows-fanned-e261f4b251be190ec3f7a7e44e768c39166cd0b7
14:12:15 remote: fatal: bad object refs/heads/jenkins-node-test-commit-windows-fanned-55230557d1b20e1597c13ae293adde102628db94        
14:12:16 fatal: bad object refs/heads/jenkins-node-test-commit-windows-fanned-55230557d1b20e1597c13ae293adde102628db94
14:12:16 To 67.158.54.159:binary_tmp.git
14:12:16  ! [remote rejected]         jenkins-node-test-commit-windows-fanned-e261f4b251be190ec3f7a7e44e768c39166cd0b7 -> jenkins-node-test-commit-windows-fanned-e261f4b251be190ec3f7a7e44e768c39166cd0b7 (missing necessary objects)
14:12:16 error: failed to push some refs to '67.158.54.159:binary_tmp.git'
14:12:16 Build step 'Execute shell' marked build as failure

    From https://ci.nodejs.org/job/git-rebase/132438/console (git-rebase@2 path)

    14:11:44  > git rev-parse --resolve-git-dir /home/iojs/build/workspace/git-rebase@2/.git # timeout=10
[...]
14:12:06 + git checkout -B jenkins-node-test-commit-arm-fanned-e261f4b251be190ec3f7a7e44e768c39166cd0b7
14:12:07 Switched to and reset branch 'jenkins-node-test-commit-arm-fanned-e261f4b251be190ec3f7a7e44e768c39166cd0b7'
14:12:07 + grep -q '^67.158.54.159' /home/iojs/.ssh/known_hosts
14:12:07 + git push binary_tmp@67.158.54.159:binary_tmp.git +jenkins-node-test-commit-arm-fanned-e261f4b251be190ec3f7a7e44e768c39166cd0b7
14:12:13 remote: fatal: bad object refs/heads/jenkins-node-test-commit-windows-fanned-55230557d1b20e1597c13ae293adde102628db94        
14:12:14 fatal: bad object refs/heads/jenkins-node-test-commit-windows-fanned-55230557d1b20e1597c13ae293adde102628db94
14:12:14 To 67.158.54.159:binary_tmp.git
14:12:14  ! [remote rejected]         jenkins-node-test-commit-arm-fanned-e261f4b251be190ec3f7a7e44e768c39166cd0b7 -> jenkins-node-test-commit-arm-fanned-e261f4b251be190ec3f7a7e44e768c39166cd0b7 (missing necessary objects)
14:12:14 error: failed to push some refs to '67.158.54.159:binary_tmp.git'
14:12:14 Build step 'Execute shell' marked build as failure

  3. sxa commented on Jul 27, 2026

    @sxa
    sxa
    on Jul 27, 2026
    Member

    Noting that this also applies to node-test-commit-windows-fanned which uses the same git-rebase job.

  4. sxa commented on Jul 27, 2026

    @sxa
    sxa
    on Jul 27, 2026
    Member

    Regenerated the ~binary_tmp/binary_tmp.git based on https://github.com/nodejs/build/blob/main/ansible/roles/jenkins-workspace/tasks/main.yml and an initial re-run of the git-rebase job appears to have run green. Hopefully it's doing the right thing.

  5. sxa commented on Jul 27, 2026

    @sxa
    sxa
    on Jul 27, 2026
    Member

    Noting that the old binary_tmp.git directory had got rather large so we may need to revisit the cleanup scripts associated with that workspace too

    # du -sh binary_tmp.git
62G     binary_tmp.git
  6. sxa
    marked node-compile-windows is failing since 2026-07-25 on win-vs2022_clang(-arm64) with error remote rejected #4410 as a duplicate of this issue on Jul 27, 2026

  7. sxa commented on Jul 27, 2026

    @sxa
    sxa
    on Jul 27, 2026
    Member

    Noting that this also impacted the testing of the builds for the security releases this week.

  8. sxa commented on Jul 28, 2026

    @sxa
    sxa
    on Jul 28, 2026
    Member

    git-rebase jobs have been failing again for the last four hours. Will re-extract the binary_tmp.git.new.tar.xz tarball I made after re-cloning yesterday (as the git clone was slow - I'm not convinced the performance of the disks on the workspace machines are great)

  9. sxa commented on Jul 28, 2026

    @sxa
    sxa
    on Jul 28, 2026
    Member

    First failure today was in git-rebase#132502 from arm-fanned#48651 initiated by the daily v24.x build.
    Previous job git-rebase#132501 from windows-fanned#78746 PR64792

  10. sxa commented on Jul 29, 2026

    @sxa
    sxa
    on Jul 29, 2026
    Member

    Appears to be behaving but I feel the underlying issue may not be resolved. I'll close this for now and re-open if it recurs.

  11. sxa
    closed this as completedon Jul 29, 2026

  12. sxa commented on Sep 1, 2026

    @sxa
    sxa
    on Sep 1, 2026
    Member

    Occurred again this morning on test-mnx-ubuntu2204-x64-1:

    06:02:54  > git fetch --tags --force --progress -- git@github.com:nodejs/node.git +refs/heads/*:refs/remotes/origin/* +refs/heads/v24.x-staging:refs/remotes/origin/_jenkins_local_branch # timeout=30
06:03:02  > git rev-parse refs/remotes/origin/_jenkins_local_branch^{commit} # timeout=10
06:03:02 Checking out Revision 067d305cca05b1d4bc9024dd004b3038d9ab2c6e (refs/remotes/origin/_jenkins_local_branch)
06:03:02  > git config core.sparsecheckout # timeout=10
06:03:02  > git checkout -f 067d305cca05b1d4bc9024dd004b3038d9ab2c6e # timeout=10
06:03:30 Commit message: "events: inline iterationCondition hybrid dispatch closure"
06:03:30 Using 'Changelog to branch' strategy.
06:03:30 Cleaning workspace
06:03:30  > git rev-parse --verify HEAD # timeout=10
06:03:30 Resetting working tree
06:03:30  > git reset --hard # timeout=10
06:03:31  > git clean -fdx # timeout=10
06:03:32 [Checks API] No suitable checks publisher found.
06:03:32 Run condition [Always] enabling prebuild for step [[Execute shell]]
06:03:32 [git-rebase@2] $ /bin/sh -xe /tmp/jenkins5660034071802766396.sh
06:03:32 + [ -z  ]
06:03:32 + [ nodejs/node != nodejs/node ]
06:03:32 + echo refs/heads/v24.x-staging
06:03:32 + grep -qE ^(refs/)?pull/[0-9]+
06:03:32 + echo Allowing refs/heads/v24.x-staging for nodejs/node without a specified COMMIT_SHA_CHECK
06:03:32 Allowing refs/heads/v24.x-staging for nodejs/node without a specified COMMIT_SHA_CHECK
06:03:32 + [ -n  ]
06:03:32 + echo Skipping rebase.
06:03:32 Skipping rebase.

    Noting that subsequent jobs all failed on the binary_tmp` push phase:

    06:25:34 Successfully rebased and updated detached HEAD.
06:25:34 + git rev-parse HEAD
06:25:38 c13e689cb736cb334e5683734b4dbddb31cc7c8d
06:25:38 [git-rebase] $ /bin/bash -ex /tmp/jenkins429014443002337696.sh
06:25:38 + git checkout -B jenkins-node-test-commit-windows-fanned-207aae40ecc64fd5308f332ea92eaddd6b89c012
06:25:38 Switched to a new branch 'jenkins-node-test-commit-windows-fanned-207aae40ecc64fd5308f332ea92eaddd6b89c012'
06:25:38 + grep -q '^67.158.54.159' /home/iojs/.ssh/known_hosts
06:25:38 + git push binary_tmp@67.158.54.159:binary_tmp.git +jenkins-node-test-commit-windows-fanned-207aae40ecc64fd5308f332ea92eaddd6b89c012
06:25:38 remote: fatal: bad object refs/heads/jenkins-node-test-commit-windows-fanned-2dc89ec8107cf2a612c4b58b0b67279d46b33427        
06:25:38 fatal: bad object refs/heads/jenkins-node-test-commit-windows-fanned-2dc89ec8107cf2a612c4b58b0b67279d46b33427
06:25:38 remote: warning: The last gc run reported the following. Please correct the root cause        
06:25:38 remote: and remove gc.log        
06:25:38 remote: Automatic cleanup will not be performed until the file is removed.        
06:25:38 remote: 
06:25:38 remote: error: pack-objects died of signal 7        
06:25:38 remote: fatal: failed to run repack        
06:25:38 remote: 
06:25:39 To 67.158.54.159:binary_tmp.git
06:25:39  ! [remote rejected]         jenkins-node-test-commit-windows-fanned-207aae40ecc64fd5308f332ea92eaddd6b89c012 -> jenkins-node-test-commit-windows-fanned-207aae40ecc64fd5308f332ea92eaddd6b89c012 (missing necessary objects)
06:25:39 error: failed to push some refs to '67.158.54.159:binary_tmp.git'
06:25:39 Build step 'Execute shell' marked build as failure

    Resolved with git update-ref -d refs/heads/jenkins-node-test-commit-windows-fanned-2dc89ec8107cf2a612c4b58b0b67279d46b33427

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions