Skip to content

v8-canary builds missing since May 17, linux/darwin builds missing since Febrary #312

Open
Open
v8-canary builds missing since May 17, linux/darwin builds missing since Febrary#312

Description

@joyeecheung
joyeecheung
opened on May 21, 2026

Last one is AIX-only https://nodejs.org/download/v8-canary/v27.0.0-v8-canary202605175b9294c0c1/

I don't have access to release CI, so can't see what's wrong.

Also there has not been any linux/darwin builds since Feburary. Last one is https://nodejs.org/download/v8-canary/v26.0.0-v8-canary20260223fd2a99f348/

cc @nodejs/build

Activity

  1. targos commented on May 21, 2026

    @targos
    targos
    on May 21, 2026
    Member

    Currently the main issue is the Rust version requirement

  2. richardlau commented on May 21, 2026

    @richardlau
    richardlau
    on May 21, 2026
    Member

    We should be getting Linux builds now (I updated Rust on the RHEL 8 release machines (all archs)).

    I'm suspicious that 20260517 is the last available downloadable build as we have in staging, e.g

    total 644852
drwxrwxr-x    3 staging staging      4096 May 21 16:42 .
drwxrwxr-x 1968 staging staging    131072 May 21 11:14 ..
drwxrwxr-x    3 staging staging      4096 May 21 16:15 docs
-rw-rw-r--    1 staging staging         0 May 21 16:15 docs.done
-rw-rw-r--    1 staging staging  84122889 May 21 11:47 node-v27.0.0-v8-canary20260521da8e25c394-aix-ppc64.tar.gz
-rw-rw-r--    1 staging staging         0 May 21 11:42 node-v27.0.0-v8-canary20260521da8e25c394-aix-ppc64.tar.gz.done
-rw-rw-r--    1 staging staging  10088167 May 21 16:15 node-v27.0.0-v8-canary20260521da8e25c394-headers.tar.gz
-rw-rw-r--    1 staging staging         0 May 21 16:16 node-v27.0.0-v8-canary20260521da8e25c394-headers.tar.gz.done
-rw-rw-r--    1 staging staging    596044 May 21 16:16 node-v27.0.0-v8-canary20260521da8e25c394-headers.tar.xz
-rw-rw-r--    1 staging staging         0 May 21 16:16 node-v27.0.0-v8-canary20260521da8e25c394-headers.tar.xz.done
-rw-rw-r--    1 staging staging  59259661 May 21 14:46 node-v27.0.0-v8-canary20260521da8e25c394-linux-arm64.tar.gz
-rw-rw-r--    1 staging staging         0 May 21 14:48 node-v27.0.0-v8-canary20260521da8e25c394-linux-arm64.tar.gz.done
-rw-rw-r--    1 staging staging  31185060 May 21 14:48 node-v27.0.0-v8-canary20260521da8e25c394-linux-arm64.tar.xz
-rw-rw-r--    1 staging staging         0 May 21 14:48 node-v27.0.0-v8-canary20260521da8e25c394-linux-arm64.tar.xz.done
-rw-rw-r--    1 staging staging  62065750 May 21 11:50 node-v27.0.0-v8-canary20260521da8e25c394-linux-ppc64le.tar.gz
-rw-rw-r--    1 staging staging         0 May 21 11:52 node-v27.0.0-v8-canary20260521da8e25c394-linux-ppc64le.tar.gz.done
-rw-rw-r--    1 staging staging  31804332 May 21 11:52 node-v27.0.0-v8-canary20260521da8e25c394-linux-ppc64le.tar.xz
-rw-rw-r--    1 staging staging         0 May 21 11:53 node-v27.0.0-v8-canary20260521da8e25c394-linux-ppc64le.tar.xz.done
-rw-rw-r--    1 staging staging  61479246 May 21 11:12 node-v27.0.0-v8-canary20260521da8e25c394-linux-s390x.tar.gz
-rw-rw-r--    1 staging staging         0 May 21 11:15 node-v27.0.0-v8-canary20260521da8e25c394-linux-s390x.tar.gz.done
-rw-rw-r--    1 staging staging  31380780 May 21 11:14 node-v27.0.0-v8-canary20260521da8e25c394-linux-s390x.tar.xz
-rw-rw-r--    1 staging staging         0 May 21 11:15 node-v27.0.0-v8-canary20260521da8e25c394-linux-s390x.tar.xz.done
-rw-rw-r--    1 staging staging  59602003 May 21 16:39 node-v27.0.0-v8-canary20260521da8e25c394-linux-x64.tar.gz
-rw-rw-r--    1 staging staging         0 May 21 16:42 node-v27.0.0-v8-canary20260521da8e25c394-linux-x64.tar.gz.done
-rw-rw-r--    1 staging staging  32253548 May 21 16:42 node-v27.0.0-v8-canary20260521da8e25c394-linux-x64.tar.xz
-rw-rw-r--    1 staging staging         0 May 21 16:42 node-v27.0.0-v8-canary20260521da8e25c394-linux-x64.tar.xz.done
-rw-rw-r--    1 staging staging 131238376 May 21 16:02 node-v27.0.0-v8-canary20260521da8e25c394.tar.gz
-rw-rw-r--    1 staging staging         0 May 21 16:11 node-v27.0.0-v8-canary20260521da8e25c394.tar.gz.done
-rw-rw-r--    1 staging staging  65089448 May 21 16:11 node-v27.0.0-v8-canary20260521da8e25c394.tar.xz
-rw-rw-r--    1 staging staging         0 May 21 16:12 node-v27.0.0-v8-canary20260521da8e25c394.tar.xz.done

    which suggests something is going wrong with promotion for the canary builds.

  3. richardlau commented on May 21, 2026

    @richardlau
    richardlau
    on May 21, 2026
    Member

    This is the cause:

    dist@infra-digitalocean-ubuntu2204-x64-2:~$ /home/staging/tools/promote/promote_nightly.sh nodejs
Promoting v26.0.0-test5becfbc91f904c6bf93ca0098ce34694a1bb4aa3/node-v26.0.0-test5becfbc91f904c6bf93ca0098ce34694a1bb4aa3-win-x64.7z...
rm: cannot remove '/home/dist/nodejs/test/v26.0.0-test5becfbc91f904c6bf93ca0098ce34694a1bb4aa3/node-v26.0.0-test5becfbc91f904c6bf93ca0098ce34694a1bb4aa3-win-x64.7z': Permission denied
dist@infra-digitalocean-ubuntu2204-x64-2:~$

    File cannot be removed because the target directory is old enough that it was made write only for the dist user:

    total 93936
drwxr-xr-x   3 root root     4096 Apr 22 20:30 .
drwxr-xr-x 208 dist dist    20480 May  4 12:30 ..
-rw-r--r--   1 root root 24080359 Apr 22 20:00 node-v26.0.0-test5becfbc91f904c6bf93ca0098ce34694a1bb4aa3-win-x64.7z
-rw-r--r--   1 root root 38229424 Apr 22 20:00 node-v26.0.0-test5becfbc91f904c6bf93ca0098ce34694a1bb4aa3-win-x64.zip
-rw-r--r--   1 root root 33837056 Apr 22 20:00 node-v26.0.0-test5becfbc91f904c6bf93ca0098ce34694a1bb4aa3-x64.msi
-rw-r--r--   1 root root      742 Apr 22 20:30 SHASUMS256.txt
drwxr-xr-x   2 root root     4096 Apr 22 20:00 win-x64

    I believe the culprit is https://ci-release.nodejs.org/job/iojs+release-mefi-pgo/4/ (cc @StefanStojanovic ) which hasn't set a datestring field so it ends up reusing the target directory and promotion breaks because it cannot overwrite the now restricted directory. To fix this I'm going to removed the staged directory for v26.0.0-test5becfbc91f904c6bf93ca0098ce34694a1bb4aa3 so that the promote script doesn't try to overwrite the existing files.

  4. richardlau commented on May 21, 2026

    @richardlau
    richardlau
    on May 21, 2026
    Member

    https://nodejs.org/download/v8-canary/v27.0.0-v8-canary20260521da8e25c394/ now exists with AIX and Linux builds. As @targos mentioned, the other missing platforms will need a newer version of Rust.

  5. StefanStojanovic commented on May 22, 2026

    @StefanStojanovic
    StefanStojanovic
    on May 22, 2026

    @richardlau yes, it is my fault, and I apologize. One question: in the datestring description, it says Only needed for nightly, and I was using test builds. For future builds of this job (I plan to run some today), what should I do to prevent the issue I caused? Just add datestring?

  6. StefanStojanovic commented on May 22, 2026

    @StefanStojanovic
    StefanStojanovic
    on May 22, 2026

    I'll start another build soon. I'll put a datestring to avoid the issues described here.

  7. richardlau commented on Jun 4, 2026

    @richardlau
    richardlau
    on Jun 4, 2026
    Member

    @richardlau yes, it is my fault, and I apologize. One question: in the datestring description, it says Only needed for nightly, and I was using test builds. For future builds of this job (I plan to run some today), what should I do to prevent the issue I caused? Just add datestring?

    Yes, setting the datestring would be the simplest way.

  8. richardlau commented on Sep 18, 2026

    @richardlau
    richardlau
    on Sep 18, 2026
    Member

    Looking at current ci-release v8-canary jobs, the only recent failing platform is Windows.

    11:37:55 CUSTOMBUILD : Torque error : Cannot open file path/uri: src/builtins/wasm-to-js.tq"src/ebug/debug-wasm-objects.tq [c:\ws\tools\v8_gypfiles\run_torque.vcxproj]
11:37:55 C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): error MSB8066: Custom build for '..\..\out\Release\\torque.exe' exited with code 3. [c:\ws\tools\v8_gypfiles\run_torque.vcxproj]

    which looks like a quoting/delimiter issue?

    However it also does look like https://nodejs.org/download/v8-canary/ is behind what is on the DO droplet.

    @nodejs/web-infra I checked and it looks like we do have v27.0.0-v8-canary20260918239c3aa75c in R2's dist so this looks like another caching issue with https://github.com/nodejs/release-cloudflare-worker/ similar to the nightlies (nodejs/release-cloudflare-worker#1018).

  9. richardlau commented on Sep 18, 2026

    @richardlau
    richardlau
    on Sep 18, 2026
    Member

    Looking at current ci-release v8-canary jobs, the only recent failing platform is Windows.

    11:37:55 CUSTOMBUILD : Torque error : Cannot open file path/uri: src/builtins/wasm-to-js.tq"src/ebug/debug-wasm-objects.tq [c:\ws\tools\v8_gypfiles\run_torque.vcxproj]
11:37:55 C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): error MSB8066: Custom build for '..\..\out\Release\\torque.exe' exited with code 3. [c:\ws\tools\v8_gypfiles\run_torque.vcxproj]

    which looks like a quoting/delimiter issue?

    Above was from vs2022_clang-x64. Failure on vs202_clang-arm64:

    11:17:48 CUSTOMBUILD : Torque error : Cannot open file path/uri: src/builtins/wasm-to-j.tq [c:\ws\tools\v8_gypfiles\run_torque_host.vcxproj]
11:17:48 C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): error MSB8066: Custom build for '..\..\out\Release\\torque_host.exe' exited with code 3. [c:\ws\tools\v8_gypfiles\run_torque_host.vcxproj]

  10. MattIPv4 commented on Sep 18, 2026

    @MattIPv4
    MattIPv4
    on Sep 18, 2026
    Member
    • The DO droplet has v27.0.0-v8-canary20260918239c3aa75c (build is in progress so more platforms will appear as they complete).

    @nodejs/web-infra I checked and it looks like we do have v27.0.0-v8-canary20260918239c3aa75c in R2's dist so this looks like another caching issue with https://github.com/nodejs/release-cloudflare-worker/ similar to the nightlies (nodejs/release-cloudflare-worker#1018).

    How recent is this? There is caching in place, so I would not expect a new canary to be available instantly?

  11. richardlau commented on Sep 18, 2026

    @richardlau
    richardlau
    on Sep 18, 2026
    Member
    • The DO droplet has v27.0.0-v8-canary20260918239c3aa75c (build is in progress so more platforms will appear as they complete).

    @nodejs/web-infra I checked and it looks like we do have v27.0.0-v8-canary20260918239c3aa75c in R2's dist so this looks like another caching issue with https://github.com/nodejs/release-cloudflare-worker/ similar to the nightlies (nodejs/release-cloudflare-worker#1018).

    How recent is this? There is caching in place, so I would not expect a new canary to be available instantly?

    That's fair, but then all the builds between 20260904 to today are missing on https://nodejs.org/download/v8-canary/ but are present on the DO droplet and R2 dist. i.e.

    • v27.0.0-v8-canary202609153f670614ec
    • v27.0.0-v8-canary20260916045505e938
    • v27.0.0-v8-canary20260917d5e7bad223

    we didn't have any builds between 4 September and 15 September. The directory cache for the worker was manually refreshed on 7 September.

  12. MattIPv4 commented on Sep 18, 2026

    @MattIPv4
    MattIPv4
    on Sep 18, 2026
    Member

    🤔 https://github.com/nodejs/release-cloudflare-worker/actions/workflows/update-links.yml runs every day and is meant to update the directory cache. @nodejs/web-infra I'm not sure I know enough about how the worker works to know why this cache isn't refreshing? Does the canary directory need to be added to https://github.com/nodejs/release-cloudflare-worker/blob/1bb1f5403a333564dde7d99dcc6cb75a01697a4a/scripts/constants.mjs#L61-L65 perhaps?

  13. targos commented on Sep 18, 2026

    @targos
    targos
    on Sep 18, 2026
    Member

    The Windows build should be (at least partially) fixed by nodejs/node@8273447

    We can either wait for V8 15.2 to land on main or cherry-pick that fix to canary-base (I don't have the permission for that)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions