SLOW OR FAILED (500 ERROR) NODE.JS DOWNLOADS

^{Edited by the Node.js Website Team}

⚠️ PLEASE AVOID CREATING DUPLICATED ISSUES

Learn more about this incident at https://nodejs.org/en/blog/announcements/node-js-march-17-incident

tl;dr: The Node.js website team is aware of ongoing issues with intermittent download instability.

More Details: nodejs/build#1993 (comment)

_{Original Issue Below}

URL: https://nodejs.org/dist/latest-v16.x/node-v16.14.1-linux-x64.tar.xz (or basically any specific file, as opposed to just browsing the dirs, which mostly works)
Browser version: Firefox 98.0.0, Firefox Nightly 100.0a1 (2022-03-15), curl 7.47.0 (Travis CI), curl 7.68.0 (my local machine) (Doesn't feel like a browser issue, feels like a server issue...)
Operating system: Ubuntu 20.04 / Linux Mint 20

When trying to get files off of nodejs.org/dist/... or nodejs.org/download/..., I get a server error.

"500 Internal Server Error"

(error page served by nginx)

Full error message page (HTML snippet, click to expand)

<html>
<head><title>500 Internal Server Error</title></head>
<body bgcolor="white">
<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx</center>
</body>
</html>

Browsing around the dirs, like https://nodejs.org/dist/latest-v16.x/, seems to work. Also, Downloading really small files such as https://nodejs.org/dist/latest-v16.x/SHASUMS256.txt seems to work sporadically, whereas downloading tarballs doesn't seem to work.

Given that the outage seems sporadic: Maybe it's a resource exhaustion issue over at the server? Running out of RAM or something?? I don't know.

~~Edit to add: The error message page seems to be served by Cloudflare. (According to the server: cloudflare response header, when looking in browser dev tools). So I guess this is a Cloudflare issue?~~ Actually that's probably not what that means.

For some reason, iojs.org redirected to nodejs.org, causing errors with nvm (https://github.com/SheetJS/sheetjs/runs/5564331772 test failed when trying to fetch https://iojs.org/dist/v3.3.1/iojs-v3.3.1-linux-x64.tar.xz)

I wonder if the site/Cloudflare are just getting overwhelmed with traffic for people downloading the latest release?

Node v16.14.1 was released about an hour ago... #4494 / https://github.com/nodejs/node/releases/tag/v16.14.1

It's the latest LTS release, so everybody in the world looking for the default version of Node is downloading uncached right now, basically. (Other than as apparently cached by Cloudflare).

Should settle down in a few hours, or by tomorrow??

Note: I have nothing to do with Node's web servers or site, I'm just a regular person speculating at this point.

Closing as the issue isn't happening right now, but also pinging @nodejs/build in case there's anything to add here or something to be aware of.

People are still reporting this over in the OpenJS Slack, so I'm going to re-open this until that gets sorted.

I have CloudFlare LB error notices spanning from ~2:40 am UTC to ~3:40 am UTC, which I suppose correlates with the latest 16.x release. Overloaded primary server, switching to the backup. Users may have got the 500 while hitting the primary before it switched.

I don't have a good explanation beyond that, I'm not sure what's caused the overload, I don't believe that server actually has trouble serving, but we have witnessed some weird I/O issues connected with rsync that maybe all happened at the same time. Perhaps during the next release someone should be on the server watching for weirdness.

This seems to be happening again; https://iojs.org/dist/index.json is 500ing, and v12.22.11 just went out.

(after about 5-10 minutes, the 500s seemed to stop, but it's still worth looking into)

FWIW I am still seeing either "500 internal server error" or very slow file downloads (less than 50 kilobytes a second. Often less than 10 kilobytes a second, especially at the beginning of a download.)

More details (click to expand):

The "slow download" symptom is less obvious for small files, because they complete quickly anyway.

I have seen a tarball download start, take a long time (over a minute), and ultimately fail to download mid-way through.

I hope that's useful to diagnose the problem. (Results may vary across the globe, since the path through the CDN is probably not identical everywhere?)

Edit to add: My experience is basically identical (as described in this comment) for nodejs.org/dist/ and iojs.org/dist. And identical experience today compared to 2 days ago.

@mhdawson

After promoting 12.22.11 I went to rerun the same release script to promote 14.19.1 and it just hung during the bit when it fetches the promotable builds. Same behaviour from another machine on a different network. Weirdly I was able to ssh into the machine in an interactive session and manually run the command the promotion script was trying to run 😕.

I don't have a good explanation beyond that, I'm not sure what's caused the overload, I don't believe that server actually has trouble serving, but we have witnessed some weird I/O issues connected with rsync that maybe all happened at the same time. Perhaps during the next release someone should be on the server watching for weirdness.

I logged into the machine, ran ps -ef and saw we had two rsyncs in progress

root     12174 12137  0 00:08 ?        00:00:00 rsync --server --sender -logDtprze.iLsfxC . /home/dist/iojs/
nodejs   27646 27638  0 00:00 ?        00:00:00 ssh benchmark rsync --server --sender -logDtprze.iLsfx . coverage-out/out/

The first of those, running as root, is from the backup server -- I've left that alone. I killed the second one, which is the coverage data, and suddenly my other terminal running the promotion script got unstuck. This would kind of suggest the coverage data syncing is partly, or wholly, responsible. In the past either @mhdawson or I would do an annoying manual clear up of old coverage data to reduce the volume of stuff being synced but I'm going to recommend now we turn off running coverage on Jenkins and the associated data shuffling to populate coverage.nodejs.org and switch exclusively to codecov.io.

Actually as I typed the above paragraph my promotion execution has broken 😢 so there's still something up with the machine:

Are you sure you want to promote the v14.19.1 assets? [y/n] y
client_loop: send disconnect: Broken pipe

Currently there are no rsync processes -- but running the promotion script is "hung" again for me 😞.
These are the top things via top:

top - 00:33:22 up 195 days, 16:59,  1 user,  load average: 0.72, 0.80, 0.84
Tasks: 165 total,   2 running, 163 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.7 us,  3.3 sy,  0.0 ni, 83.6 id,  0.1 wa,  0.0 hi,  8.2 si,  0.1 st
KiB Mem : 16432136 total,  1251072 free,  1222728 used, 13958336 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 14391196 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
25151 www-data  20   0  595256 456632 375512 S   9.0  2.8 510:58.65 nginx
25149 www-data  20   0  595796 456892 375408 S   8.3  2.8 529:56.28 nginx
25152 www-data  20   0  597244 458740 375596 S   8.0  2.8 524:21.40 nginx
25150 www-data  20   0  594556 456176 375488 S   7.3  2.8 505:23.46 nginx
25153 www-data  20   0  596728 458372 375512 S   7.0  2.8 517:06.30 nginx
25154 www-data  20   0  595496 456944 375552 S   6.3  2.8 515:04.23 nginx
   23 root      20   0       0      0      0 R   5.0  0.0 238:39.12 ksoftirqd/3
12273 telegraf  20   0 6316820  38820   7492 S   3.0  0.2  17:58.94 telegraf
   58 root      20   0       0      0      0 S   2.0  0.0   2773:53 kswapd0
    7 root      20   0       0      0      0 S   0.7  0.0 595:13.23 rcu_sched
   18 root      20   0       0      0      0 S   0.3  0.0  12:58.42 ksoftirqd/2
16257 root      20   0   42092   3784   3128 R   0.3  0.0   0:00.13 top

I think I'm going to bounce nginx.

Happening same here. Know this type of comments are not useful in most cases, but in this case at least serves to know that there are users still being affected 😬

Edit: worked now for me

I did systemctl restart nginx. I'm still seeing rsync processes from the backup server still running.

@pughpugh

I actually just filed a parallel ticket for that one @pughpugh before I saw this: #5818 . Not sure why Cloudflare is caching 500s

Seen on an Azure DevOps agent:

Downloading: https://nodejs.org/dist/v18.17.1/node-v18.17.1-linux-x64.tar.gz
##[error]Aborted
##[warning]Content-Length (44417128 bytes) did not match downloaded file size (8204861 bytes).

gyp ERR! stack Error: 500 status code downloading checksum

Please reopen as it still happens multiple times a day!

Same, still happening

we were facing this as well for quite some time. only solution to mitigate this is to use cached nodejs that ships with container :-/ not the best thing in the world, but it is what it is microsoft/fluentui#29552

Have been facing this intermittently too:

Downloading: https://nodejs.org/dist/v16.16.0/node-v16.16.0-win-x64.7z
##[error]Aborted
##[warning]Content-Length (17121358 bytes) did not match downloaded file size (5534315 bytes).

and happening again :(

Downloading: https://nodejs.org/dist/v16.16.0/node-v16.16.0-darwin-x64.tar.gz
##[error]Aborted
##[warning]Content-Length (30597385 bytes) did not match downloaded file size (5075560 bytes).

This was happening continually today at https://nodejs.org/dist/v20.8.1/node-v20.8.1-darwin-arm64.tar.gz

Any update on why this is closed?

This is closed because it's a known issue, and we're working on it.

I've been getting this on GitHub Actions (ubuntu-latest) for the last couple days:

> Could not get resource 'https://nodejs.org/dist/v18.14.0/node-v18.14.0-linux-x64.tar.gz'.
            > Read timed out

@ovflowd

@ovflowd

This is closed because it's a known issue, and we're working on it.

I think it would better communicate your intent if you closed this issue after you have solved it.

revert back to official nodejs mirror

revert back to official nodejs mirror (#265692)

Navigation Menu

Repository navigation

SLOW OR FAILED (500 ERROR) NODE.JS DOWNLOADS #4495

⚠️ PLEASE AVOID CREATING DUPLICATED ISSUES

44 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

SLOW OR FAILED (500 ERROR) NODE.JS DOWNLOADS #4495

Description

⚠️ PLEASE AVOID CREATING DUPLICATED ISSUES

Activity

SheetJSDev commented on Mar 16, 2022

DeeDeeG commented on Mar 16, 2022

DeeDeeG commented on Mar 16, 2022

Trott commented on Mar 16, 2022

Trott commented on Mar 16, 2022

rvagg commented on Mar 16, 2022

ljharb commented on Mar 17, 2022

DeeDeeG commented on Mar 18, 2022

richardlau commented on Mar 18, 2022

richardlau commented on Mar 18, 2022

richardlau commented on Mar 18, 2022

KevLehman commented on Mar 18, 2022

richardlau commented on Mar 18, 2022

44 remaining items

broksonic21 commented on Sep 20, 2023

joaomoreno commented on Oct 10, 2023

shailendra043 commented on Oct 17, 2023

pbering commented on Oct 18, 2023

jeremills commented on Oct 18, 2023

Hotell commented on Oct 18, 2023

jfuinsure commented on Oct 18, 2023

jfuinsure commented on Oct 19, 2023

broksonic21 commented on Oct 24, 2023

ovflowd commented on Oct 24, 2023

iansedano commented on Oct 26, 2023

horttanainen commented on Oct 31, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions