SLOW OR FAILED (500 ERROR) NODE.JS DOWNLOADS

@targos

^{Edited by the Node.js Website Team}

⚠️ PLEASE AVOID CREATING DUPLICATED ISSUES

Learn more about this incident at https://nodejs.org/en/blog/announcements/node-js-march-17-incident

tl;dr: The Node.js website team is aware of ongoing issues with intermittent download instability.

More Details: #1993 (comment)

_{Original Issue Below}

All binary downloads from nodejs.org are very slow currently. E.g. https://nodejs.org/dist/v13.0.1/node-v13.0.1-darwin-x64.tar.xz.

In many cases, they will time out, which affects CI systems like Travis who rely on nvm to install Node.js.

According to @targos, the CPU of the server is spinning 100% being used by nginx, but we can't figure out what's wrong.

Top of the top:

@nodejs/build-infra

Same here, using tj/n, the download just terminates after a while

Azure DevOps pipelines also suffer, that's because a standard Node.js setup task downloads the binary from nodejs.org.

Kudos to those who are looking into the CPU usage ✌️ You'll nail it! 🙌

@targos

@targos what machine is the top output from?

Fwiw, only infra team members are: https://github.com/orgs/nodejs/teams/build-infra/members and Gibson isn't current, he should be removed.

I wonder what traffic looks like... could be a DOS

We've talked about it in the past, but we really should be serving these resources via a CDN... I believe that the issue was maintaining our metrics. I can try and find some time to help out with the infrastructure here if others don't have the time

@MylesBorins

It could also be that 13.x is super popular? Hard to know without access to access logs.

@MylesBorins the infrasture team is even less staffed than the build team as a whole, if you wanted to pick that up, it would be 🎉 fabulous.

@joaocgreis

The issue has been the metrics in terms of why they are not on a CDN, @joaocgreis and @rvagg do have an active discussion on what we might do on that front.

@MylesBorins if you can get help for this specific issue that might be good (please check with @rvagg and @joaocgreis ). Better would be ongoing sustained engagement and contribution to the overall build work.

@sam-github

@sam-github it's from direct.nodejs.org. I don't have root access to the machine, so top is basically the only thing I could do to help.

I don't even have non-root access, I guess you do because you are on the release team.

We'll have to wait for an infra person.

I can help take a look after the TSC meeting.

In addition to downloads only intermittently succeeding, when they do succeed it seems that sometimes (?) the bundled node-gyp is broken. I'm seeing that behavior in CI builds on Azure Pipelines.

For linux builds with successful downloads of Node.js v8.x and v10.x, node-gyp fails to build sources that were previously unproblematic.

For windows builds with successful downloads of Node.js v8.x and v10.x, I'm seeing errors like this:

gyp ERR! stack Error: win-x86/node.lib local checksum 629a559d347863902b81fad9b09ba12d08d939d20a0762578e904c6ee1a7adab not match remote fea7d0c5a94fc834a78b44b7da206f0e60ef11dd20d84dc0d49d57ee77e20e16

That is, in the case of windows node-gyp doesn't attempt to compile; rather, it fails outright owing to the checksum error.

@michaelsbradleyjr

@michaelsbradleyjr I suspect that the node-gyp failures would have been related to downloading header files.

@targos

Was looking into this with @targos. It seems to have resolved itself at this point.

Looking at the access logs it was a bit hard to tell if there was more traffic or not as the files for the previous day are compressed. We should be able to compare tomorrow more easily.

Top on the machine looks pretty much the same as it did while there were issues as well

Thank you to everyone for their hard work, sending virtual 🤗.

Apparently encountering this type of issue still. For the past two days I get around a 20% failure rate consistently:

$ for i in {01..10}; do echo "$i"' '"$(date -u)"; curl https://nodejs.org/dist/v12.13.1/node-v12.13.1-linux-x64.tar.xz -o node-v12.13.1-linux-x64.tar.xz; done
01 Wed Nov 20 19:46:43 UTC 2019
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.4M  100 13.4M    0     0   217k      0  0:01:03  0:01:03 --:--:-- 1620k
02 Wed Nov 20 19:47:46 UTC 2019
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.4M  100 13.4M    0     0  4237k      0  0:00:03  0:00:03 --:--:-- 4236k
03 Wed Nov 20 19:47:49 UTC 2019
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.4M  100 13.4M    0     0  1651k      0  0:00:08  0:00:08 --:--:-- 1247k
04 Wed Nov 20 19:47:57 UTC 2019
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 62 13.4M   62 8527k    0     0  28569      0  0:08:12  0:05:05  0:03:07 19759
curl: (18) transfer closed with 5335120 bytes remaining to read
05 Wed Nov 20 19:53:02 UTC 2019
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.4M  100 13.4M    0     0  3489k      0  0:00:03  0:00:03 --:--:-- 3488k
06 Wed Nov 20 19:53:06 UTC 2019
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 83 13.4M   83 11.2M    0     0  25695      0  0:09:07  0:07:39  0:01:28 19389
curl: (18) transfer closed with 2255352 bytes remaining to read
07 Wed Nov 20 20:00:46 UTC 2019
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.4M  100 13.4M    0     0  4398k      0  0:00:03  0:00:03 --:--:-- 4398k
08 Wed Nov 20 20:00:49 UTC 2019
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.4M  100 13.4M    0     0  2798k      0  0:00:04  0:00:04 --:--:-- 2798k
09 Wed Nov 20 20:00:54 UTC 2019
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.4M  100 13.4M    0     0  2340k      0  0:00:05  0:00:05 --:--:-- 2282k
10 Wed Nov 20 20:01:00 UTC 2019
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.4M  100 13.4M    0     0  2149k      0  0:00:06  0:00:06 --:--:-- 2197k

@matsaman

Where are you @matsaman? could be a cloudflare problem in your region. See https://www.cloudflarestatus.com/ and check whether one of the edges close to you is having trouble. Without further reports of problems or something we can reproduce ourselves our best guess is going to be that it's a problem somewhere between Cloudflare and your computer since CF should have it cached.

Items on that page in my region say 'Operational' at this time. Should I encounter the issue again I will check that page after my tests and include my findings here.

Hi,

I am still facing this problem on Azure DevOps.
Do we have a fix for this by any chance?

Thanks in advance.

I'm getting fast downloads. Please open a new issue if you believe there is still a problem versus commenting on this close issue.

@appu24

@appu24 tar failure doesn't necessarily indicate a download error. Are you able to get more log output for that to see exactly what it's failing on. I would assume something in the chain is generating some stderr logging at least.

very slow download for me

Experiencing issues for weeks now; from different IPs and servers. Is there any known reason for this?

Is there any known reason for this?

Yes, the origin server is being overloaded with traffic, in part due to every release of Node.js purging the Cloudflare cache for the entire domain.

There are a bunch of issues open that aim to help rectify this:

Request: Cloudflare configuration update #3473 (fix some edge cases bypassing the cache currently)
Cloudflare R2 Workers #3461 (replace the overwhelmed origin with Cloudflare R2)
Restrict NGINX to Cloudflare IPs only #3439 (disallow folks bypassing Cloudflare cache and hitting the origin directly)
DigitalOcean www server #3424 (meta issue for the overwhelmed origin as it is currently)
Redesign Cloudflare cache purging #3410 (changing how cache purges happen for releases so the entire cache isn't wiped)

nodejs.org file delivery is currently not working again. This might be a working mirror:
https://mirrors.cloud.tencent.com/nodejs-release/

Hi there, can't download node. Continued "Failed - Network Error" from my mac

Navigation Menu

Repository navigation

SLOW OR FAILED (500 ERROR) NODE.JS DOWNLOADS #1993

⚠️ PLEASE AVOID CREATING DUPLICATED ISSUES

43 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

SLOW OR FAILED (500 ERROR) NODE.JS DOWNLOADS #1993

Description

⚠️ PLEASE AVOID CREATING DUPLICATED ISSUES

Activity

targos commented on Oct 23, 2019

targos commented on Oct 23, 2019

atulmy commented on Oct 23, 2019

kachkaev commented on Oct 23, 2019

sam-github commented on Oct 23, 2019

MylesBorins commented on Oct 23, 2019

sam-github commented on Oct 23, 2019

mhdawson commented on Oct 23, 2019

targos commented on Oct 23, 2019

sam-github commented on Oct 23, 2019

mhdawson commented on Oct 23, 2019

michaelsbradleyjr commented on Oct 23, 2019

mhdawson commented on Oct 23, 2019

mhdawson commented on Oct 23, 2019

mhdawson commented on Oct 23, 2019

43 remaining items

OriginalEXE commented on Nov 6, 2019

matsaman commented on Nov 20, 2019

rvagg commented on Nov 21, 2019

matsaman commented on Nov 22, 2019

appu24 commented on Jan 14, 2020

mhdawson commented on Jan 14, 2020

rvagg commented on Jan 14, 2020

tbsvttr commented on Aug 21, 2023

saschanos commented on Aug 30, 2023

MattIPv4 commented on Aug 30, 2023

hellmediadev commented on Sep 27, 2023

stevenmettler commented on Oct 25, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions