Repository navigation
[BUG] Npm opens many connections when installing #7072
Description
Is there an existing issue for this?
- I have searched the existing issues
This issue exists in the latest npm version
- I am using the latest npm
Current Behavior
My project needs around 2000 packages. When running "npm install" it starts out by opening 2000 connections and then when all the connections are open it downloads 1 package on each connection.
Is this intended?, because it causes a quite high load on our package proxy. Is there a way to limit the number of connections?
With npm 9.9 it only uses 18/20 connections.
Expected Behavior
"npm install" only opens a few connetions to the package server.
Steps To Reproduce
- Clear cache
- npm install or npm ci
Environment
- npm: 10.2.5
- Node.js: 18.19.0
- OS Name: Windows 11
- System Model Name:
- npm config:
registry = "http://fosspackages/npm/default-npm/"
save-exact = true
save-prefix = ""
; node bin location = C:\Users\lj\.nvm\versions\node\v18.19.0\bin\node.exe
; node version = v18.19.0
; npm local prefix = C:\work\Dexter\web-client
; npm version = 10.2.5
; cwd = C:\work\Dexter\web-client
; HOME = C:\Users\lj
; Run `npm config ls -l` to show all defaults.
Activity
Larsjep commentedon Dec 12, 2023
I noticed that there is a config option called maxsockets. But the default is 15 and
npm config get maxsocketsalso returns 15.So if this should limit the number of connections it doesn't seem to work in npm 10.x.
MartinFalatic commentedon Dec 20, 2023
@Larsjep Just curious what
npm config get agentshows. Supposedly if that's set,maxsocketswill be ignored innpm-registry-fetch.Could also be that's something that's buried in NPM and is somehow overridden in some other way, but it caught my eye when I searched the org for
maxsockets.... might also help narrow down the problem at least.Larsjep commentedon Dec 20, 2023
@MartinFalatic
npm config get agentreturnsundefinedI have also looked in the code base, but I can't find anywhere it refers to the
maxsocketssettings.I can find it in the 9.x codebase, but the download part seem to radically changed in 10.x
rosen-dimitrov commentedon Dec 20, 2023
+1 here.
We are using self-hosted runners(ARC) on our own cluster and when using npm 10 we noticed a lot of dropped packets due to NAT ports exhaustion all coming from the arc runner executing npm install with npm 10. We have dynamic port allocation, which means that npm creates a lot of connections in a burst at the same time before the new ports can be allocated to the worker node running the runner pod.
Npm 9 does not seem to have this issue. Looks an issue with the new
@npmcli/agenthttp client introduced in npm 10.What is also interesting is that the workflow does not fail. npm seems to use whatever number of connections it managed to establish to install all packages.
mirayashi commentedon Dec 20, 2023
Hello there,
I would like to add that this is actually a bigger issue for very large projects (3000+ dependencies):
npm installrandomly fails with ECONNRESET and ETIMEDOUT errors (even more consistently on Windows), and seems to ignore most of .npmrc options (maxsockets, fetch-retries, etc)Th3S4mur41 commentedon Jan 8, 2024
We're affected by the same issue...
Our Jenkins CI is constantly failing to install private dependencies from GitHub (through our Proxy).
Two potential workarounds are disabling the proxy (can't do that in corporate environment) or downgrading to npm 9
silverwind commentedon Feb 5, 2024
It seems the
maxsocketsoption (default value 15) has no effect at all innpm@10, at least in situations where there is no proxy involved.When I run a
npm installwith around 1k dependencies inside a local VMWare VM, I'm consistently hitting a bug in VMWare where npm exhausts the CPU resources ofvmnet-natd, which leads to npm hanging for minutes and finally terminating with random network errors. Downgrading tonpm@9completely resolves it.ajones513 commentedon Feb 6, 2024
+1 here
My company is having to hold back NPM 10 because of this issue when connecting to a private self-hosted registry. To help with our decision making, is there any indication when this might be triaged?
greg-md commentedon Feb 21, 2024
Same issue encountered on our vm lima. After upgrading to node v20 and npm v10, npm install freezes the entire vm. Downgrading to v9 fixed the issue.
llimllib commentedon Feb 23, 2024
We are seeing about half of our
npm ciruns with npm version 10 on github actions fail; downgrading to npm version 9 speeds up install by about 45 seconds (2m45 -> 2m) and doesn't seem to fail at all.I can reproduce easily so please let me know if I can help debug this somehow
dharmeshgordhan commentedon Mar 8, 2024
Potentially being looked at under #7272.
melroy89 commentedon Mar 10, 2024
My gitlab ci/cd is also very unstable due to this behavior. Often causing time outs due to the large amount of connections. Especially since I have some jobs running in parallel as well.
38 remaining items
🌱 Update project to use nodejs 20, npm >=10.5.2
🌱 Update project to use nodejs 20, npm >=10.5.2
🌱 Update project to use nodejs 20, npm >=10.5.2
🌱 Update project to use nodejs 20, npm >=10.5.2 (#2062)
🌱 Update project to use nodejs 20, npm >=10.5.2 (konveyor#2062)
🌱 Update project to use nodejs 20, npm >=10.5.2 (#2062) (#2064)
Set minimum node version to 20.13.0
👻 Change minimum node version to 20.13.0 (#38)
Remove --maxsockets 1
Remove --maxsockets 1 (#7070)
Remove --maxsockets 1 (#7070)
Add operation to update User email correctly (#7184)
Remove --maxsockets 1 (#7070)
Add operation to update User email correctly (#7184)
🌱 Update project to use nodejs 20, npm >=10.5.2 (konveyor#2062)