Repository navigation
node-www out of space? #2162
Description
Lots of the release builds failed overnight
e.g. https://ci-release.nodejs.org/job/iojs+release/nodes=centos7-ppcle-release-64/5054/console
10:56:07 ssh node-www "mkdir -p nodejs/release/v13.8.0"
10:56:08 mkdir: cannot create directory ‘nodejs/release/v13.8.0’: No space left on device
10:56:08 Makefile:1139: recipe for target 'binary-upload' failed
Activity
targos commentedon Feb 5, 2020
Ping @nodejs/build-infra
targos commentedon Feb 5, 2020
@mhdawson @jbergstroem @joaocgreis @rvagg
sam-github commentedon Feb 5, 2020
For the future: maybe someone on the release team should have build-infra access? Or at least node-www?
targos commentedon Feb 5, 2020
We do have limited access to the machine, but not root.
I found that the releases (mostly nightlies) take up about all the available space (2 TB).
sam-github commentedon Feb 5, 2020
We have nightlies going back to 2016
https://nodejs.org/download/nightly/
I might misunderstand their value, but I don't see why keeping more than a few weeks is useful.
addaleax commentedon Feb 5, 2020
I’ve occasionally used them for bisecting Node.js – it’s definitely faster than building Node.js from scratch multiple times.
But yeah, they should be low-priority and deleting one or two years’ worth of nightlies should be just fine, imo.
joyeecheung commentedon Feb 5, 2020
Do we have backup for these? I guess we could save them somewhere else and bring them back when we have more space? (Or ideally put these nightlies in some cloud storage?)
EDIT: oh but if they are all reproducible I guess we just need to save the commit hashes and could always rebuild them all later.
rvagg commentedon Feb 5, 2020
Gee, that filled up much quicker than anticipated. Doing an analysis now of what we have but in the meantime I've freed up ~3G of stuff littering staging so there should be space to publish some releases if needed now @BethGriggs.
rvagg commentedon Feb 5, 2020
OK, I have 38G of space freed up for breathing room while we decide next steps.
Here's a breakdown of dist/nodejs which takes up the majority of space on that device:
Some options:
BethGriggs commentedon Feb 5, 2020
Thanks, @rvagg. I'll start kicking off the new release builds now - https://ci-release.nodejs.org/job/iojs+release/5055 (v13.x)
rvagg commentedon Feb 6, 2020
I'm expanding on-server storage for now, at least as an interim measure since I doubt we'll make these decisions quickly.
targos commentedon Feb 11, 2020
I think we should have an incremental removal policy for both v8-canary and nightly builds.
Something like:
rvagg commentedon Feb 12, 2020
^ that would be a good basis for a PR for a new policy doc so we can nail it down and agree on what we're going to do (if anything).
For now I've finalised an upgrade which doubles the size of that disk, so we have 2Tb of headroom now before needing to take more action. Old disk still mounted for reference but I'll discard it at some point in the near future.