Repository navigation
Disk full on downloads server #3125
Description
Found out because release builds fail to push the binaries:
07:13:16 mkdir: cannot create directory ‘nodejs/nightly/v20.0.0-nightly202212191d6fda0856’: No space left on device
{{ message }}
Activity
targos commentedon Dec 19, 2022
targos commentedon Dec 19, 2022
targos commentedon Dec 19, 2022
@nodejs/build-infra
What should we do? Has this already happened before?
richardlau commentedon Dec 19, 2022
Yes, for example:
sxa commentedon Dec 19, 2022
I think what @targos suggested in #2162 (comment) is reasonable and I've got some scripts that I use elsewhere which can likely enforce that if we're happy with not retaining all 2Tb of nightlies that we're storing at present.
As a "quick fix" I've moved some the v9 nightlies to
/root/nightlyrather than deleting them which will give us a bit of grace pending any objections to implementing something similar to the above.The obvious concern would be that we might want to do it separately for each version since there were overlapping major versions in the nightlies.
Do we want to run the deletion of old stuff past the TSC for approval before implementing anything here?
sxa commentedon Dec 19, 2022
@rvagg Do you know the history of the
/mnt/nodejs_org_home_oldfile system? As theoldwould imply it doesn't seem to have been touched in almost three years so I guess it's the one referenced in #2162 (comment) which you hadn't got round to discarding? I could clear it and use that as a temporary area while testing out a replacement withmvcommands rather than rm`mhdawson commentedon Dec 19, 2022
I agree as well that what @targos suggested in #2162 (comment) makes sense.
I think if we reach consensus an FYI to the TSC in advance of implementing would make sense in case there are any concerns.
sxa commentedon Dec 19, 2022
I've got a prototype script that I've tested which will:
The above will delete 2708 of the 2913 Node.js nightlies given the status at the time of posting.
rvagg commentedon Dec 24, 2022
if something is still mounted with "old" in its name it can probably be deleted - that smells like me doing a disk resize upgrade and leaving the old one around for "some period of time" to be sure that the sync properly happened; it obviously did because I don't remember when I last did this
+1 to pruning nightlies, I think the only reason we've baulked on scripting it is that people do rely on them for benchmarking and other hackery -- "here's my cool thing but you need to be running Node.js nightly .... which you can download from here". Question for the TSC with input from other folks that might be prone to do this kind of thing.
rvagg commentedon Dec 24, 2022
or .. we could just up the disk yet again .. or even better, move to an object store model like we've been talking about since almost the beginning of this current layout.
targos commentedon Jan 3, 2023
The disk is full again.
mhdawson commentedon Jan 4, 2023
From chat in slack for record:
Michael Dawson (New)
< 1 minute ago
I propose we just delete the
150G ./chakracore-nightly
directory
Michael Dawson (New)
< 1 minute ago
chakracore was abandoned by MS a long time ago and I don't think anybody should be using the old nightlies
Michael Dawson (New)
< 1 minute ago
not sure how long that would give us but it would unblock for now I assume
6 remaining items
sxa commentedon Jan 10, 2023
Good shout on the chakracore nightlies given that we do still have the releases if people want to play with that version.
However I've reinstated the 'last' one for a few of the versions back across in case anyone wants them since it includes a Node v11 and a v12 which isn't in the chakracore-release directory
sxa commentedon Jan 10, 2023
I'm not too familiar with the canaries - how are they identified and what would the requirements be for those? I was planning to implement the above later today but I can hold off if you think there's a concern here.
targos commentedon Jan 10, 2023
I think we can treat canaries the same as nightlies.
sxa commentedon Jan 11, 2023
Pruning now ... This is the script I'm using for it if anyone wishes to enhance it (Stored in
/home/dist). In addition to what I said previously it should try to retain the latest one (by the sorting done byls) . We can set this to run as a cronjob (Needs to run as root since that's what the other of the files are)prune.sh.gz
sxa commentedon Jan 11, 2023
I'll leave this for any further comments before proceeding. I have NOT updated the index.tab/index.json in those directories (this hasn't been done fo the chakracore-nightlies either) but I goess we should clean them up too with the deletions so I believe the remaining actions are:
sxa commentedon Jan 11, 2023
For reference, this is the current size of the different nodejs artifact directories - there are 1385 v8-canary builds in here totalling 829Gb
mhdawson commentedon Jan 13, 2023
@sxa I assume that for the v8-canary that is before you have pruned them similarly to what you did for nightlies?
sxa commentedon Jan 25, 2023
Correct
mhdawson commentedon Jan 25, 2023
@sxa then I figure we should just prune the canaries in the same was as for nightlies and see if there is an issue
mhdawson commentedon Jan 27, 2023
@targos any concerns if we apply the same scripts that we used/are planning to run regularly on the nightlies to prune older builds.
sxa commentedon Feb 2, 2023
ping @targos - if no objections in the next day or so I'll plan to implement this automation on canaries and nightlies at the start of next week.
targos commentedon Feb 3, 2023
I have no objections, but note that while I maintain canary builds, I personally almost never use them.
sxa commentedon May 23, 2023
crontab enabled for both nightlies and v8 canaries, so closing this. Scheduled once a month and the logs are written to
/home/distif anyone wants to analyse them in the case of any problems (Since it's only being run once a month they will be available for a while for analysis)