Out of space on www server

@RafaelGSS

Reported by @RafaelGSS when trying to promote 19.5.0:

# Promoting v19.5.0...
... Checking assets
... Expecting a total of 39 assets for v19.x
... 39 assets waiting in staging
... 0 assets already promoted
 ✓  Complete set of expected assets in place for v19.x
Are you sure you want to promote the v19.5.0 assets? [y/n] y
Promoting v19.5.0/docs...
...
Promoting v19.5.0/win-x86...
/home/nodejs/queue-cdn-purge.sh: line 13: echo: write error: No space left on device

This is a different disk (/dev/vda1) to #3125 and is not the disk that holds the downloads.

Filesystem      Size  Used Avail Use% Mounted on
udev            7.9G     0  7.9G   0% /dev
tmpfs           1.6G  3.8M  1.6G   1% /run
/dev/vda1       315G  302G     0 100% /
tmpfs           7.9G   76K  7.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda        3.9T  1.7T  2.0T  47% /home
/dev/sdb        2.0T  1.8T   27G  99% /mnt/nodejs_org_home_old
tmpfs           1.6G     0  1.6G   0% /run/user/0

I'm guessing we might want to look at /var/log:

# du -hs /var/*
12M     /var/backups
123M    /var/cache
4.0K    /var/crash
101G    /var/lib
4.0K    /var/local
0       /var/lock
181G    /var/log
4.0K    /var/mail
4.0K    /var/opt
0       /var/run
4.0K    /var/snap
28K     /var/spool
36K     /var/tmp
12K     /var/www

Even after freeing it, I think we might need to clean up all v19.5.0 artifacts present. Otherwise, the release script will return:

# Checking for releases ...
No releases to promote!

@RafaelGSS

@RafaelGSS Once we fix the space problem I think you can rerun the release script with -s:

# To _only_ sign an existing release, use:
#  release.sh -s vx.y.z

The binaries were moved from staging to the downloads, https://nodejs.org/download/release/v19.5.0/ exists (but has no shasums).

Are all the binaries there? from the logs looks like some of them might be conflicted

The largest logs are nginx:

# du -hs /var/log/*/
108K    /var/log/apt/
4.0K    /var/log/cloudflare/
4.0K    /var/log/dist-upgrade/
4.0K    /var/log/fsck/
4.0K    /var/log/iptraf/
4.1G    /var/log/journal/
160K    /var/log/letsencrypt/
4.0K    /var/log/lxd/
176G    /var/log/nginx/
4.0K    /var/log/telegraf/
12K     /var/log/unattended-upgrades/

of which

# du -hs /var/log/nginx/*/
35M     /var/log/nginx/benchmarking/
449M    /var/log/nginx/coverage/
33G     /var/log/nginx/iojs/
128M    /var/log/nginx/libuv/
136G    /var/log/nginx/nodejs/
5.0G    /var/log/nginx/nodejs-backup/
2.3G    /var/log/nginx/old/

Are all the binaries there? from the logs looks like some of them might be conflicted

I've checked the content size of the binaries and it looks ok. So, when it's freed let me know to sign the release. Thank you!

@RafaelGSS

@RafaelGSS They're no longer in staging. Both staging and downloads are under /home which has 2 TB of free space.

I've removed /root/nightly (#3125 (comment)) which has recovered 16G while we work out what to do about /var/log/.

# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.9G     0  7.9G   0% /dev
tmpfs           1.6G  3.7M  1.6G   1% /run
/dev/vda1       315G  287G   16G  95% /
tmpfs           7.9G   76K  7.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda        3.9T  1.7T  2.0T  47% /home
/dev/sdb        2.0T  1.8T   27G  99% /mnt/nodejs_org_home_old
tmpfs           1.6G     0  1.6G   0% /run/user/0

@richardlau

I was able to perform the release, thanks again @richardlau!

I'll keep this issue open so we can decide how to handle the nginx logs.

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

Ping to keep this alive

An out of space situation occurred again this morning. We have about 50% of the space in use by the nightly builds so we should look at adding automation to clear that up.

I put something simple together in 2023 but it was never actually put on a schedule so I'll see if I can use that to do another cleanup cycle.

There was about 1.7TB in use by /home/dist/nodejs/nightly

I've run /home/sxa/prune.sh nightly delete which has reduced that to 340GB. v8-canary is currently using 790G so that should likely be pruned too

We can have a discussion on scheduling this regularly and for safety perhaps having the permissions on the nightly, v8-canary and test directories modified so they don't have to be removed as root.

Navigation Menu

Repository navigation

Out of space on www server #3157

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Out of space on www server #3157

Description

Activity

richardlau commented on Jan 24, 2023

RafaelGSS commented on Jan 24, 2023

richardlau commented on Jan 24, 2023

RafaelGSS commented on Jan 24, 2023

richardlau commented on Jan 24, 2023

RafaelGSS commented on Jan 24, 2023

richardlau commented on Jan 24, 2023

richardlau commented on Jan 24, 2023

RafaelGSS commented on Jan 24, 2023

richardlau commented on Jan 24, 2023

github-actions commented on Dec 15, 2023

sxa commented on Dec 15, 2023

sxa commented on Aug 20, 2026

sxa commented on Aug 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions