No space left on macOS host?

See https://ci.nodejs.org/job/node-test-commit-osx/44401/nodes=osx1015/#showFailuresLink

🤷
There have been three node-test-commit-osx builds on the same machine, https://ci.nodejs.org/computer/test-nearform-macos10.15-x64-3/builds, since then which all passed:

And

test-nearform-macos10:~ iojs$ df -h /Users/iojs/build/workspace/
Filesystem     Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s1   51Gi   25Gi   15Gi    63%  309782 536799578    0%   /System/Volumes/Data
test-nearform-macos10:~ iojs$

hmm. https://ci.nodejs.org/job/node-test-commit-osx/44429/nodes=osx1015/console also ran out of space on the same machine, test-nearform-macos10.15-x64-3.

13:54:44 fatal error: /Library/Developer/CommandLineTools/usr/bin/libtool: can't write to output file (No space left on device)
13:55:20 make[2]: *** [/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1015/out/Release/libv8_initializers.a] Error 1

Again we have around 13G of free space.

test-nearform-macos10:~ iojs$ df -h
Filesystem      Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s5    51Gi   10Gi   13Gi    44%  487478 536621882    0%   /
devfs          186Ki  186Ki    0Bi   100%     642         0  100%   /dev
/dev/disk1s1    51Gi   26Gi   13Gi    67%  357543 536751817    0%   /System/Volumes/Data
/dev/disk1s4    51Gi  1.0Gi   13Gi     8%       2 537109358    0%   /private/var/vm
map auto_home    0Bi    0Bi    0Bi   100%       0         0  100%   /System/Volumes/Data/home
test-nearform-macos10:~ iojs$ df -h /Users/iojs/
Filesystem     Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s1   51Gi   26Gi   13Gi    67%  357543 536751817    0%   /System/Volumes/Data
test-nearform-macos10:~ iojs$ df -h /Users/iojs/build/workspace/
Filesystem     Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s1   51Gi   26Gi   13Gi    67%  357543 536751817    0%   /System/Volumes/Data
test-nearform-macos10:~ iojs$ df -h /tmp
Filesystem     Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s1   51Gi   26Gi   13Gi    67%  357543 536751817    0%   /System/Volumes/Data
test-nearform-macos10:~ iojs$

The test-commit job does a git clean at the end. So somehow the job is chewing through 13G while running? (!)

@AshCripps

@AshCripps do you know if it's normal for /dev and /System/Volumes/Data/home to be 100% used on macs?

im going to say no, /System/Volumes/Data is the map of the user directory IIRC so its likely is a full disk. Catalina changed this reporting into a whole weird mess.

So far, all three CI runs on nodejs/node#42628 failed with ENOSPC on osx.

I'm just more and more confused:

FYI the builds really are using about 14GB while in flight:
During build (while running tests):

/dev/disk1s1    53710936 39596968   1474908    97%  284892 536824468    0%   /System/Volumes/Data

After build (Workspace cleared up):

Filesystem    1024-blocks     Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk1s5     53710936 10873180  14855648    43%  487478 536621882    0%   /

Some of the files in out/Release are pretty big - these two account for 5GB alone;

3004068	libv8_base_without_compiler.a
2001752	libv8_initializers.a

and (probably obviously from the above) obj.target is quite large at over 6Gb

So we need to either get these expanded or find something else to delete. The disks are showing as around 51Gb at the moment. /var/db is quite large at about 8.7Gb but I don't know whether anything can be done about that. ccache for the iojs user is using 4.5Gb

Note - the -1 machine has a slightly larger disk than -2 and -3:

[sxa@sainz .ssh]$ for A in 1 2 3; do ssh test-nearform-macos10.15-x64-$A df -k .; done
Filesystem   1024-blocks     Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk1s1    68359372 26111720  29608124    47%  293179 683300541    0%   /System/Volumes/Data
Filesystem   1024-blocks     Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk1s1    53710936 24257204  16802816    60%  296611 536812749    0%   /System/Volumes/Data
Filesystem   1024-blocks     Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk1s1    53710936 39309828   1762048    96%  284926 536824434    0%   /System/Volumes/Data
[sxa@sainz .ssh]$

Note these machines cant really be any bigger as the host they are on started hitting issues when the VMs were a bigger size. (This was the roughly the largest size that was stable)

issues when the VMs were a bigger size.

Is that disk size specifically? Any idea what the issues were? I could understand if RAM/CPU allocation was overdone it would lead to issues but disk seems a little odd.

I think it was due to apples autoupdate feature meaning the host needed enough headroom other it crashes - im sure there was an issue on this in the build repo.

Jenkins has automatically taken test-nearform-macos10.15-x64-3 offline due to low disk space.

We're now also seeing disk space issues on test-orka-macos11-x64-2,
e.g. https://ci.nodejs.org/job/node-test-commit-osx/44542/nodes=osx11-x64/console

10:55:10 not ok 3236 pummel/test-fs-readfile-tostring-fail
10:55:10   ---
10:55:10   duration_ms: 0.203
10:55:10   severity: fail
10:55:10   exitcode: 7
10:55:10   stack: |-
10:55:10     /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/test/pummel/test-fs-readfile-tostring-fail.js:69
10:55:10       throw err;
10:55:10       ^
10:55:10     
10:55:10     [Error: ENOSPC: no space left on device, write] {
10:55:10       errno: -28,
10:55:10       code: 'ENOSPC',
10:55:10       syscall: 'write'
10:55:10     }
10:55:10     
10:55:10     Node.js v18.0.0-pre
10:55:10   ...

For test-orka-macos11-x64-2 I've removed /Users/iojs/build/workspace/node-test-commit-osx-as-shared-mdawson which was 1.5GB.

https://ci.nodejs.org/job/node-test-commit-osx/48075/nodes=osx11-x64/testReport/junit/(root)/test/pummel_test_fs_readfile_tostring_fail/

https://ci.nodejs.org/job/node-test-commit-osx/48110/nodes=osx1015/testReport/junit/(root)/test/pummel_test_fs_largefile/

I don't know if it's only because of this issue, but at the moment, macOS CI is almost permanently red.

Ive just logged into orka-macos11-x64-2 which is the latest failure and there is 15gb of free space available - surely thats enough to build node is it not? or has the binary really ballooned this past year?

Ive just logged into orka-macos11-x64-2 which is the latest failure and there is 15gb of free space available - surely thats enough to build node is it not? or has the binary really ballooned this past year?

Maybe someone with a mac can verify?

The test CI runs make clean at the end of the job so it's hard to catch peak disk usage, but the release jobs do not and the workspaces for the release machines are in that ballpark, with the arm64 almost double presumably because it builds both x64 and arm64:

$ ssh release-orka-macos10.15-x64-1 du -hs /Users/iojs/build/ws/
 17G    /Users/iojs/build/ws/
$ ssh release-nearform-macos10.15-x64-1 du -hs /Users/iojs/build/ws/
 17G    /Users/iojs/build/ws/
$ ssh release-nearform-macos11.0-arm64-1 du -hs /Users/iojs/build/ws/
 31G    /Users/iojs/build/ws/

and the release job doesn't run tests either so that doesn't account for extra space required by the tests such as test-fs-readfile-tostring-fail.

My node folder takes 16G (arm64 mac)

ah ok well that is a problem then. Also once again the private folder on the machine I rebooted a couple weeks back is over 7.6GB again so this needs some consideration. It might be worth putting in a weekly reboot into these machines (for reference my mac private folder is ~60% of that size and its turned off every day) but thats also only a stop gap.

I think we need to have a think about what to do with these machines and what solutions we can put in place.

test-orka-macos11-x64-2 had a file /System/Volumes/Data/System/Library/Caches/com.apple.coresymbolicationd/data which was 39GB!! have removed, rebooted and brought back on line.

I got the same error on today's build: #3071

@AshCripps

I was doing some research about this topic. Here are some useful logs 😄

Note: I could't test all the osx machines (see: #3084, #3083 and #3034)

Machine Logs

test-nearform-macos10.15-x64-1