OSX tests are stuck

Hey folks,

I'm trying to finish the CI for nodejs/node#54560 (target is today) and osx test is taking too long (it seems stuck) >12h~

Alpine might be facing a similar problem. It's stuck on CITGM: https://ci.nodejs.org/job/citgm-smoker/3472/

Hey folks,

I'm trying to finish the CI for nodejs/node#54560 (target is today) and osx test is taking too long (it seems stuck) >12h~

It's not stuck -- https://ci.nodejs.org/job/node-test-commit-osx/60982/ hasn't started yet (it's been waiting for a osx11-x64 machine for 12 hours).

https://ci.nodejs.org/job/node-test-commit-osx/nodes=osx11-x64/ is incredibly backlogged -- looks like we only have one machine serving jobs.

Running the cleanup script on test-orka-macos11-x64-2.

test-orka-macos11-x64-2 is back online and processing jobs. Backlog is currently 16 queued jobs (+1 citgm job) for osx11-x64.

Getting java.io.IOException: No space left on device on test-orka-macos11-x64-1

e.g.: https://ci.nodejs.org/job/node-test-commit-osx/61013/nodes=osx11-x64/console

Ran the cleanup script on test-orka-macos11-x64-1.

Disk space is too low again on test-orka-macos11-x64-2. What's happening?

Now both nodes are offline. We can't keep cleaning them manually every day.

Have the macOS Node.js builds got considerably larger?

18 GB (#3878 (comment)) sounds really high -- builds on Linux are only ~3 GB.

According to my local folders:

 18G	canary/out/Release
 18G	node/out/Release
 17G	v20.x/out/Release
 18G	v22.x/out/Release

I don't have a v18.x build.

I've run the cleanup script on test-orka-macos11-x64-2 and rebooted the machine. This is now reporting ~21GB of free space -- a node-test-commit-osx run has started and it is currently consuming 1.8 GB of that (and would be expected to grow to 18 GB).

test-orka-macos11-x64-2:~ iojs$ du -hs build/workspace/node-test-commit-osx/
1.8G    build/workspace/node-test-commit-osx/
test-orka-macos11-x64-2:~ iojs$ df -h
Filesystem       Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk2s5s1   90Gi   14Gi   21Gi    41%  553788 941116212    0%   /
devfs           188Ki  188Ki    0Bi   100%     650         0  100%   /dev
/dev/disk2s4     90Gi  1.0Mi   21Gi     1%       1 941669999    0%   /System/Volumes/VM
/dev/disk2s2     90Gi  305Mi   21Gi     2%    1038 941668962    0%   /System/Volumes/Preboot
/dev/disk2s6     90Gi  592Ki   21Gi     1%      17 941669983    0%   /System/Volumes/Update
/dev/disk2s1     90Gi   54Gi   21Gi    73%  529196 941140804    0%   /System/Volumes/Data
map auto_home     0Bi    0Bi    0Bi   100%       0         0  100%   /System/Volumes/Data/home
test-orka-macos11-x64-2:~ iojs$

I suppose one other question -- where is the tmp dir on the macOS machines? /tmp/ looks surprisingly empty when we know that the tests are leaving behind node-coverage-* directories (#3864 -- I assume behaviour on macOS would be the same and these directories are being written somewhere).

echo $TMPDIR should print the location. I think the actual value is random and different on each macOS installation.

test_runner: improve code coverage cleanup

This was a symptom of having long lived OSX runners, which will soon be fixed by our transition to ephemeral Orka macos runners.

In the meantime, the disk was filling because macos spotlight indexing was indexing the builds , and creating new, unique uuid to filename mappings, which was filling up /private/var/db/uuidtext

I've disabled spotlight, and removed the spotlight databases.

We've now got 31GB free with a workspace on test-orka-macos11-x64-1

And 52 GB free on test-orka-macos11-x64-2

both of the orka-macos10.15-x64 machines have over 50GB available as well.

We should be able to re-enable these now, and they should last until they are replaced, shortly.

test_runner: improve code coverage cleanup

All of the legacy OSX machines have been running successfully now for about 36 hours. The build results are very flappy between green and yellow status, and almost every yellow status is one particular test that keeps flagging as flaky:

Im not sure if that test is specifically flaky on OSX or if thats a problematic test in general, but its the vast majority of flaky results for the current OSX builds.

Can we close it?

Can we close it?

But part of what was being discussed here is that nodejs is using too much disk space to build. This still affects others, even if it does not affect your CI anymore. What has been done to fix that? Or is it being tracked elsewhere?

Navigation Menu

Repository navigation

OSX tests are stuck #3887

13 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

OSX tests are stuck #3887

Description

Activity

RafaelGSS commented on Sep 2, 2024

richardlau commented on Sep 2, 2024

richardlau commented on Sep 2, 2024

richardlau commented on Sep 2, 2024

aduh95 commented on Sep 2, 2024

richardlau commented on Sep 3, 2024

targos commented on Sep 5, 2024

targos commented on Sep 5, 2024

richardlau commented on Sep 5, 2024

targos commented on Sep 5, 2024

richardlau commented on Sep 5, 2024

richardlau commented on Sep 5, 2024

targos commented on Sep 5, 2024

13 remaining items

ryanaslett commented on Oct 1, 2024

ryanaslett commented on Oct 3, 2024

RafaelGSS commented on Nov 26, 2024

ryandesign commented on Dec 25, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions