/tmp filling up on CI hosts

https://ci.nodejs.org/job/node-test-commit-smartos/buildTimeTrend

It's slowing down core development.

/tmp on test-equinix_mnx-smartos20-x64-3 is full:

$ ansible -m command -a "df -h" test-equinix_mnx-smartos20-x64-?
test-equinix_mnx-smartos20-x64-3 | CHANGED | rc=0 >>
Filesystem                                  Size  Used Avail Use% Mounted on
zones/356655a2-12e6-e1d7-ac7b-b5188ad37cb0  100G   30G   71G  30% /
/.zonecontrol                                56T   57G   56T   1% /.zonecontrol
/lib                                        314M  292M   23M  93% /lib
/lib/svc/manifest                            56T  1.5M   56T   1% /lib/svc/manifest
/usr                                        433M  394M   40M  91% /usr
swap                                         16G  4.6G   12G  29% /etc/svc/volatile
swap                                        4.0G  4.0G     0 100% /tmp
swap                                         16G  4.6G   12G  29% /var/run
test-equinix_mnx-smartos20-x64-4 | CHANGED | rc=0 >>
Filesystem                                  Size  Used Avail Use% Mounted on
zones/c6e3d47a-1421-ee11-c52d-c3c80c198e95  100G   25G   76G  25% /
/.zonecontrol                                56T   57G   56T   1% /.zonecontrol
/lib                                        314M  292M   23M  93% /lib
/lib/svc/manifest                            56T  1.5M   56T   1% /lib/svc/manifest
/usr                                        433M  394M   40M  91% /usr
swap                                         16G  2.3G   14G  15% /etc/svc/volatile
swap                                        4.0G  1.8G  2.3G  45% /tmp
swap                                         16G  2.3G   14G  15% /var/run

It's full of node-coverage-* directories. I've removed them:

[root@test-equinix_mnx-smartos20-x64-3 ~]# rm -rf /tmp/node-coverage-*
[root@test-equinix_mnx-smartos20-x64-3 ~]# df -h
Filesystem                                  Size  Used Avail Use% Mounted on
zones/356655a2-12e6-e1d7-ac7b-b5188ad37cb0  100G   30G   71G  30% /
/.zonecontrol                                56T   57G   56T   1% /.zonecontrol
/lib                                        314M  292M   23M  93% /lib
/lib/svc/manifest                            56T  1.5M   56T   1% /lib/svc/manifest
/usr                                        433M  394M   40M  91% /usr
swap                                         16G  955M   16G   6% /etc/svc/volatile
swap                                        4.0G   44K  4.0G   1% /tmp
swap                                         16G  955M   16G   6% /var/run
[root@test-equinix_mnx-smartos20-x64-3 ~]#

I've cleared out /tmp/node-coverage-* on test-equinix_mnx-smartos20-x64-4 as well:

[root@test-equinix_mnx-smartos20-x64-4 ~]# df -h
Filesystem                                  Size  Used Avail Use% Mounted on
zones/c6e3d47a-1421-ee11-c52d-c3c80c198e95  100G   25G   76G  25% /
/.zonecontrol                                56T   57G   56T   1% /.zonecontrol
/lib                                        314M  292M   23M  93% /lib
/lib/svc/manifest                            56T  1.5M   56T   1% /lib/svc/manifest
/usr                                        433M  394M   40M  91% /usr
swap                                         16G  2.3G   14G  15% /etc/svc/volatile
swap                                        4.0G  1.8G  2.3G  45% /tmp
swap                                         16G  2.3G   14G  15% /var/run
[root@test-equinix_mnx-smartos20-x64-4 ~]# rm -rf /tmp/node-coverage-*
[root@test-equinix_mnx-smartos20-x64-4 ~]# df -h
Filesystem                                  Size  Used Avail Use% Mounted on
zones/c6e3d47a-1421-ee11-c52d-c3c80c198e95  100G   25G   76G  25% /
/.zonecontrol                                56T   57G   56T   1% /.zonecontrol
/lib                                        314M  292M   23M  93% /lib
/lib/svc/manifest                            56T  1.5M   56T   1% /lib/svc/manifest
/usr                                        433M  394M   40M  91% /usr
swap                                         16G  467M   16G   3% /etc/svc/volatile
swap                                        4.0G   44K  4.0G   1% /tmp
swap                                         16G  467M   16G   3% /var/run
[root@test-equinix_mnx-smartos20-x64-4 ~]#

Good catch!
So there's a bug in our tests. They shouldn't be using the system's tmpdir.

Good catch! So there's a bug in our tests. They shouldn't be using the system's tmpdir.

FWIW I ran make -j88 test-ci on a dev machine I have access to and it created these directories/files in /tmp:

drwx------.  2 rlau   rlau       94 Aug 14 13:48 node-coverage-09AcaL
drwx------.  2 rlau   rlau      138 Aug 14 13:48 node-coverage-6qYENJ
drwx------.  2 rlau   rlau       94 Aug 14 13:48 node-coverage-afMxbz
drwx------.  2 rlau   rlau       94 Aug 14 13:48 node-coverage-e0jO5f
drwx------.  2 rlau   rlau      138 Aug 14 13:48 node-coverage-F6ZUIw
drwx------.  2 rlau   rlau       50 Aug 14 13:48 node-coverage-hHzLhL
drwx------.  2 rlau   rlau       94 Aug 14 13:48 node-coverage-Lc7I1n
drwx------.  2 rlau   rlau       94 Aug 14 13:48 node-coverage-otzELS
drwx------.  2 rlau   rlau       94 Aug 14 13:48 node-coverage-p2PeI0
drwx------.  2 rlau   rlau       94 Aug 14 13:48 node-coverage-q6CVl4
drwx------.  2 rlau   rlau       50 Aug 14 13:48 node-coverage-RdCe2w
drwx------.  2 rlau   rlau       94 Aug 14 13:48 node-coverage-SnPBDM
drwx------.  2 rlau   rlau       94 Aug 14 13:48 node-coverage-tw4y03
drwx------.  2 rlau   rlau       94 Aug 14 13:48 node-coverage-x0GveC
drwx------.  2 rlau   rlau       50 Aug 14 13:48 node-coverage-zBbuSF
-rw-r--r--.  1 rlau   rlau   138954 Aug 14 13:47 perf-426641.map
-rw-r--r--.  1 rlau   rlau   112792 Aug 14 13:47 perf-426677.map

I think some of those are from test/parallel/test-runner-coverage.js which has specific tests checking coverage is produced when the NODE_V8_COVERAGE environment variable is not set.

I think some of those are from test/parallel/test-runner-coverage.js which has specific tests checking coverage is produced when the NODE_V8_COVERAGE environment variable is not set.

If I exclude test/parallel/test-runner-coverage.js from parallel there still at least one more test writing out coverage:

$ rm -rf /tmp/node-coverage-* /tmp/perf-*.map
$ tools/test.py -J --skip=parallel/test-runner-coverage parallel
[00:28|% 100|+ 3508|-   0]: Done

All tests passed.
$ ls -al /tmp
...
drwx------.  2 rlau   rlau       51 Aug 14 17:20 node-coverage-8Oj2sa
drwx------.  2 rlau   rlau       96 Aug 14 17:20 node-coverage-Xwd98L
-rw-r--r--.  1 rlau   rlau   138954 Aug 14 17:20 perf-1822721.map
-rw-r--r--.  1 rlau   rlau   112792 Aug 14 17:20 perf-1822738.map
...
$

Can you try to change the permissions so only root is allowed to write to /tmp. Hopefully we can identify the tests by observing failures.

Not on that machine, but should be able to in a container.

$ mkdir /tmp/cantwrite
$ chmod u-w /tmp/cantwrite/
$ ls -al /tmp/cantwrite
total 4
dr-xr-xr-x.  2 rlau   rlau      6 Aug 14 19:15 .
drwxrwxrwt. 12 nobody nobody 4096 Aug 14 19:15 ..
$ TMPDIR=/tmp/cantwrite/ tools/test.py
...
[03:29|% 100|+ 4014|-   2]: Done

Failed tests:
out/Release/node /home/rlau/sandbox/github/node/test/parallel/test-runner-output.mjs
out/Release/node /home/rlau/sandbox/github/node/test/parallel/test-runner-coverage.js
$

With an actual unwritable /tmp:

[03:26|% 100|+ 4012|-   4]: Done

Failed tests:
out/Release/node /home/nodejs/node/test/parallel/test-cli-node-options.js
out/Release/node /home/nodejs/node/test/parallel/test-runner-coverage.js
out/Release/node /home/nodejs/node/test/parallel/test-runner-output.mjs
out/Release/node /home/nodejs/node/test/parallel/test-trace-events-http.js
$

I opened test: prevent V8 from writing into the system's tmpdir node#54395 for test-cli-node-options.js
I don't know how to handle test-runner-coverage.js and test-runner-output.mjs
I can't find why test-trace-events-http.js is affected by reading the code.

For test-runner-coverage.js - When NODE_V8_COVERAGE is set the report will be written into the dest directory, currently for test-runner-coverage.js is set to tmpdir.path maybe what we need to do is to clear the report directory after the test is finished to prevent the report fill up the disk space, will take another when I wake up.

For test-runner-coverage.js - When NODE_V8_COVERAGE is set the report will be written into the dest directory, currently for test-runner-coverage.js is set to tmpdir.path maybe what we need to do is to clear the report directory after the test is finished to prevent the report fill up the disk space, will take another when I wake up.

I think those cases are working as expected -- but that test has variants that deliberately do not set NODE_V8_COVERAGE and those are causing writes to the system tmp dir which are never cleaned up.

test_runner: improve code coverage cleanup

/tmp on test-equinix_mnx-smartos20-x64-3 was full again. I have deleted all of those node-coverage-* directories.

test_runner: improve code coverage cleanup

Is this resolved?

I just ran make test-ci with main and that still appears to be creating node-coverage-* directories under /tmp.

Navigation Menu

Repository navigation

/tmp filling up on CI hosts #3864

15 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

/tmp filling up on CI hosts #3864

Description

Activity

richardlau commented on Aug 14, 2024

richardlau commented on Aug 14, 2024

targos commented on Aug 14, 2024

richardlau commented on Aug 14, 2024

richardlau commented on Aug 14, 2024

richardlau commented on Aug 14, 2024

targos commented on Aug 14, 2024

richardlau commented on Aug 14, 2024

richardlau commented on Aug 14, 2024

richardlau commented on Aug 14, 2024

targos commented on Aug 15, 2024

jakecastelli commented on Aug 15, 2024

richardlau commented on Aug 15, 2024

15 remaining items

richardlau commented on Sep 11, 2024

targos commented on Dec 20, 2024

richardlau commented on Dec 20, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions