vm Script memory leak in Node.js 14 / 16

Version

14 / 16

Platform

x64 macOS / Linux

Subsystem

vm

What steps will reproduce the bug?

'use strict';

let times = 0;

function run() {
  let str = 'var a = "';
  for (let i = 0; i < 100 * 1024 / 18; i++) str += Math.random().toString();
  str += '";';

  const script = new (require('vm').Script)(str);
  times++;
  if (times % 1000 === 0) console.log(times);
}

(async () => {
  while (true) {
    run();

    await new Promise(resolve => {
      setTimeout(() => {
        resolve();
      }, 10);
    });

    gc();
  }
})();

Run with:

$ node --expose-gc --max-heap-size=100 test.js

It will occur heap OOM under Node.js 14 and 16, and Node.js 12 is OK.

<--- Last few GCs --->                                                         
                                                                                                                                                               
[1430950:0x55695b1dc4b0]     6372 ms: Mark-sweep 72.0 (91.6) -> 71.6 (91.8) MB, 4.3 / 0.0 ms  (average mu = 0.728, current mu = 0.778) testing GC in old space requested
[1430950:0x55695b1dc4b0]     6388 ms: Mark-sweep 72.2 (91.8) -> 71.8 (92.1) MB, 3.1 / 0.0 ms  (average mu = 0.760, current mu = 0.804) testing GC in old space requested
[1430950:0x55695b1dc4b0]     6403 ms: Mark-sweep 72.4 (92.1) -> 72.0 (92.3) MB, 2.9 / 0.0 ms  (average mu = 0.781, current mu = 0.806) testing GC in old space requested
                                                                                                                                                               
                                                                               
<--- JS stacktrace --->                                                                                                                                        
                                                                               
FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory

How often does it reproduce? Is there a required condition?

Every time.

What is the expected behavior?

Not OOM.

What do you see instead?

OOM.

Additional information

No response

And another amazing thing:

run node --expose-gc --max-heap-size=100 --inspect=9229 test.js for a while, OOM.
run node --expose-gc --max-heap-size=100 --inspect=9229 test.js and then run node inspect localhost:9229, everything will be OK and there's no OOM any more.

I added

if (times === 330) {
  require('v8').writeHeapSnapshot();
}

To the code, and found there're full of the str object like:

The leak seems in CompilationCache.

That means when I run command below in Node.js 14 / 16:

$ node --expose-gc --max-heap-size=100 --no-compilation-cache test2.js

The behavior is right - no longer OOM.

/cc @nodejs/v8

In Node.js 12, the heap size will up to about 100M either. But after it cannot allocate more space, it will trigger kLastResort GC because:

https://github.com/nodejs/node/blob/v12.22.6/deps/v8/src/heap/heap.cc#L4904-L4925

After kLastResort GC, the heap size will down to very small.

And Node.js 14 / 16 will never trigger kLastResort GC:

https://github.com/nodejs/node/blob/v14.16.0/deps/v8/src/heap/heap.cc#L5205-L5226

And heap size still increases until OOM.

https://github.com/nodejs/node/blob/master/deps/v8/src/heap/mark-compact.cc#L1458-L1469

EvacuateNewSpaceVisitor::Visit() in GC, this new code (After Node.js 14) requires allocating an old space block and returns an AllocateResult.IsRetry() object. heap_->FatalProcessOutOfMemory() here at last.

So we should think what should we do when we're under GC but we have no enough space to allocate old space block.

Old V8 (Node.js 12) just no this if logic, and allocate old space anyway. And it will do kLastResort GC later.

What's more, it seems that I can't disable always promote young mc by --no-always-promote-young-mc because of DEFINE_IMPLICATION(array_buffer_extension, always_promote_young_mc) in v8's src/flags/flag-defination.h.

Refs

I think we have 2 ways to fix this bug:

Fix _always_promote_young's logic in GC - make allocate old space not throw error when heap size up to exceeded.
Set array_buffer_extension in gyp back to 0, and set --always-promote-young-mc defaults to false.

btw. why UnboundScript's script cache and its code string won't GCed until kLastResort (aka. heap size up to exceeded)? Can this be fixed?

@targos

There're several problems (bugs) occur this one issue:

All Script's CompilationCache in the heap and won't be GCed until Last Resort GC; (It will make heap size to the limit and --no-compilation-cache can resolve it)
--always-promote-young-mc will OOM when old space cannot allocate more memory (e.g. when heap size limit exceeded); (I think it should do a Last Resort GC when we meet this situation like Node.js 12's V8)
We cannot close --always-promote-young-mc because --array-buffer-extension always true at compile time.

/cc @targos
/ping @nodejs/v8

https://bugs.chromium.org/p/v8/issues/detail?id=12198

This answer helped me: StackOverflow

@XadillaX

@XadillaX given you did such detailed analysis do you have any idea what is the path forward now v8 closed the bug you raised as won’t fix ? With some suggestions for the node team ?

@XadillaX

@XadillaX given you did such detailed analysis do you have any idea what is the path forward now v8 closed the bug you raised as won’t fix ? With some suggestions for the node team ?

Hmmm... I'm not sure since V8 closed that issue. I think we can't do anything without v8's change.

Oh yes - sorry for the bad link, you are right. I'll also update my comment

@XadillaX

@XadillaX @nodejs/vm @nodejs/v8 are there any updates on this issue or anything that changed in the last year?

Also more context jestjs/jest#12205 which implies maybe @nodejs/loaders is also the right ping since adding importModuleDynamically causes the leak

The problem will be gone in the example of the OP if we simply remove the gc() call, as explained in https://bugs.chromium.org/p/v8/issues/detail?id=12198#c4. So I'd believe this issue can be closed.

@legendecas

@legendecas my understanding is that the "explicit gc calling" issue is in addition and not instead of Jest's issue which they are experiencing without any gc() calls.

Namely jestjs/jest#12205 changes code with 0 gc calls to 0 gc calls and resolves the issue except if they also pass importModuleDynamically which causes the leak again.

@benjamingr

@benjamingr I would take it as a different problem so I'd suggest opening a dedicated issue to track it.

@benjamingr

@benjamingr I've opened #44211 to track the issues with vm.compileFunction/importModuleDynamically.

Update dependency jest to v29 (#241)

Hi, We have been unable to upgrade our node beyond 16.10.0 and have been stuck since higher version regresses the performance of our jest tests. Is there any solution coming for this from node?

Same here...

You might try using https://www.npmjs.com/package/jest-light-runner to use Jest without vm.Script.

FYI there is a rewrite of the vm.Script / vm.*Module memory management that might fix many issues in #48510 (the OP referenced the bugs with small reproducible cases that can be confirmed to be fixed. Can't say it definitely fixes other bugs without reproducible examples, but it should fix a bunch more issues in the issue tracker that I didn't dig out). It is pending on a V8 patch which is probably going to be relanded soon. (The fix however relies on ABI-breaking changes in V8 so it may or may not get released in v20.x)

Update dependency jest to v29 (#108)

I think this can be closed now - for normal uses of vm.Script, #48510 should have fixed the leaks. For "leaks" caused by the use of --expose-gc and gc() (which disables code aging), it was marked as wontfix by https://bugs.chromium.org/p/v8/issues/detail?id=12198, which I think is fair because gc() is not a publicly supported feature, and it's more of a bug on the repro's side to use it as a way to "clean up memory thoroughly" (which can backfire).

Navigation Menu

Repository navigation

`vm` `Script` memory leak in Node.js 14 / 16 #40014

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior?

What do you see instead?

Additional information

Refs

5 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

vm Script memory leak in Node.js 14 / 16 #40014

Description

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior?

What do you see instead?

Additional information

Activity

XadillaX commented on Sep 6, 2021

XadillaX commented on Sep 6, 2021

XadillaX commented on Sep 7, 2021

targos commented on Sep 7, 2021

XadillaX commented on Sep 7, 2021

XadillaX commented on Sep 7, 2021

Refs

XadillaX commented on Sep 7, 2021

XadillaX commented on Sep 8, 2021

XadillaX commented on Sep 8, 2021

sidverma32 commented on Oct 29, 2021

lukeapage commented on Dec 15, 2021

XadillaX commented on Dec 16, 2021

5 remaining items

mbyrne00 commented on Jun 28, 2022

benjamingr commented on Jul 28, 2022

benjamingr commented on Jul 28, 2022

legendecas commented on Jul 28, 2022

benjamingr commented on Jul 28, 2022

legendecas commented on Jul 28, 2022

legendecas commented on Aug 15, 2022

sunilsurana commented on Sep 1, 2023

RaphaelMechali commented on Sep 5, 2023

nicolo-ribaudo commented on Sep 5, 2023

joyeecheung commented on Sep 5, 2023

joyeecheung commented on Oct 27, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`vm` `Script` memory leak in Node.js 14 / 16 #40014