Repository navigation
v8.writeHeapSnapshot doesn't release its resources correctly #33790
Description
- Version: v12.18.0
- Platform: Linux desktop 5.6.15-300.fc32.x86_64 deps: update openssl to 1.0.1j #1 SMP Fri May 29 14:23:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
- Subsystem: v8
What steps will reproduce the bug?
'use strict';
const v8 = require('v8');
const {PerformanceObserver} = require('perf_hooks');
process.on('SIGUSR2', () => {
console.log(`SIGUSR2 killed, Generating heapdump`);
console.log(process.memoryUsage());
console.log(v8.getHeapStatistics());
let fileName = v8.writeHeapSnapshot();
console.log(`Created heapdump file: ${fileName}`);
console.log(process.memoryUsage());
console.log(v8.getHeapStatistics());
});
// log GC times
let obs = new PerformanceObserver((items) => {
for (let entry of items.getEntries()) {
console.log(`GC kind: ${entry.kind}, duration: ${entry.duration}`);
}
});
obs.observe({entryTypes: ['gc'], buffered: true});
function main() {
// allocate 100MB+ of memory
let arr = [];
for (let i = 0; i < 3000000; i++) {
arr.push({num: Math.random()});
}
// dummy interval to keep arr from being collected
// also perform some dummy allocations to trigger GC
setInterval(() => {
let len = arr.length;
let collectMe = [];
for (let i = 0; i < 1000; i++) {
collectMe.push({num: Math.random()});
}
}, 20);
}
main();- run the code snippet
- send SIGUSR2 by callinng
kill -USR2 <pid>
How often does it reproduce? Is there a required condition?
Every time
What is the expected behavior?
Take heap snapshot and release the underlying resources after the operation is done.
What do you see instead?
Resident set size increases significantly (3GB+) with the above example and is never released
Additional information
- subsequent calls to v8.writeHeapSnapshot doesn't have significant effect on the RSS growth
- calling v8.writeHeapSnapshot on a project with more complex heap (80 MB Total Heap), resulted in significant minor GC slowdown going from average of 7ms to 30ms, effect on the major GC was very small (5-10%), above snippet won't demonstrate this slowdown
- the SIGUSR2 approach here is only used as an example of how you would manually take snapshot in prod (inspired by the "heapdump" package) + makes it easy to trigger subsequent heapdumps manually, the issue manifests the same way when the signaling is stripped away
Activity
bnoordhuis commentedon Jun 9, 2020
Thanks for the good bug report. This is a variation on #8871 (memory fragmentation caused by glibc's malloc/free) and not something that's really under Node's control, I'm afraid.
You can see it when you start node with
strace -fe brk node script.jsand send it a SIGUSR2: the "program break" - the end of the C/C++ heap - keeps going up and some longer-lived allocations block it from going back down again.If you switch to jemalloc, an alternative memory allocator that doesn't favor
brk(2)so much, the problem should go away. It does for me locally, at least.(The location and name of the jemalloc library may be different on your system.)
fx1738 commentedon May 21, 2023
Its been 3 years and this is still not fixed lol
(Node 16.14)
bnoordhuis commentedon May 21, 2023
Lack of reading comprehension certainly is hilarious.
I'll close this out.
znewsham commentedon Jun 23, 2023
I know this is closed out - but I've just come across (what I believe is) this issue and I'm wondering why I'd start seeing it now.
We just moved our applications from running on a combination of
node:14.19.1andnode:14.19.1-alpinetonode:18.16.0-bullseye.As far as I can tell, none of these use
jemallocby default (cat/proc/<PID>/smaps | grep jemreturns nothing), yet with the 18-bullseye variant we're seeing this large RSS increase over time, and with node14 we were not. It does seem that switching to jemalloc resolves, but I'm a little nervous of making this change without understanding why it would happen now.Any thoughts as to why it would happen now? Seems the issue existed before node 14 and still exists now