Repository navigation
Sending large amount of data to ServerResponse causes RSS and External memory leak #30902
Description
- Node.js Version: 10.17.0, 8.6.12
- OS: Ubuntu 16.04
- Scope (install, code, runtime, meta, other?): Runtime
- Module (and version) (if relevant): Stream. Most probably related to
http_outgoing.jsstream_writable.js
Hello!
When sending lots of data to response( for example downloading big files) using Buffer there is significant increase in RSS and External memory usage.
As you can see in my example I stop to send data when stream is paused and continue after resume call so it shouldn’t be buffered in response stream.
Even though resume/pause called properly RSS/external memory is growing. Heap seems more or less stable.
Here are results of the calls to process.memoryUsage after starting downloading.
{ rss: 29802496,
heapTotal: 7684096,
heapUsed: 5211920,
external: 8224 }
// Download triggered
{ rss: 33968128,
heapTotal: 10305536,
heapUsed: 5312576,
external: 26222624 }
{ rss: 38477824,
heapTotal: 10829824,
heapUsed: 5743944,
external: 56631328 }
{ rss: 60731392,
heapTotal: 8208384,
heapUsed: 5257744,
external: 92282912 }
Debugging using Chrome inspector made me suspect flow which is below.
-
When we try to write lots of data to response the response connection will be corked and uncorked on next tick here
https://github.com/nodejs/node/blob/v8.x/lib/_http_outgoing.js#L652 -
write_method will return false to initiate backpressure andpausewill be called:
https://github.com/nodejs/node/blob/v8.x/lib/internal/streams/legacy.js#L17 -
On the next tick connection will be uncorked and
clearBufferwill be called
https://github.com/nodejs/node/blob/v8.x/lib/_stream_writable.js#L312 -
It results in calling
doWritehere
https://github.com/nodejs/node/blob/v8.x/lib/_stream_writable.js#L500 -
Then
stream._writevwill be called and result of this call will be ignored
https://github.com/nodejs/node/blob/v8.x/lib/_stream_writable.js#L394 -
Then as a result
afterWritewill be called which causes emitting ofdrainevent and resuming stream even though we don't know if data on Step 5 was buffered or not.
https://github.com/nodejs/node/blob/v8.x/lib/_stream_writable.js#L461
https://github.com/nodejs/node/blob/v8.x/lib/_stream_writable.js#L473
My suspect is that memory leaking happens on Step 5 but unfortunately I am not sure as I can't get V8 memory profile to see what is leaking in RSS/External memory.
Below is simple Node JS code which emulates downloading of large file with chunk of 1MBs sent to stream using Buffer as data.
const http = require("http");
const PORT = 8080;
const Stream = require("stream");
const requestHandler = (req, res) => {
let mystream = new Stream();
let paused = false;
mystream.resume = () => {
paused = false;
};
mystream.pause = () => {
paused = true;
};
mystream.readable = true;
mystream.once("data", function(data) {
if (res.headersSent) return;
res.writeHead(200, {
"Content-Disposition": "attachment; filename*=utf-8''test",
"Content-Type": "application/zip"
});
res.write(data);
mystream.pipe(res);
});
setInterval(() => {
if (!paused) {
mystream.emit("data", new Buffer(1024 * 1024));
}
}, 10);
}
const server = http.createServer(requestHandler);
server.listen(PORT, (err) => {
if (err) {
console.log("Error occurred", err) ;
}
console.log(`Server is listening on ${PORT}`);
setInterval(() => {
console.log(process.memoryUsage());
}, 5000);
})
Activity
andrewnester commentedon Dec 11, 2019
Seems very similar to this issue even though it’s closed
#6078
andrewnester commentedon Dec 23, 2019
I did a bit more investigation and apparently the issue is caused by allocating
Buffer.In this particular case buffers are allocated a lot, V8 allocates them on external memory, frees the memory after Buffer is not used anymore but GC does not de-allocate the issue. So external memory is actually free just not de-allocated.
As a workaround to force memory de-allocation we can call
gc()directlypuzpuzpuz commentedon Jun 22, 2020
@andrewnester there is a chance that this issue may be related with glibc memory fragmentation? See #8871 and #33790
andrewnester commentedon Jun 22, 2020
@puzpuzpuz #33790 doesn't look like relevant but #8871 might be but I don't know how to prove if it's relevant for us or not
puzpuzpuz commentedon Jun 23, 2020
@andrewnester both of these issues are related with glibc memory fragmentation. To diagnose your issue you could try to switch to jemalloc (see #8871 (comment) for more details) and if that fixes your issue, it's most likely has something to do with memory fragmentation.
gireeshpunathil commentedon Nov 7, 2020
pls evaluate the discussion in #31641 too - in that case, there is no
realleak, instead incorrect accounting of memory against the owning process. (I am not claiming this one IS that, just a speculation)rudolf commentedon Nov 17, 2021
@andrewnester Using your script I can't reproduce this on v14.18.1 or v16.13.0. On both OSX and Ubuntu the external memory stays stable.
bnoordhuis commentedon May 21, 2023
OP didn't follow up, @rudolf wasn't able to reproduce, and there's been no movement since late 2021 so I'll go ahead and close this out.