Repository navigation
Unusually large (but constant) memory usage with transparent huge pages enabled #11077
Description
I'm working with a client that runs a simple Node process to consume messages off of a queue (NSQ) and send them to a downstream API (over TLS). We are using superagent as a HTTP client. Some of the processes we are running were using over 1.8GB of memory.
After a long investigation we discovered that the memory bloat was because transparent huge pages were enabled. Disabling transparent huge pages immediately took the memory usage down to about 150MB per process. This is most dramatically visualized here:
We're unsure exactly what mechanism was responsible for allocating so much memory, but took a core dump and noticed about 160,000 copies of the certificate list in memory - specifically an engineer ran strings | grep -i verisign and found that string 160,000 times. We suspect it's related to TLS negotiation or zlib or both.
We were running Node v6.9.2. The symptoms were extremely high (but not growing) memory usage that is off the Node heap - we ran node-heapdump on the process and we were only able to account for about 18MB of 1.8GB. Checking /proc/<pid>/smaps revealed that most of the memory was in AnonHugePages.
I'm mostly posting because there appear to be no Node-specific resources that point to transparent huge pages as a potential problem for memory management with Node. A Google search for "node transparent huge pages" yields nothing suggesting it might be a problem for Node users.
Several databases warn about the consequences of enabling transparent huge pages:
- TokuMX https://www.percona.com/blog/2014/07/23/why-tokudb-hates-transparent-hugepages/
- Redis https://www.digitalocean.com/company/blog/transparent-huge-pages-and-alternative-memory-allocators/
- Oracle https://blogs.oracle.com/linux/entry/performance_issues_with_transparent_huge
- MongoDB https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/
Activity
[-]Unusually large (but constant) memory usage with transparent huge pages turned on[/-][+]Unusually large (but constant) memory usage with transparent huge pages enabled[/+]kevinburke commentedon Jan 30, 2017
I found this blog post from 2011 about Node copying SSL certificates around:
Any chance that hack is still in place?
mscdex commentedon Jan 30, 2017
/cc @nodejs/crypto ^
indutny commentedon Jan 30, 2017
Hm... given that paging scheme affects it, I suspect that it is V8 that causes it. V8
mmap()s memory pages at randomized addresses to facilitate allocation of JavaScript objects. Considering that they may not be next to each other, and the kernel may be allocating more memory than actually needed.Is connection to TLS usage an already tested hypothesis? FWIW I think it may be not related to TLS at all.
cc @bnoordhuis I wonder what are your thoughts on this?
bnoordhuis commentedon Jan 31, 2017
V8 seems like the most likely culprit, yes. Since it maps memory in 4 kB chunks at randomized addresses, when huge pages are enabled, 98% or 99% of that memory is going to be wasted. Some sleuthing with perf(1) should be able to confirm that.
Problem is, I don't really know of a workaround except recommend that users disable THP. There isn't really a way to opt out at the application level. Perhaps V8 could be taught to use 2 MB pages but that's arguably less secure.
@ofrobots Have Chromium or V8 received bug reports about this before? Google didn't turn up anything except this issue. :-)
ofrobots commentedon Jan 31, 2017
My search doesn't find anything for Chromium/V8 either, which perhaps isn't surprising as people using browsers on the desktop aren't likely to have transparent huge pages turned enabled. V8 does randomize anonymous memory that it acquires from mmap.
AFAIK The V8 heap spaces are allocated as contiguous 'heap pages' which happen to be large enough (512KiB or 1MiB depending on version) that you wouldn't see this large a difference in memory usage. Pages with jitted executable code however might be a different matter?
Adding some V8 memory folks: @hannespayer, @mlippautz, @ulan.
I am not sure if there is a real workaround here other than to disable transparent huge pages, or switch them to madvise mode. Using large pages for code memory would indeed be less secure on the browser, but perhaps less so for the server side use cases?
mlippautz commentedon Jan 31, 2017
V8 GC/memory engineer here.
TL;DR: What @indutny and @ofrobots already said. I can see system-wide THP causing problems with fragmentation since V8 doesn't integrate well with them.
Details:
As already mentioned, the most recent V8 version allocates pages for regular objects in 512KiB pages (previously we used to have 1MiB pages). We allocate using regular mmap, making full use of the 64bit address space on 64bit. Since THP are afaik mapped eagerly by the kernel, you will definitely see a lot of fragmentation. Due to the large size of the 64bit address space I expect this to grow linearly with the size of huge pages, i.e., 4x for 2MiB huge pages.
Code space is special as we already need to put code pages into ranges that are close together because of certain calling schemes, which should actually limit fragmentation. So code pages are only really randomized on a global level, and don't use the full 64bit addressing scheme for individual code pages.
kevinburke commentedon Jan 31, 2017
Can we add additional documentation, or possibly a warning when Node starts? Is there currently Node documentation on debugging memory leaks or high memory usage?
rbranson commentedon Jan 31, 2017
(I'm working with @kevinburke on this) We observed the hugepages issue entirely in the [heap] mapping (as in
/proc/<pid>/smaps), which AFAICT isn't allocated by mmap, but is controlled by brk/sbrk syscalls. The V8 managed regions were (relatively) small.ofrobots commentedon Jan 31, 2017
Based on #11077 (comment) and what @mlippautz stated about code memory, I find it unlikely that V8 can cause this much fragmentation.
@rbranson: Perhaps you can run the application with
straceand try to correlate the memory ranges that show up w/ high fragmentation in the memory map?stracehas a-koption to print stacks on each system call. Beware that this might produce a lot of output.bnoordhuis commentedon Feb 1, 2017
perf record -galong with some tracepoints on brk/mmap/mmap2/etc. syscalls will probably be easier to work with.Intuitively, that doesn't sound right. brk allocates a contiguous range of virtual memory. Barring kernel bugs I wouldn't expect that to suffer much from fragmentation or wastage.
bnoordhuis commentedon Feb 7, 2017
@kevinburke @rbranson Any updates?
springmeyer commentedon Feb 13, 2017
@kevinburke I see you linked to https://www.digitalocean.com/company/blog/transparent-huge-pages-and-alternative-memory-allocators/. Are you using an alternative allocator in this case? If you are using jemalloc, consider upgrading - refs jemalloc/jemalloc#243
rbranson commentedon Feb 22, 2017
@bnoordhuis I hear you about the fragmentation being unintuitive. The data segment is contiguous in virtual memory, where-as hugepages are contiguous physical memory pages. The issues happen when the program break is being continuously adjusted up and down. For instance, use brk to add 10MB to the heap: THP might cause this to be 4 hugepages. If the program is able to free up the last 1MB of this allocation, it might decide to issue a brk to allow the OS to reclaim that 1MB of memory. In this case only 50% of the hugepage is actually usable by the system at all. When the program extends the data segment again using brk, there's nothing that will force it to re-use the already-allocated hugepage at the tail end of the data segment (See the very bottom section of https://www.kernel.org/doc/Documentation/vm/transhuge.txt). THP has defrag functionality to theoretically address these problems.
There is additional memory usage that we're trying to address outside of the THP fix. THP just exacerbates an already problematic memory consumption problem by severely hindering the kernel's ability to quickly reclaim unused memory. Right now our most promising lead on extra memory consumption is that we're loading the CA certificates from disk ourselves, which causes unreasonable space and time overhead for HTTPS. This boils down to what is effectively a workaround for #4175.
We don't have any additional metrics that were requested. This is code that is incredibly difficult to instrument in production.
Trott commentedon Jul 26, 2017
Should this remain open? Or can it be closed?
bnoordhuis commentedon Jul 28, 2017
I'll close it out, I don't think there is anything we can do here. (Node could print a warning when THP is enabled but THP isn't always a problem so I don't think that's the right thing to do.)
Suggestions welcome, though!