Repository navigation
Segfault with different nodejs procs (I have coredump info) #47324
Description
Version
v18.15.0
Platform
Linux server-host 5.15.0-69-generic #76-Ubuntu SMP Fri Mar 17 17:19:29 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Subsystem
No response
What steps will reproduce the bug?
I don't know how to reproduce. But this seems to happen on our busiest servers. The server has plenty of RAM available (80GB out of 128GB). And it has 96 CPUs.
How often does it reproduce? Is there a required condition?
It seems to happen when my server is under heavy traffic (although CPU and RAM usage are not high). It happens maybe once or twice per day. I cannot reproduce it readily.
What is the expected behavior? Why is that the expected behavior?
it should not seg fault
What do you see instead?
I opened the coredump with coredump info and coredumpctl debug. Here is some info. I got the backtrace as well. I'm happy to send this coredump if it's helpful. It's only about 10Mb.
coredumpctl info
PID: 35768 (PM2 v5.3.0: God)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Thu 2023-03-30 14:35:27 EDT (2min 33s ago)
Command Line: $'PM2 v5.3.0: God Daemon (/root/.pm2)' "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
Executable: /root/.nvm/versions/node/v18.15.0/bin/node
Control Group: /user.slice/user-0.slice/session-7.scope
Unit: session-7.scope
Slice: user-0.slice
Session: 7
Owner UID: 0 (root)
Boot ID: cf3aaa25cae342f8903aefa322ba0e8f
Machine ID: 53ee02db24a24b008c218ca45204acc3
Hostname: server-host
Storage: /var/lib/systemd/coredump/core.PM2\x20v5\x2e3\x2e0:\x20God.0.cf3aaa25cae342f8903aefa322ba0e8f.35768.1680201327000000.zst (present)
Disk Size: 10.5M
Message: Process 35768 (PM2 v5.3.0: God) of user 0 dumped core.
Found module /root/.nvm/versions/node/v18.15.0/bin/node with build-id: 417328b1953d476d486d4bfa7f9a79d6571d694c
Found module linux-vdso.so.1 with build-id: 2dfff2e3b9908e870457adfd8ab8457adccee5ca
Found module ld-linux-x86-64.so.2 with build-id: 61ef896a699bb1c2e4e231642b2e1688b2f1a61e
Found module libc.so.6 with build-id: 69389d485a9793dbe873f0ea2c93e02efaa9aa3d
Found module libpthread.so.0 with build-id: e62798b68557abb4bc5548aba2640cd5ab948f36
Found module libgcc_s.so.1 with build-id: 09c4935b79388431a1248f6a98e00d7dc81b8513
Found module libm.so.6 with build-id: 27e82301dba6c3f644404d504e1bb1c97894b433
Found module libstdc++.so.6 with build-id: f57e02bfadacc0c923c82457d5e18e1830b5faea
Found module libdl.so.2 with build-id: 952123c3bdce7c3370553717942b4878d1fad797
Stack trace of thread 35768:
#0 0x00007fb7447122a9 tcache_get (libc.so.6 + 0xa52a9)
#1 0x00007fb744a519cc _Znwm (libstdc++.so.6 + 0xae9cc)
#2 0x00000000010805c0 n/a (/root/.nvm/versions/node/v18.15.0/bin/node + 0xc805c0)
#3 0x0000000000d7a1a4 n/a (/root/.nvm/versions/node/v18.15.0/bin/node + 0x97a1a4)
#4 0x0000000000b11fa3 n/a (/root/.nvm/versions/node/v18.15.0/bin/node + 0x711fa3)
#5 0x0000000000c6f528 n/a (/root/.nvm/versions/node/v18.15.0/bin/node + 0x86f528)
#6 0x0000000001676dba n/a (/root/.nvm/versions/node/v18.15.0/bin/node + 0x1276dba)
#7 0x0000000001677790 n/a (/root/.nvm/versions/node/v18.15.0/bin/node + 0x1277790)
#8 0x000000000167d534 n/a (/root/.nvm/versions/node/v18.15.0/bin/node + 0x127d534)
#9 0x000000000166b95e n/a (/root/.nvm/versions/node/v18.15.0/bin/node + 0x126b95e)
#10 0x0000000000abda2d n/a (/root/.nvm/versions/node/v18.15.0/bin/node + 0x6bda2d)
#11 0x0000000000bc1874 n/a (/root/.nvm/versions/node/v18.15.0/bin/node + 0x7c1874)
#12 0x0000000000b36434 n/a (/root/.nvm/versions/node/v18.15.0/bin/node + 0x736434)
#13 0x0000000000b3a02f n/a (/root/.nvm/versions/node/v18.15.0/bin/node + 0x73a02f)
#14 0x00007fb744696d90 __libc_start_call_main (libc.so.6 + 0x29d90)
#15 0x00007fb744696e40 __libc_start_main_impl (libc.so.6 + 0x29e40)
#16 0x0000000000aba37e n/a (/root/.nvm/versions/node/v18.15.0/bin/node + 0x6ba37e)
coredump debug with backtrace
Type "apropos word" to search for commands related to "word"...
Reading symbols from /root/.nvm/versions/node/v18.15.0/bin/node...
[New LWP 35768]
[New LWP 35773]
[New LWP 35771]
[New LWP 35778]
[New LWP 35775]
[New LWP 35815]
[New LWP 35772]
[New LWP 35814]
[New LWP 35817]
[New LWP 35774]
[New LWP 35816]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `PM2 v5.3.0: God Daemon (/root/.pm2) '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fb7447122a9 in tcache_get (tc_idx=<optimized out>) at ./malloc/malloc.c:3196
3196 ./malloc/malloc.c: No such file or directory.
[Current thread is 1 (Thread 0x7fb74466a7c0 (LWP 35768))]
(gdb) bt
#0 0x00007fb7447122a9 in tcache_get (tc_idx=<optimized out>) at ./malloc/malloc.c:3196
#1 __GI___libc_malloc (bytes=64) at ./malloc/malloc.c:3313
#2 0x00007fb744a519cc in operator new(unsigned long) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00000000010805c0 in v8::internal::BackingStore::Allocate(v8::internal::Isolate*, unsigned long, v8::internal::SharedFlag, v8::internal::InitializedFlag) ()
#4 0x0000000000d7a1a4 in v8::ArrayBuffer::NewBackingStore(v8::Isolate*, unsigned long) ()
#5 0x0000000000b11fa3 in node::Environment::allocate_managed_buffer(unsigned long) ()
#6 0x0000000000c6f528 in node::LibuvStreamWrap::ReadStart()::{lambda(uv_handle_s*, unsigned long, uv_buf_t*)#1}::_FUN(uv_handle_s*, unsigned long, uv_buf_t*) ()
#7 0x0000000001676dba in uv__read (stream=stream@entry=0x7fb7403cc0c0) at ../deps/uv/src/unix/stream.c:1105
#8 0x0000000001677790 in uv__stream_io (loop=<optimized out>, w=0x7fb7403cc148, events=1) at ../deps/uv/src/unix/stream.c:1270
#9 0x000000000167d534 in uv__io_poll (loop=loop@entry=0x526dfc0 <default_loop_struct>, timeout=<optimized out>) at ../deps/uv/src/unix/epoll.c:374
#10 0x000000000166b95e in uv_run (loop=0x526dfc0 <default_loop_struct>, mode=UV_RUN_DEFAULT) at ../deps/uv/src/unix/core.c:406
#11 0x0000000000abda2d in node::SpinEventLoop(node::Environment*) ()
#12 0x0000000000bc1874 in node::NodeMainInstance::Run() ()
#13 0x0000000000b36434 in node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResult const*) ()
#14 0x0000000000b3a02f in node::Start(int, char**) ()
#15 0x00007fb744696d90 in __libc_start_call_main (main=main@entry=0xab1f70 <main>, argc=argc@entry=2, argv=argv@entry=0x7ffe26fd3918) at ../sysdeps/nptl/libc_start_call_main.h:58
#16 0x00007fb744696e40 in __libc_start_main_impl (main=0xab1f70 <main>, argc=2, argv=0x7ffe26fd3918, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe26fd3908)
at ../csu/libc-start.c:392
#17 0x0000000000aba37e in _start ()
Additional information
Runny Ubuntu 22.04 Jammy with latest patches.
Activity
bnoordhuis commentedon Mar 31, 2023
The crash happens inside tcache_get() and that's part of glibc's internal thread-local arena allocator. It could mean either a bug in glibc or memory corruption in node or V8 but it's impossible to tell which. Is the stack trace always the same?
amit777 commentedon Mar 31, 2023
@bnoordhuis I just got another coredump on a different node process (not pm2). It's on one of regular processes which is not actually heavily loaded at all:
amit777 commentedon Mar 31, 2023
Also, I'm happy to run do whatever instrumentation is necessary to help isolate the issue. So if you would like me to run a debug version (or if it's easy to install a debug version with nvm), please let me know.
More information, the node process is started by pm2 in cluster mode.
amit777 commentedon Mar 31, 2023
lscpuoutput incase it's useful[-]Segfault with PM2 Daemon (I have coredump info)[/-][+]Segfault with different nodejs procs (I have coredump info)[/+]amit777 commentedon Mar 31, 2023
Also, I wanted to note that (I think) I got this as well in Node 16.19.1. We recently updated to Node 18.15.0 because it's LTS and because we wanted to see if node 18 magically fixed the problem.
bnoordhuis commentedon Mar 31, 2023
You have a bunch of native add-ons loaded (the *.node files). Try excluding those because any one of them can be a source of memory corruption bugs. We don't accept bug reports where native add-ons are involved.
I didn't see them in your original report but the stack trace looked different there too. The second stack trace includes the "malloc(): unaligned tcache chunk detected" error message, which is highly indicative of a use-after-free or a buffer overrun/underrun.
One thing you could try is running node under valgrind but, caveat emptor, it'll be very slow. Building node from source with Address Sanitizer support (
./configure --enable-asan && make -j32) is another option.amit777 commentedon Mar 31, 2023
Thank you for the pointers! I will explore those native addons.
would building with Address Sanitizer support have significant performance impact?
bnoordhuis commentedon Mar 31, 2023
It's usually on the order of 2-4x but it really depends on what you're doing.
amit777 commentedon Mar 31, 2023
Tried building from source with
--enable-asanbut got the following error (I was able to successfully build without the--enable-asanflag):I'm wondering if the build error may provide a clue or is it probably totally unrelated?
bnoordhuis commentedon Mar 31, 2023
Welcome to the hell of trying to build modern C++. Check the requirements in BUILDING.md; you can't stray too far from them without getting errors like you're seeing.
bnoordhuis commentedon Mar 31, 2023
Ha, that particular error stirred a memory: #43370 (comment)
amit777 commentedon Mar 31, 2023
No problem.. I'm actually just getting this entire server replaced. We've had too many issues with it. I'm going to close this ticket out. Thanks for taking a look so quickly though!
amit777 commentedon Apr 1, 2023
Hi @bnoordhuis , does it look like similar issue happened without any 3rd party native modules from this stack trace? This is the pm2 God daemon crash which I don't believe loads any 3rd party stuff. Also worth noting this crash happened on a completely different server than crashes from above.
bnoordhuis commentedon Apr 2, 2023
Looks like the same stack trace, yes. One other thing you can try is downgrading to ubuntu 20.04 and see if the crashes go away. That would suggest the bug is in glibc or maybe libstdc++.
FWIW, no one else has reported similar crashes so far. My hunch is it's something in your environment rather than in node itself.
amit777 commentedon Apr 3, 2023
You are probably right.. I'm just confused because we doing nothing special in our env. In any case, we will probably be downgrading to Debian 11 from jammy.. But before we do that, I'm trying one last ditched effort to try and start the processes with
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so pm2 ...In the off chance it helps us to avoid a
malloc()bug withtcache_get.amit777 commentedon Apr 3, 2023
One last note in case anyone comes via google search. So the jemalloc shim seemed to work for a bit, but I did notice periodic core dumps with it for various procs started by pm2 with the following stack trace.
I'm going to move to Debian 11..
amit777 commentedon Apr 8, 2023
as a quick note, we rebuilt the node_modules folder and it seems to have fixed things. This is odd, because we were pretty sure we had already done that. In any case, hope that's helpful for anyone else who sees something like this. We did not downgrade to Debian 11..