fuzz: differential V8 engine occasionally crashes #4786

Activity

1. abrown

   added

   bugIncorrect behavior in the current implementation that needs fixingIncorrect behavior in the current implementation that needs fixing

   on Aug 25, 2022
2. abrown commented on Aug 25, 2022

   

   abrown

   on Aug 25, 2022

   MemberAuthor

   More actions

   cc: @alexcrichton, guess that crash we saw wasn't the spec interpreter after all.
3. abrown

   added

   fuzzingIssues related to our fuzzing infrastructureIssues related to our fuzzing infrastructure

   on Aug 25, 2022
4. alexcrichton commented on Aug 30, 2022

   

   alexcrichton

   on Aug 30, 2022

   Member

   More actions

   I haven't been able to reproduce this locally unfortunately. When I let the fuzzer run long enough it hit #4812 before hitting this fault.

   Can you try running the fuzzer in gdb to see if you can grab a stack trace of the fault? (presuming it doesn't take too long to fault). When using gdb though it's difficult b/c segfaults can normally happen for wasm and there's no great way to handle this. I was able to do handle SIGILL nostop locally in gdb but if you do handle SIGSEGV nostop I think that when it actually hits the crashing failure it lets the process exit which is no good, so you may have to babysit it a bit and double-check that SIGSEGV doesn't look like a normal wasm SIGSEGV (rr might be super helpful here in this regard)
5. bjorn3 commented on Aug 30, 2022

   

   bjorn3

   on Aug 30, 2022

   Contributor

   More actions

   Maybe you can do handle SIGSEGV nostop but also add a breakpoint where the signal handler propagates the SIGSEGV if it didn't originate from the wasm module?
6. alexcrichton commented on Oct 11, 2024

   

   alexcrichton

   on Oct 11, 2024

   Member

   More actions

   I've been unable to fuzz with v8 locally for the longest time so I tried to dig into this today. Turns out it's an issue with memory protection keys so this'll only affect new enough x64 hardware with new enough glibc. That I think would explain why Andrew initially ran into this and I couldn't reproduce. I think that would also help explain why it hasn't showed up on oss-fuzz at all despite being trivial to reproduce locally.

   My best read of the situation is that if you run with v8 and differential fuzzing long enough locally it will eventually crash. This crash, when I've inspected it, always looks like this:

   Thread 1 "differential" received signal SIGSEGV, Segmentation fault.
   0x00002061775c8170 in ?? ()
   (gdb) disas $rip,$rip+20
   Dump of assembler code from 0x2061775c8170 to 0x2061775c8184:
   => 0x00002061775c8170:  jmp    *0x2(%rip)        # 0x2061775c8178
   

   The memory being accessed is just below %rip itself, and the memory address of 0x00002061775c8178 is indeed readable in gdb. This is in fact an indirect jump to Builtins_WasmLiftoffFrameSetup all the time. The problem is that if you print $_siginfo.si_code in gdb it yields 4 which corresponds to SEGV_PKUERR.

   So the problem here is that JIT code isn't allowed to read this code due to MPK and the status of the PKRU. The corruption here seems to be:

   • V8 looks like it uses a single pkey at most.
   • This is entirely unrelated to Wasmtime's PKRU support, if that's disabled the issue still happens.
   • V8 assumes that JIT code executes with pkey 1.
   • V8 uses pkeys to disallow writes to JIT code with pkey 1.
   • V8 assumes a steady-state of pkey 1 is accessible (executable/readable) but not writable
   • V8, when it modifies JIT code, frobs the PKRU to teporarily allow writes

   This is all fine and dandy and Wasmtime shouldn't interfere with it, except it does. The problem is apparently signal handlers (it's always signal handlers). V8 assumes a steady-state PKRU register of 0x5...58. That means no access to keys >=2, readonly/executable access to pkey 1, and all access to pkey 0. When a signal is received, however, it resets the PKRU register to 0x5..54. This notably disables access to all keys >=1, disagreeing with V8's steady state.

   The problem here is that when Wasmtime executes, hits a signal, recovers, and goes back into V8, now V8 is in a corrupt state. V8 thinks JIT code can access pkey 1, but it in fact cannot. A reproduction of this problem can be found in this gist which uses the v8 crate to run some code, then a segfault is generated and caught, then code is run again. This second execution crashes.

   ---

   Ok so what to do about this. I found that v8::V8::set_flags_from_string("--no-memory_protection_keys") looks promising. That fixes the standalone test case here but it causes V8 to segfault in a different location when passed during fuzzing. According to the man page about pkeys signal handlers need to manually restore the state after they're received. Looks like sig{set,long}jmp don't solve this automatically.

   I have run out of ideas of how to fix this. If OSS-Fuzz picks up hardware with MPK we'll probably have to disable differential execution with v8 if nothing else changes.
7. alexcrichton commented on Oct 11, 2024

   

   alexcrichton

   on Oct 11, 2024

   Member

   More actions

   That fixes the standalone test case here but it causes V8 to segfault in a different location when passed during fuzzing.

   Looks like this flag doesn't fully disable MPK. The crash is in V8 with si_code = 4 which shows pkey misconfiguration again. So looks like that flag only partially disables MPK.