perf: adopt the mimalloc-pprof allocator for heap/leak profiling; verify on-CPU, off-CPU, and symbolization from running-process #1361

Ask

Swap fbuild's global allocator to the mimalloc-pprof allocator that comes in through the zccache / running-process stack, so the daemon can track allocation leaks and dump heap state. While we're in there, verify that on-CPU, off-CPU, and symbolization all actually work end-to-end when driven from zccache / running-process against an fbuild process.

Where we are today

crates/fbuild-daemon/src/main.rs:1:

#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;

That is plain mimalloc ([dependencies.mimalloc] workspace = true). It gives us mimalloc's allocation performance and nothing else — no sampling, no heap dump, no way to answer "what is holding this memory" when a long-lived daemon grows. Every leak investigation today is a guess.

What the stack already exports

mimalloc-pprof (the 0.9 line, the v3 mimalloc port) is already in the dependency graph via running-process, which pins it unconditionally — it's a Windows-first mimalloc fork, so unlike the usual heap profilers there is no platform gate and no MSVC gap to work around. The reference usage is running-process's own fixture, testbins/src/bin/mimalloc_leaker.rs:

#[global_allocator]
static ALLOC: mimalloc_pprof::MiMalloc = mimalloc_pprof::MiMalloc;

mimalloc_pprof::prof::start_seeded(SAMPLE_RATE, SAMPLE_SEED);
mimalloc_pprof::prof::dump_proto_file(Path::new(&path));
mimalloc_pprof::prof::stop();

Two properties that matter for us:

• It is a drop-in for the allocator we already use. Same mimalloc underneath; the type just moves from mimalloc::MiMalloc to mimalloc_pprof::MiMalloc.
• The dump is pprof protobuf directly — dump_proto_file writes the wire format, so nothing in fbuild has to parse or lower a bespoke text format. It feeds straight into the same tooling that consumes CPU profiles.

Sampling is statistical (sample every Nth byte), not every-allocation, so the steady-state cost is tunable rather than a hard tax on the daemon's hot path.

On the consumption side, running-process-probe-daemon already advertises both ops (crates/running-process-probe-daemon/src/query.rs:366):

"heap_profile"    => Some(wire::SupportedOp::HeapProfile as i32),
"off_cpu_profile" => Some(wire::SupportedOp::OffCpuProfile as i32),

and its profile/ module is built around exactly the split we want — the sampler pushes raw instruction pointers on the hot path, and symbolization (PDB/DWARF/Mach-O) happens on a worker thread off that path, late enough that it still works after the target exits. Backpressure drops samples and counts them (ProfileMetrics::samples_dropped) rather than blocking the profiled process.

Scope

1. Allocator swap. Move fbuild-daemon from mimalloc::MiMalloc to mimalloc_pprof::MiMalloc. Decide whether this is unconditional or behind a feature — the argument for unconditional is that the profiler is off until started, and a build-flag-gated profiler is never available when you actually need it (production daemon, user's machine, one reproduction). Pin mimalloc-pprof in the workspace next to the existing zccache/running-process pins, which the root Cargo.toml already requires to move in lockstep.
2. Control surface. Wire prof::start_seeded / dump_proto_file / stop to daemon endpoints so a heap profile can be requested from a running daemon without restarting it. A restart destroys the leak, which makes a restart-only profiler useless for the thing it's for. Land the dump under the existing dev/prod-isolated paths rather than a new location.
3. Verify the three modes actually work against fbuild, not just against running-process' own fixtures:
   • on-CPU — sampled profile of a real build, with frames that name fbuild functions.
   • off-CPU — the off_cpu_profile op. This is the one we care most about: fbuild's daemon spends most of its wall clock waiting on subprocesses, sockets, and the filesystem, so an on-CPU-only profile is close to blind for our actual latency.
   • symbolization — on all three CI platforms. This is where it will break first: Windows needs PDBs, and our release profile's debug-info settings decide whether a profile comes back as function names or as bare hex addresses. Check what [profile.release] currently emits before assuming it works.
4. Note in the results whether the sampled allocator measurably changes build wall-clock; if it does, that decides question 1.

Why now

The daemon is long-lived by design and holds caches, so it is the process where a slow leak is both most likely and least visible. We already ship the allocator that can answer this — we are just not using the half that reports.

Related

• dylint: Windows-only code is never linted — the Dylint job runs on ubuntu only #1359 — the Dylint job runs ubuntu-only, so Windows-only code is unlinted. Same shape of gap: a tool we have, pointed away from the platform that needs it most.