Skip to content

feat(daemon): report resident memory in daemon info/status so leak reports carry numbers (#1360) #1785

Closed

Description

@zackees

Summary

fbuild-daemon can reach ~3.9 GB RSS and stop answering /health (#1360), and every report of it so far — including the measurement I just ran — depends on an external sampler (tasklist, ps, /proc) because fbuild exposes nothing about its own memory. fbuild daemon status shows state, pid, operation, uptime: no resident size, no trend.

That gap is why #1360 took months to characterise: by the time anyone looks, the evidence is a screenshot from another tool.

Proposal

Report resident size from the daemon itself, so the timeline is available during and after the event:

  • /api/daemon/info gains rss_bytes (and peak_rss_bytes, which is what a wedged daemon is judged by).
  • DaemonStatus (the on-disk status file, crates/fbuild-daemon/src/status_manager.rs) gains rss_bytes, refreshed on the existing maintenance tick — cheap, one Option<u64>, and it survives the daemon dying, which the HTTP endpoint does not.
  • fbuild daemon status prints it next to the pid.

Acceptance criteria

  • A value is reported on Linux, macOS and Windows through fbuild_core::platform::process (no raw cfg!, no /proc reads outside the platform tree; the boundary scanner already owns this).
  • None — not a bogus zero — when the host cannot report it (older kernel, permission, recycled pid).
  • A test asserts the field round-trips through the status file, and that a daemon started without the capability still serialises.
  • Post-dump heap_dump response (already shipped, perf: adopt the mimalloc-pprof allocator for heap/leak profiling; verify on-CPU, off-CPU, and symbolization from running-process #1361) and rss_bytes agree directionally: a leak large enough to wedge the daemon shows up as growth in both.

Notes

  • Adding these fields means touching every DaemonInfoResponse literal (handlers/health.rs, broker/backend.rs, the models.rs test module), so it is one PR, not a drive-by.
  • Not to be confused with CI: board build jobs flake with 'daemon did not become healthy after 3 spawn attempts' (2 release dry runs in a row) #1627: the spawn-budget and readiness diagnostics there are already shipped; this is about memory observability.
  • Measurement that motivates it: idle daemon on main 2ba57549 held 84–90 MiB RSS flat for 8.75 min (peak 181.8 MiB HWM), 148 live heap samples — i.e. the mechanism works and is cheap enough to always-on log at a coarse interval.

Activity

  1. added a commit that references this issue on Oct 29, 2026
  2. zackees commented on Oct 29, 2026

    @zackees
    MemberAuthor

    Shipped in #1786 (squash f2f6883, merged 2026-10-29T17:05:55Z).

    Verified on origin/main @ f2f6883:

    • fbuild_core::platform::process::resident_memory(pid) -> Option<MemoryUsage> with
      resident_bytes / peak_resident_bytes; Linux /proc VmRSS+VmHWM,
      macOS proc_pidinfo + getrusage (self), Windows GetProcessMemoryInfo.
      Returns None — never 0 — when the host cannot answer.
    • /api/daemon/info and daemon_status.json carry rss_bytes / peak_rss_bytes
      (crates/fbuild-daemon/src/handlers/health.rs, models.rs, status_manager.rs,
      broker/backend.rs), refreshed on the maintenance tick so the last sample
      survives a wedged daemon.
    • fbuild daemon status prints Memory: 97.0 MB (peak 188.1 MB).

    Proof: live isolated daemon returned rss_bytes=106532864, peak=197287936.
    Gates: bash test 3956 passed / 0 failed (47 binaries), workspace clippy -D warnings rc=0,
    fmt 0, dylint core/daemon/cli rc=0, platform boundary 44/0, cross-check for
    x86_64-pc-windows-msvc and x86_64-apple-darwin green. Merge-queue CI: linux Check,
    Dylint, CI selected coverage all SUCCESS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions