Skip to content

bench: intermittent per-command daemon restart ('daemon binary updated') adds ~200 ms to every fbuild command #1476

Closed

Description

@zackees

Found while landing #1464/#1466 (part of #1469).

Symptom

Some benchmark-build-comparison.yml runs show fbuild warm ≈ 251 ms (normally ≈ 46 ms) and cold inflated by about 200 ms. In those runs every fbuild command in benchmark.log prints daemon binary updated, restarting..., and the value holds steady across all 3 trials. It comes from the direct-HTTP acquisition path in crates/fbuild-cli/src/daemon_client.rs (ensure_direct_daemon_running → should_restart_daemon: same version, CLI-side sibling fbuild-daemon mtime > the daemon's reported source_mtime).

It is pre-existing and intermittent

Ruled out

  • An mtime change during the run: stat of target/release/fbuild-daemon (mtime and inode) stayed constant across a run. That was checked in the debug steps, which happened to be good runs.
  • Inherited stdio holding Command::output() open: the daemon's fds 0/1/2 are /dev/null, /dev/null and daemon.log.
  • soldr cache hit vs miss for fbuild-daemon: the forced-miss run was good.
  • JSON f64 precision: serde_json round-trips f64 exactly.

Next step

Catch a bad run with RUST_LOG=fbuild_cli=info on the benchmark step. The daemon needs restart (daemon v… mtime=…, cli v… mtime=…) tracing line gives both sides of the comparison.

Independently, should_restart_daemon could compare a build identity (the git sha or a build id the daemon reports) instead of raw mtimes. That would make the same-version dev-rebuild check immune to anything that perturbs mtimes.

Activity

  1. zackees commented on Sep 26, 2026

    @zackees
    MemberAuthor

    Additional finding: across 5 runs, per-phase data (cold_phases_ms, #1465) is present only in the restart runs:

    run restart lines phases
    36234375809 (#1474) 9 19
    36236358015 (#1474 rerun) 0 0
    36236206722 (debug) 0 0
    36238206814 (#1475) 9 19
    36239761342 (#1475 rerun) 0 0

    The runner passes FBUILD_PERF_LOG / FBUILD_PERF_LOG_JSON only to the timed fbuild build commands, not to the untimed clean cache that (re)starts the daemon. So the phase log is written only when a timed build command itself starts the daemon, which is exactly the restart mode. In normal runs cold_phases_ms is empty.

    That makes two things to fix together:

    1. The benchmark runner: pass the perf-log env to the cold-prep clean cache / daemon-restart commands too, or have the CLI forward the perf-log request to an already-running daemon.
    2. The restart mode itself: see above. One of its visible effects is that the phase log gets populated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions