Skip to content

Inspect rayon pool is per-root, so a multi-root daemon runs N×8 inspect threads #202

Closed

Description

@iceteaSA

start_dispatch_loop builds a rayon pool sized available_parallelism()/2 clamped to 8, with 8 MB stacks:

// inspect/dispatch.rs:22
rayon::ThreadPoolBuilder::new()
    .num_threads(default_pool_size())          // ncores/2, clamp(1,8)
    .thread_name(|index| format!("aft-inspect-{index}"))
    .stack_size(8 * 1024 * 1024)               // :31 — raised so the duplicates AST walker doesn't SIGABRT

That pool is owned by InspectManager, which is constructed inside AppContext (context.rs:1793) — and AppContext is per project root. So the pool count scales with bound roots rather than with the machine.

Observed

A aft --subc daemon on a 16-core box with 7+ roots bound over its lifetime:

Threads: 195
   88  aft-inspect-*      ← 11 pools × 8
   56  aft-executor-wo*
   24  aft-subc
   10  notify-rs inotify
    8  reqwest-internal
    6  aft-fs-lock-hea*

88 inspect threads at 8 MB of stack reservation each. The count plateaus at roughly (roots × 8) rather than growing per request — 195 → 197 over 90 s with inspect flat at 88 — so this is accumulation, not a leak.

Why it only shows up now

In standalone mode each session got its own short-lived worker process, so one pool existed at a time and died with the process. The daemon is the first thing that holds many roots in one address space for hours, which turns a per-root pool into a multiplier.

It also amplifies the allocator issue in #201: each allocating thread gets its own glibc arena, so thread count drives arena count directly.

Suggested direction

The inspect pool looks like a process-wide resource rather than a per-root one — the work is CPU-bound scanning, and default_pool_size() is already derived from machine parallelism, which only makes sense if there is one pool. A shared pool behind a OnceLock/LazyLock with per-root job routing would keep the current concurrency semantics while making the thread count machine-shaped again.

If per-root isolation is load-bearing for a reason I have not spotted (cancellation scoping, or the heavy_root_work_allowed gate), then sizing the pool by max(1, ncores/2/expected_roots) would at least bound the total — but a shared pool seems cleaner and the gate is already passed in per root.

Happy to send a PR if you want it in the shared-pool shape; I did not want to guess at the isolation requirement first.

Activity

  1. ualtinok commented on Aug 9, 2026

    @ualtinok
    Collaborator

    Accepted. The per-root pool is a holdover from the standalone one-root-per-process world — under the daemon it's N pools × default threads with no global budget, exactly as you measured. The fix direction is a process-wide inspect pool (or a shared budget the per-root schedulers draw from), which also composes with the existing ColdBuildLimiter cap so inspect scans and index builds share one notion of "how much background CPU this process may burn". Queued for the next train alongside #201; both are daemon-era resource-model debts of the same class.

  2. ualtinok commented on Aug 9, 2026

    @ualtinok
    Collaborator

    Fixed on main (8bd35ff4), CI green. Inspect scanning now runs on one process-wide pool sized min(8, available_parallelism) — under the multi-root daemon that's a hard cap instead of N×8, with cross-root scans interleaving at file-task granularity so a large cold scan can't starve a small root's warm scan. The pool composes with the existing process-wide cold-build limiter (limiter admits operations, pool bounds their combined threads; the boundary is documented at the pool site). Threads keep the aft-inspect- naming for operator continuity. Ships in the next release. Both of this week's resource-model reports (this and #201) were the daemon-era debts you said they were — much appreciated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions