Skip to content

Trigram index has no file limit: a 6.5M-file non-git root took 16 min and about 9 GB #1007

Open

Description

@iceteaSA

Problem

The trigram (search) index has no limit on the number of files. When a session starts in a very large directory that is not a git repository, AFT builds a trigram index of all files in it. In our case this one build took 16 minutes and increased the module's committed memory by about 9 GB. Other roots on the same module did not change.

Evidence (aft 0.62.0, subc module)

  • A session started with its directory set to a shared scratch directory under /tmp (147 GB, not a git repository, no project markers). AFT bound that directory as the project root.

  • Log:

    • index_event kind=build_started plane=search … root=<scratch dir> at 19:45:29Z
    • index_event kind=build_ready plane=search … root=<scratch dir> elapsed_ms=969598 files=6509755 trigrams=651600 at 20:01:38Z
  • Committed memory of the module (Anonymous + Swap from /proc/<pid>/smaps_rollup), with 14 open roots in all samples:

    Time (UTC) Committed
    19:40:52 3.60 GB
    20:10:52 13.0 GB
    20:41:04 13.0 GB
    21:11:07 13.0 GB
  • The root was quiesced at 20:02:40Z (quiesced unbound root … cause=goodbye_unbound). Committed memory stayed at about 13 GB until an external memory guard restarted the module at 21:18Z. We did not find whether the index data or allocator retention kept the memory.

  • The semantic index refused the same root (more than 20000 files exceeds limit of 20000). The trigram build has no equivalent check.

Code (main abdc5b4cc)

  • The only build-time file limit is semantic.max_files (crates/aft/src/config.rs:300-307, default 20,000 at config.rs:367).
  • project_root_is_too_broad (crates/aft/src/commands/configure.rs:678-697) disables indexing only for path shapes (volume root, account home, their ancestors). It does not look at size, so a large directory below the home or under /tmp gets a full trigram build.
  • The max_files values in search_index.rs (for example GrepExaminationLimits at search_index.rs:1415-1420) limit queries, not the build.

Steps to reproduce

  1. Make a directory that is not a git repository and has a few million small files (for example, many copies of a node_modules tree).
  2. Start a session with this directory as its working directory, so that AFT binds it as the project root.
  3. Look for the build_ready plane=search line and the module's memory during and after the build.

Expected

A limit on the trigram build, the same as for the semantic index. For example, a search file limit with a clear warning such as "trigram index skipped: more than N files", or the root_too_broad behaviour (explicit-path tools stay available) when the walk passes the limit. The memory that the build uses should also go back after the root is quiesced or evicted.

Activity

  1. iceteaSA commented on Oct 28, 2026

    @iceteaSA
    ContributorAuthor

    Second occurrence on the same 0.62.0 module, with two more facts.

    1. A module restart does not stop it. A session whose directory was the same large non-git root still held a route. When the module started again after an unrelated stall exit, the host bound that route again within one second, and AFT built the trigram index again:

      18:55:42Z  project root set: <large root>
      …          build_ready plane=search … files=6927588 elapsed_ms=423365
      

      Committed memory (Anonymous + Swap): 10.0 GB 8 minutes after the start, 15.3 GB after 38 minutes.

    2. The memory did not go back after the root was quiesced. The root was quiesced at 19:56:49Z (cause=goodbye_unbound), but committed memory stayed at 13.1 to 15.4 GB for the next 2 h 50 min, until an external memory guard restarted the module.

    So a file limit on the trigram build is the important part of this issue. With no limit, one long-lived session in a large directory makes every module restart build the index again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions