You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Trigram index has no file limit: a 6.5M-file non-git root took 16 min and about 9 GB #1007
The trigram (search) index has no limit on the number of files. When a session starts in a very large directory that is not a git repository, AFT builds a trigram index of all files in it. In our case this one build took 16 minutes and increased the module's committed memory by about 9 GB. Other roots on the same module did not change.
Evidence (aft 0.62.0, subc module)
A session started with its directory set to a shared scratch directory under /tmp (147 GB, not a git repository, no project markers). AFT bound that directory as the project root.
Log:
index_event kind=build_started plane=search … root=<scratch dir> at 19:45:29Z
Committed memory of the module (Anonymous + Swap from /proc/<pid>/smaps_rollup), with 14 open roots in all samples:
Time (UTC)
Committed
19:40:52
3.60 GB
20:10:52
13.0 GB
20:41:04
13.0 GB
21:11:07
13.0 GB
The root was quiesced at 20:02:40Z (quiesced unbound root … cause=goodbye_unbound). Committed memory stayed at about 13 GB until an external memory guard restarted the module at 21:18Z. We did not find whether the index data or allocator retention kept the memory.
The semantic index refused the same root (more than 20000 files exceeds limit of 20000). The trigram build has no equivalent check.
Code (main abdc5b4cc)
The only build-time file limit is semantic.max_files (crates/aft/src/config.rs:300-307, default 20,000 at config.rs:367).
project_root_is_too_broad (crates/aft/src/commands/configure.rs:678-697) disables indexing only for path shapes (volume root, account home, their ancestors). It does not look at size, so a large directory below the home or under /tmp gets a full trigram build.
The max_files values in search_index.rs (for example GrepExaminationLimits at search_index.rs:1415-1420) limit queries, not the build.
Steps to reproduce
Make a directory that is not a git repository and has a few million small files (for example, many copies of a node_modules tree).
Start a session with this directory as its working directory, so that AFT binds it as the project root.
Look for the build_ready plane=search line and the module's memory during and after the build.
Expected
A limit on the trigram build, the same as for the semantic index. For example, a search file limit with a clear warning such as "trigram index skipped: more than N files", or the root_too_broad behaviour (explicit-path tools stay available) when the walk passes the limit. The memory that the build uses should also go back after the root is quiesced or evicted.
Second occurrence on the same 0.62.0 module, with two more facts.
A module restart does not stop it. A session whose directory was the same large non-git root still held a route. When the module started again after an unrelated stall exit, the host bound that route again within one second, and AFT built the trigram index again:
Committed memory (Anonymous + Swap): 10.0 GB 8 minutes after the start, 15.3 GB after 38 minutes.
The memory did not go back after the root was quiesced. The root was quiesced at 19:56:49Z (cause=goodbye_unbound), but committed memory stayed at 13.1 to 15.4 GB for the next 2 h 50 min, until an external memory guard restarted the module.
So a file limit on the trigram build is the important part of this issue. With no limit, one long-lived session in a large directory makes every module restart build the index again.
Activity
mason: cut cold trigram build per-file memory (cortexkit#1007)
mason: make the symbol prewarm share the index file table and stop on…
mason: size the build memory test for CI and assert on its slope (cor…
mason: tighten the build memory slope bound to catch one extra path c…
Merge pull request cortexkit#1023 from cortexkit/land/trigram-build-m…
iceteaSA commentedon Oct 28, 2026
Second occurrence on the same 0.62.0 module, with two more facts.
A module restart does not stop it. A session whose directory was the same large non-git root still held a route. When the module started again after an unrelated stall exit, the host bound that route again within one second, and AFT built the trigram index again:
Committed memory (
Anonymous + Swap): 10.0 GB 8 minutes after the start, 15.3 GB after 38 minutes.The memory did not go back after the root was quiesced. The root was quiesced at 19:56:49Z (
cause=goodbye_unbound), but committed memory stayed at 13.1 to 15.4 GB for the next 2 h 50 min, until an external memory guard restarted the module.So a file limit on the trigram build is the important part of this issue. With no limit, one long-lived session in a large directory makes every module restart build the index again.