Skip to content

core: global location watcher subscribes to all of $HOME — 2M+ file home dirs starve the event loop (~900% CPU, multi-GB memory), /api/model 503 #55937

Open
Open
core: global location watcher subscribes to all of $HOME — 2M+ file home dirs starve the event loop (~900% CPU, multi-GB memory), /api/model 503#55937

Description

@CHANGGELY
CHANGGELY
opened on Oct 16, 2026

Environment

  • opencode: v2.0.30 (also reproduced on v2.0.21; both installed via npm @opencode/cli)
  • OS: macOS 26.6.2 (Tahoe, Build 25G83), Apple Silicon M4 (arm64)
  • Zero user plugins, zero custom providers, brand-new config (fully reproducible with a pristine HOME)

Symptoms

  1. opencode serve boots fine, then the server process pins 300–900% CPU and memory keeps climbing to several GB (%mem ~20% and rising). It never recovers.
  2. GET /api/model hangs until the location-plugin activation timeout and returns 503 "Location plugins did not finish activating" — hence the TUI shows "No items available" in the model picker and "No provider selected".
  3. On v2.0.21 the blowup is deferred to the first chat message: as soon as a message is sent, the server child process hits ~587% CPU + multiple GB RSS, and the TUI logs Timed out connecting repeatedly, then shows "Connection lost… Reconnecting".
  4. TUI↔server SSE repeatedly logs Event stream stalled → "Restarting service…" loops, even with all proxy env vars unset.

Debug-log evidence (v2.0.30, --log-level debug)

The server's location watcher subscribes to the entire home directory and filesystem root:

watcher subscribe path=/ type=entries ignores=0 role=server
watcher subscribe path=/private type=entries ignores=0
watcher subscribe path=/Library/Application Support type=entries ignores=0
watcher subscribe path=/Users/chuan type=entries ignores=0     <-- $HOME
watcher subscribe path=/Users/chuan/.config/opencode type=directory

This machine's $HOME contains 2,130,792 files (iCloud Drive sync + years of dev projects). After the watcher subscribes, CPU explodes (sampled while at ~900% CPU: all hot samples burn inside JIT-compiled JS frames, no native symbol names available).

Decisive A/B experiment

Run $HOME file count Server CPU GET /api/model
Real home 2,130,792 909% peak → ~330% sustained, memory climbing 503, activation timeout
Fake empty home (env HOME=/tmp/oc-home opencode serve) 0 0.0–0.2% 200 OK in 165 ms, full free-model list returned 
# reproduce (A: broken)
opencode serve &
curl -u "opencode:$PW" http://127.0.0.1:4096/api/model   # 503 after activation timeout, CPU pinned

# reproduce (B: healthy)
env HOME=$(mktemp -d) opencode serve &
curl -u "opencode:$PW" http://127.0.0.1:4096/api/model   # 200 OK in 165 ms

Relation to #55272

Same failure shape (CPU-heavy work starving the shared event loop → HTTP/SSE stall), but completely different trigger surface: this reproduces with zero user plugins — no 30×3 native-plugin fixture needed. The trigger is the global location watcher subscribing to $HOME// on machines where those trees are huge. Likely same root (event-loop starvation), different entry point.

Expected fix / ask

  • Don't recursively watch the whole $HOME (and /) by default, or
  • provide an escape hatch (env var / config) to disable or scope the global location watcher, or
  • make the initial watch scan incremental/async so it cannot starve the shared event loop.

Happy to provide sample call-stack dumps or any further data points.

Activity

  1. opencode-agent commented on Oct 16, 2026

    @opencode-agent
    opencode-agentBot
    on Oct 16, 2026 – with OpenCode Agent
    Contributor

    Thanks for the detailed report and the A/B comparison. I tried to reproduce this on Linux with @opencode/cli@2.0.30: a HOME with 500k files and a loop that kept creating files deep inside it while opencode serve --log-level debug ran. The debug log showed the same watches you posted (/ and $HOME as type=entries, ~/.config/opencode as type=directory). The server stayed healthy, though: around 10–20% CPU, and GET /api/model returned 200 in a few milliseconds.

    Those entries watches aren't recursive. Each is a single-directory fs.watch that only cares about a few config file names. On Linux that's an inotify watch on just that directory, so changes deeper in $HOME never reach OpenCode. On macOS, I believe the runtime uses FSEvents for directory watches, which reports changes from the whole subtree. If so, every change under $HOME or / (such as iCloud Drive syncing) would reach OpenCode only to be discarded. That fits what you're seeing, but I haven't confirmed it.

    Confirming it needs a maintainer with a Mac, so I've left this for them. Related issues: #55272 (same event-loop starvation, triggered by plugins), #53811 and #54114 (watcher CPU loops on macOS/skill dirs), #45611 (skip watching filesystem roots). If you can share a sample of the server process taken while CPU is pinned, that would help show whether the time is spent handling fs.watch events.

  2. CHANGGELY commented on Oct 16, 2026

    @CHANGGELY
    CHANGGELY
    on Oct 16, 2026
    Author

    Thanks for the quick reproduction attempt and the platform-semantics hypothesis. New experiments on macOS narrow this down further — and partly correct my own initial framing: the watcher events themselves are NOT the trigger. The trigger is the activation pass started by GET /api/model (and presumably any request that awaits location activation).

    Controlled experiment (v2.0.30, real ~2.1M-file $HOME, fresh boot each time)

    Step CPU Notes
    opencode serve, idle, no requests 0.0% watcher subscribed to /, /Users/chuan etc. per debug log — cheap
    touch 200 new files deep in $HOME 0.3% watcher event flood is NOT the problem
    GET /api/model 917% CPU pinned across 3 samples
    second GET /api/model hangs 20s+, HTTP 000 event loop fully starved, server unresponsive
    same requests with HOME=$(mktemp -d) 0.0% CPU, 200 OK in 165 ms full model list returned

    Memory: RSS climbs to ~4.5 GB (%mem 17.8 and rising) during the pinned phase.

    sample evidence: a self-recursive JS frame

    Captured while CPU was ~900% (sample <pid> 4). Bun strips symbols so frames show as offsets, but the tree contains the same code offset nested inside itself at many depths — a JS function recursively calling itself:

    ! : |   + ! : | 477 ???  (in opencode.exe)  load address 0x104bec000 + 0x31e82b4
! : |   + ! : |     +     ! : | 465 ???  (in opencode.exe)  load address 0x104bec000 + 0x31e82b4
! : |   + ! : |     +     ! : |   +   ! : | +   3 ???  (in opencode.exe)  load address 0x104bec000 + 0x31e82b4
... (same offset recursing at multiple depths)

    All hot samples (1500+/1526) sit under the main thread inside JIT frames (in <unknown binary>), consistent with a single runaway synchronous recursion rather than GC or I/O wait. The recursion depth varies between captures, which fits "enumerate/scan a huge tree synchronously" — and the multi-GB RSS growth suggests per-node allocations along the way.

    Updated picture

    • Linux repro (your 500k-file HOME staying healthy at 10–20% CPU) vs macOS pinning at ~900% on a 2.1M-file HOME: consistent with a macOS-specific cost in the activation scan (FSEvents-backed fs semantics in Bun, or a platform-specific enumeration path), but the trigger is the activation pass, not watch-event delivery.
    • Likely same family as core: CPU-heavy plugin setup and full-scope teardown starve the shared event loop #55272 (shared event loop starved by CPU-heavy work), different entry point: zero user plugins needed.

    Ask (updated)

    1. The activation pass should not synchronously enumerate the entire $HOME// — make it incremental/async or scope it.
    2. An escape hatch (env var) to skip the global location scan would unblock affected users immediately.

    Full sample output available on request. Happy to test a fix build.

  3. Dante-dan commented on Oct 16, 2026

    @Dante-dan
    Dante-dan
    on Oct 16, 2026
    Contributor

    Your follow-up rules out the watch-event flood hypothesis. I traced current dev, but haven't reproduced your macOS failure or identified the hot function. The ancestor config walk checks fixed paths; the built-in skill scans target config/compatibility skill directories. I haven't found evidence that activation enumerates all of HOME.

    There is a built-in JS CPU profiler in v2.0.30: send SIGPROF to the server PID before the first /api/model request:

    kill -PROF <server-pid>
# Once the log says "CPU profile started", immediately repeat your /api/model request.

    After ten seconds it writes cpu-<pid>-*.cpuprofile in the OpenCode log directory. Could you inspect it locally and share just the hottest function names, source module URLs (with personal paths redacted), and their sample percentages? No workspace contents or full profile are needed. This should help distinguish recursive JS work from the native frames in sample.

    Also, does the request still fail with OPENCODE_FILEWATCHER_DISABLE=1 set when launching the server, keeping HOME and the requested location directory unchanged? That existing switch disables watcher subscriptions; comparing it with the profile would narrow the activation path without adding another setting.

  4. CHANGGELY commented on Oct 16, 2026

    @CHANGGELY
    CHANGGELY
    on Oct 16, 2026
    Author

    Ran both of your asks. Results below — and one of them corrects an inference I made in my previous comment.

    1. kill -PROF cpuprofile (v2.0.30, window 10.07s, 237 nodes, 81 samples)

    Self time, all frames above 0.05%:

    self % function source module URL
    62.11% create (none) — native/Bun builtin frame, no source URL
    36.85% Socket node:net:787
    0.27% dlopen (none) — native
    0.14% values (none) — native
    0.09% parse (none) — native
    ≤0.04% everything else — all remaining frames are JS in file:///$bunfs/root/chunk-*.js

    Ancestor chains:

    • create ← L [chunk-kemvbb8p.js:3] ← (anonymous) [chunk-kemvbb8p.js:17] ← runInAsyncScope [node:async_hooks:202] ← evaluate / runLoop [chunk-kemvbb8p.js:17] ← afterScheduled / runTasks [chunk-kemvbb8p.js:3] ← (root). So the hot native create is invoked from the Effect runtime scheduler chunk.
    • Socket [node:net:787] ← Socket [node:_http_server:789] ← onServerConnection [node:_http_server:573] (root-level child).

    Root's children: processTicksAndRejections, afterScheduled, onServerConnection, onNodeHTTPRequest.

    Correction to my previous comment: my earlier sample-based "runaway JS recursion" reading was an over-interpretation of trampolined frames. The cpuprofile shows the repeating deepest path is the Effect runtime loop itself — ~effect/Effect/evaluate → runLoop → evaluate → E → uu, 6 repeats at depth 40, with self time ≈0.00% each. No user-level JS recursion is burning CPU; ~99% of sampled self time sits in two native frames.

    Caveat worth being explicit about: 81 samples over 10.07s means exactly one thread was busy for the whole window, while the process reports 800–900% CPU. So the profile appears to cover the main JS thread only; the remaining ~800% is on threads the profiler isn't sampling.

    Ignoring the profile, same run measured: CPU 824%, RSS 3.1 GB, GET /api/model → 503 after 11.6s.

    2. OPENCODE_FILEWATCHER_DISABLE=1 — switch verified effective, hang unchanged

    I first confirmed the switch is actually honored rather than silently ignored: without it and with --log-level debug, the log emits watcher subscribe … lines; with it set, zero watcher log lines appear. Yet:

    env idle CPU CPU during /api/model RSS result
    real $HOME 0.0% 866% 3.5 GB 503 after 9.7s
    real $HOME + OPENCODE_FILEWATCHER_DISABLE=1 0.0% 769–866% 3.5–4.8 GB 503 / 30s timeout (HTTP 000)

    So watcher subscriptions are excluded as the trigger — consistent with the event-flood test (idle 0.0%, touch of 200 files → 0.3%).

    3. A/B narrowing the trigger surface (relevant to "no evidence activation enumerates all of $HOME")

    $HOME config data dir CPU /api/model
    real (2,130,792 files) real real 800–900% 503
    fresh temp $HOME copied verbatim from real .config/opencode (same plugin list, accounts, skills) none 0.9% 200, 146 ms, 8404 B
    fresh temp $HOME copied verbatim copied real .local/share/opencode (sessions, storage, opencode.db; 11k files) 0.1% 200, 859 ms, 106 KB

    Not plugins, not skills config, not sessions/storage/DB. Identical config + data inside an empty $HOME is fast; the real $HOME pins. So something outside .config/opencode and .local/share/opencode is doing the work.

    4. Log evidence of what gets subscribed (run 5a8d12cf, v2.0.30)

    At boot only location services booted (durationMs=66). The subscriptions appear ~10.5 s later — i.e. lazily at activation, inside the pinned window:

    watcher subscribe path=/ type=entries ignores=0
watcher started   path=/ type=entries backend=node ignores=0
watcher subscribe path=/Users/chuan type=entries ignores=0
watcher started   path=/Users/chuan type=entries backend=node ignores=0
watcher subscribe path="/Library/Application Support" type=entries ignores=0
watcher started   path="/Library/Application Support" type=entries backend=node ignores=0
watcher subscribe path=/private type=entries ignores=0
watcher subscribe path=/private/tmp type=entries ignores=0
watcher subscribe path=/private/tmp/oc-test type=entries ignores=0
watcher subscribe path=/Users/chuan/.mirasim/skills type=directory ignores=0   (backend=fs-events)
watcher subscribe path=/Users/chuan/.config/opencode type=directory ignores=3

    type=entries subscriptions on / and on /Users/chuan land exactly in the pinned window, and /Users/chuan alone is 2.13M entries on this machine.

    One tension I'd rather flag than paper over: with OPENCODE_FILEWATCHER_DISABLE=1 those subscriptions are absent and the hang still reproduces. So either the entries enumeration runs on a path not gated by that switch, or a second code path walks $HOME independently of the watcher.

    Ask / next

    • If you can map chunk-kemvbb8p.js → the mangled L that calls the native create, that would identify the allocation site directly.
    • I can attach the path-redacted .cpuprofile, or run an instrumented build if you point me at one.
    • I can also bisect which $HOME subtree triggers this (.cache 466k / Library 1.57M / Projects 713k files) if that's useful to you.

    Repro on this machine (paths redacted): real $HOME, opencode serve --print-logs --log-level debug from any directory → kill -PROF <server-pid>, wait for CPU profile started, then GET /api/model with the printed password → 503 and event loop starved; same binary with a fresh $HOME containing copies of the real config → 200 in 146 ms.

  5. CHANGGELY commented on Oct 16, 2026

    @CHANGGELY
    CHANGGELY
    on Oct 16, 2026
    Author

    Root cause found: a symlink cycle reachable through the skill roots — not $HOME size, and not the file watcher

    First, a correction of my two earlier comments. Both of my earlier hypotheses are now falsified by direct A/B experiments on the same machine. The real trigger is a self-referencing symlink cycle inside a skill directory, and the runaway happens during skill discovery.

    The smoking gun

    ~/Projects/skill-pandaai-factor-online/.claude/skills/pandaai-factor-online -> /Users/chuan/Projects/skill-pandaai-factor-online
~/Projects/skill-pandaai-factor-online/.cursor/skills/pandaai-factor-online -> /Users/chuan/Projects/skill-pandaai-factor-online

    The symlinked skill target contains a symlink back into itself, so walking …/.claude/skills/pandaai-factor-online never terminates.

    Decisive A/B on the real home (macOS, opencode v2.0.30)

    Only variable changed: the two cycle symlinks above.

    # state CPU RSS GET /api/model
    R2 cycle symlinks present 817.8 % 3918 MB 000 (25 s timeout)
    R1 cycle symlinks removed 1.8 % 499 MB 200 OK, 0.82 s, 106,942 B → 122 models

    Everything else identical: same $HOME (2,130,792 files), same opencode.json, same storage dirs, same watcher configuration. With the cycle gone, v2.0.30 is fully usable on this machine, model list included. So home-directory size is not the cause.

    Falsified earlier claims

    earlier claim experiment result
    2M+ files in $HOME starve the event loop 2.5 M synthetic zero-byte files under a fake $HOME 200 OK, 0.0 % CPU
    the global watcher scanning $HOME is the trigger OPENCODE_FILEWATCHER_DISABLE=1, confirmed with --log-level debug (watcher subscribe lines → 0) unchanged: 866 % CPU / 503
    duplicate skill names collide 2 roots × 11 identical skill names, no cycle 200 OK, 0.5 % CPU
    the skill's content is the trigger same content, cycle symlinks stripped, mounted in 2 roots 200 OK, 0.1 % CPU
    a plain intra-directory cycle is enough 1 root, <skill>/.claude/skills -> .. 200 OK
    symlinked skill entries are enough 2 roots, symlinked entries pointing at a cycle-free directory 200 OK, 0.0 % CPU
    plugins are involved plugin: [] in opencode.json throughout —

    Minimum conditions I could reproduce

    1. a skill reachable from a skill root whose target tree contains a symlink pointing back into that target tree, and
    2. that skill being visible from ≥ 2 skill roots.

    In my environment ~/.claude/skills, ~/.agents/skills and ~/.mirasim/skills are all symlinks resolving to the same directory, so the same skill gets registered three times — and only then does the scan run away. A single live root holding the exact same cycle did not reproduce, which points at the cross-root de-duplication/merge step rather than at a plain recursive directory walk.

    Minimal repro

    TGT=/tmp/skill-cycle; mkdir -p "$TGT"
printf -- '---\nname: skill-x\ndescription: x\n---\n' > "$TGT/SKILL.md"
mkdir -p "$TGT/.claude/skills"
ln -s "$TGT" "$TGT/.claude/skills/skill-x"        # cycle

mkdir -p ~/.claude/skills ~/.mirasim/skills
ln -s "$TGT" ~/.claude/skills/skill-x
ln -s "$TGT" ~/.mirasim/skills/skill-x

opencode serve    # then GET /api/model → CPU pegs ~800 %, request times out

    Where the cycle came from (third party, not opencode)

    scripts/install.py shipped with the pandaai-factor-online skill. Its install_project() symlinks the skill into <directory>/.claude/skills/<NAME> pointing at the repo root SRC; when invoked as install.py project from inside the repo it falls back to Path.cwd(), i.e. directory == SRC, producing <SRC>/.claude/skills/<NAME> -> <SRC>. There is no guard against directory == SRC.

    Suggestion

    Whatever creates the layout, a directory shape should not be able to take opencode down. Hardening I'd suggest on the opencode side:

    1. Do not follow symlinks while walking skill/discovery roots — or, if following is intentional, keep a visited set of realpaths plus a depth cap so a cycle degrades to "skill skipped".
    2. Apply the same protection in whichever step merges/de-duplicates a skill seen from several roots.
    3. Add a node-visit budget or timeout to the discovery path, so a pathological layout yields a warning instead of a pegged event loop.

    Happy to run an instrumented build for any of these, and to re-attach the cpuprofile if useful.

  6. CerisonAutomation commented on Oct 22, 2026

    @CerisonAutomation
    CerisonAutomation
    on Oct 22, 2026

    The watcher subscribes to every ancestor directory of a discovered location — all the way up to /.

    opencode 2.1.2 (Homebrew), macOS 27.0.1, M4 Max / 36 GB.

    Distinct watcher started paths captured from the server log:

    path=/
path=/Users
path=/Users/cb
path="/Library/Application Support"
path=/private
path=/private/var
path=/private/var/folders
path=/private/var/folders/m6
path=/private/var/folders/m6/k0xt0jw57f7831mf7j2q4tqw0000gn
path=/private/var/folders/m6/k0xt0jw57f7831mf7j2q4tqw0000gn/T
path=/private/var/folders/m6/k0xt0jw57f7831mf7j2q4tqw0000gn/T/opencode
path=/Users/cb/.agents/skills
path=/Users/cb/.claude/skills
path=/Users/cb/.config/opencode/skills
path=/Users/cb/.skills-manager/skills/<name>/SKILL.md   (x ~130)

    path=/ is the filesystem root. The /private/var/folders/... chain is the ancestor chain of the agent scratch directory (see #55006) — that location alone pulls in /, /private, /private/var, /private/var/folders, /private/var/folders/m6, .../k0xt0jw57f7831mf7j2q4tqw0000gn, .../T and .../T/opencode.

    Two backends appear in the log: backend=node (3122 events) and backend=fs-events (1629). The node backend is the fallback used for the ancestor paths FSEvents cannot cover — which is exactly the set that turns a file change anywhere on the volume into an event storm inside the server.

    Reproduced symptom on this machine: ripgrep invoked with cwd=/Users/cb (688 occurrences in the log) together with --glob=*.py — i.e. globbing the entire home directory — and a single find "$HOME" -xdev -type f -not -path "$HOME/Library/*" did not finish within 180 s.

    Cost before mitigation on this machine: 90 MB free RAM, a 13.6 GB compressor, 95 % of a 12 GB swapfile in use, load averages 6.71 / 7.85 / 9.11.

    Mitigation available today — watcher.ignore does work for the root walk. With these patterns configured, the /Users and /Users/cb subscriptions drop to 0 and only explicitly-scoped directories remain watched:

    Library/, .cache/, .npm/, .local/share/, .Trash/, Applications/, Downloads/,
Documents/, Movies/, Music/, Pictures/, Public/, node_modules/, .git/,
.docker/, .orbstack/, .gradle/, .cargo/, .rustup/, .ollama/, .pnpm-store/,
Library/Application Support/, *.log, *.tmp, *.swp, .skills-manager/,
.config/opencode/skills/

    Two caveats worth knowing when reproducing:

    1. watcher.ignore does not filter the explicit per-skill-directory subscriptions (~/.claude/skills, ~/.agents/skills, and the same pair inside each project-like directory opencode discovers) — those use a different code path. They are individually cheap, but the ignore list alone cannot reduce the subscription set to zero.
    2. The expensive part is the re-subscribe cascade, not the steady-state watch set. In one log file I count 4466 watcher stopped and 4754 watcher started events. A change under one watched skills directory tears down and recrees the watchers for all of them, which is what produces the CPU spike during build churn.

    A fix that refuses to subscribe to a path that is an ancestor of the workspace root (or of the scratch dir) would remove the need for a user-maintained ignore list entirely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions