Skip to content

[Bug]: Background PR sync and settlement send 500–1,100 GitHub GraphQL calls an hour on a ~50-thread server and keep polling threads whose worktree is gone (18,787 failed lookups and 482,099 "settlement skipped" lines in 11 days) #21372

Closed as not planned

Description

@MattRiddell

Before submitting

Area

apps/server

Steps to reproduce

  1. Run a headless server with ~50 threads, most on branches with an open PR, in a GitHub repository.
  2. Count POST https://api.github.com/graphql from the server process for a day (we first metered through a gh shim; since nightly 2761 the server uses its own HTTPS client with the gh auth token value, so we metered the socket).
  3. Remove a few threads' worktrees (or let a worktree reaper do it) and watch the journal.

Expected behavior

  • One sync tick (10 minutes is fine; configurable) on which all stale PR lookups fall due together and are read in shared GraphQL documents.
  • No lookups for provider-native subagent threads whose root is archived or settled, for threads whose directory is not a repository, or for threads whose worktree no longer exists.
  • No rateLimit probe when the server is idle.
  • Well under GitHub's budget: we target ≤283 calls a day for this server.

Actual behavior

build what we measured
0.0.46-nightly.20261004.2657 one batched PullRequestsByHead document per minute: ~4/h → ~470/h
0.0.46-nightly.20261007.2761 own HTTPS client; 500–1,100 GraphQL calls/h; PR cache fixed in code at 60 s (branch with PR) / 5 min (branch without)
2026-10-13 1,213 getPullRequestSummary a day; PullRequestSyncReactor sweeps every minute; every finished run or checkpoint forces a fresh read
2026-10-05..06 a bare query { rateLimit { cost limit remaining resetAt } } every 10 minutes all night (480/day)
2026-10-01 GET .../stacks?pull_request=<n> 368 → 504 → 2,010 calls a day, every answer []
2026-10-12 493 of 547 settlement candidates were Codex subagent threads whose root thread was already archived
2026-10-14..15 10 ms summary batch window split one tick's reads into two documents; listPullRequestsByHead ran for branches whose open PR was already known

Journal, 2026-10-06 19:10 → 2026-10-17 21:29 (11 days): automatic thread settlement skipped 482,099 lines; thread branch pull request lookup failed 18,787 (GitManager.branchPullRequest.remotes ... (not_a_repository) 13,429, No unknown source control provider is registered 2,458, Failed to spawn Git process 314); pull request sync skipped 1,490; thread pull request update failed 626; PR lookup failed; keeping last known PR state. 331. Most of these are threads whose worktree was removed; the server keeps polling them.

Also: the remote regex (three copies) rejects origin <url> (fetch) [blob:none], so a partial clone resolves to provider unknown, PR badges go blank and merge settlement stops (#12764).

Impact

Major degradation or frequent failure (shares the user's GitHub rate limit with every other tool on the box; PR badges and auto-settlement stopped twice)

Version or commit

0.0.46-nightly.20261004.2657 → 0.0.46-nightly.20261008.2819. Our running build carries the fix below.

Environment

Linux (Ubuntu, kernel 7.0.0-38-generic), compiled @t3code/t3-linux-x64 binary (Node.js v26.8.2 in crash traces), t3 serve --host 127.0.0.1 --port 3773 --base-dir ~/.t3 as a systemd --user service, headless; Tailscale serve in front; clients: macOS desktop app, iOS app, web. 40–50 live threads at a time over 8 Claude Code instances (claudeAgent), 3 Codex instances and ACP providers (Grok CLI, OpenCode with GLM). ~16,900 threads in statev2.sqlite (40.4 GB). Threads are created, woken and stopped by our own tooling over the /ws RPC (orchestration.dispatchCommand), not by a human in the UI.

Running build: 0.0.46-nightly.20261008.2819 with two local server patches (GitHub PR-sync cadence, Grok multi-task wait). Each draft says whether the patch touches the reported area.

Logs or stack traces

thread branch pull request lookup failed ... GitManager.branchPullRequest.remotes ... (not_a_repository)
automatic thread settlement skipped ...
PR lookup failed; keeping last known PR state.

Workaround

We run the fix as fork patches (MattRiddell/t3code PRs #4, #6, #7, #8; four commits, tests included): PR lookup cache 60 s → 10 min; sync and settlement passes on a shared 10-minute tick (entries expire at the tick so they go stale together); finished runs and checkpoints read the cached answer; summary batch window 10 ms → 250 ms; no lookups for subagents of archived/settled roots; branch pass skips a thread whose open PR is known; [blob:none] accepted by the remote regex. Result on our server: 1,414 → 507 → ~283 calls a day. Happy to open these against main if wanted.

Activity

  1. juliusmarminge commented on Oct 18, 2026

    @juliusmarminge
    Member

    Note

    Grok responding on behalf of Julius.

    Closing as a duplicate. The parts of this report are already tracked:

    The one-minute sweep and the PR cache lengths on main are deliberate settings, not defects. A shared 10-minute tick or a configurable cadence would be a product change, so please open that as an idea: https://github.com/pingdotgg/t3code/discussions/new?category=ideas

    Your build also carries a local PR-sync cadence patch, so we can't use these counts as a baseline for main. If a part not listed here still happens on a current nightly without local patches, please open a focused issue for that part.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    duplicateThis issue or pull request already existsvia-triageFiled through npx t3 triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions