Activity

1. juliusmarminge

   added

   needs more infoInitial triage showed no bug. Awaiting more infoInitial triage showed no bug. Awaiting more info

   and removed

   needs more infoInitial triage showed no bug. Awaiting more infoInitial triage showed no bug. Awaiting more info

   on Oct 21, 2026
2. mm0nst3r commented on Oct 24, 2026

   

   mm0nst3r

   on Oct 24, 2026 – with ChatGPT Codex Connector

   More actions

   Windows confirmation on official nightly 3617: shared-reader waits disable controls; reproducible query/index evidence

   Posting here to consolidate evidence for the already-reported recovery/history query problem, rather than opening a duplicate. The corresponding permission/UI failure is tracked in #24221.

   New evidence: Windows 11, official 0.1.3-nightly.20261024.3617, a 53.98 GB state database, reader-lock waits up to 30.18 seconds, and an isolated query benchmark with exact ordered-result checks and an index removal/reapplication control. The candidate indexes have not been applied to the live installation.

   Observed behavior and impact

   During ordinary work with long-lived threads and delegated agents, sending messages and other mutating controls intermittently become unusable. Settings → Connections shows “Permissions not checked”; the user also sees “This connection cannot change threads.”

   Reloading the interface restored controls once, then failed to help during a later episode. That later episode recovered while only read-only diagnostics were being collected: no restart, index installation, or repair was performed during that recovery. This is intermittent, but it blocks foreground work when it occurs.

   Expected: normal thread history and background work should not starve session validation or leave a valid local session represented as lacking permission.

   Environment and provenance

   • Windows 11 Pro x64, build 26300 (10.0.26300).
   • Official desktop nightly 0.1.3-nightly.20261024.3617, release commit 27e0c684c585f955252d65f06206dd11a143bff5.
   • Local environment, mixed Claude and Codex work.
   • Live statev2.sqlite: 53,983,956,992 bytes (50.28 GiB) at measurement.
   • One process-counter sample during the problem: server private working set 2,463,461,376 bytes, I/O read rate 447,895,691 bytes/s, write rate 45,008,943 bytes/s. These are Windows process I/O counters, including cached I/O, not proof of physical disk throughput or disk saturation.
   • At that sample the OS reported 23,348,708 KiB free physical memory and C: had 461,810,630,656 bytes free. This was not a disk-full condition.
   • SQL benchmark runtime: Node.js 24.18.0 / SQLite 3.53.1. This names the separate benchmark runtime, not an inferred Electron SQLite version.
   • The live schema/index inventory and read-only query plans were captured before the experiment. All experimental DDL below ran only in a disposable fixture. No live history was deleted or truncated; no application patch or migration-ledger change was applied.

   Live trace evidence

   Captured around 2026-10-24 13:14–13:28 UTC. The following examples retain operation names, durations and connection/lock attributes only. Tokens, request URLs, IDs, paths and conversation data are omitted.

   {
     "name": "sql.transaction",
     "durationMs": 30186.8685,
     "parent": "ws.orchestrationV2.projectShellItems",
     "attributes": {
       "db.name": "statev2.sqlite",
       "db.connection": "reader",
       "db.lock_wait_ms": 30182.046
     }
   }
   {
     "name": "sql.transaction",
     "durationMs": 22185.4472,
     "parent": "mcp.loadCaller",
     "attributes": {
       "db.name": "statev2.sqlite",
       "db.connection": "reader",
       "db.lock_wait_ms": 22178.687
     }
   }
   {
     "name": "sql.transaction",
     "durationMs": 18112.2613,
     "parent": "ThreadSettlementServiceV2.sweep",
     "attributes": {
       "db.name": "statev2.sqlite",
       "db.connection": "reader",
       "db.lock_wait_ms": 18110.35
     }
   }

   The first transaction spent about 30,182 of 30,187 ms waiting to acquire the reader. Separate session-row SELECT spans reached 16,179.0155 ms, with a maximum recorded lock wait of 16,178.771 ms.

   The recorded expensive query paths included:

   • ProjectionStore.getRuntimeRecoveryProjection from turn start and finalization: unfinished-item subqueries for runs, nodes, subagents and provider threads.
   • ProjectionStore.readCanonicalProjection → sql.transaction → IdleCompactionService.compactWhenDue: message-order and provider-thread/history reads.
   • Message ordering by json_extract(payload_json, '$.messageId'), scanning historical turn items before grouping.

   Some SQL spans have no lock-wait annotation, so their duration cannot be treated as pure SQLite execution time. Summed waits across concurrent requests are also not wall-clock outage duration.

   Relevant release-source paths

   These links are pinned to the installed release, not a moving main branch:

   • Authentication getById explicitly uses the reader.
   • Recovery and thread-record methods are routed to that reader.
   • Idle compaction loads thread records before checking eligibility.
   • Client session-state timeout and permission handling, as described in [Bug]: Send turns grey ("This connection cannot change threads") when the session permission check times out under database load #24221.

   Together these support the shared-reader contention diagnosis. They do not prove that every long wait has a single cause, or that the indexes alone resolve the complete UI failure.

   Reproduction and isolated experiment

   Observed workload reproduction

   1. Use official nightly 3617 with a large, long-lived V2 history and normal concurrent/delegated work.
   2. Open/switch threads and attempt to send messages while background work continues.
   3. When controls become unavailable, correlate the connection-permission state with SessionStore.verify/auth SELECT spans and db.connection=reader lock waits.
   4. On an isolated database copy, inspect the recovery queries and their EXPLAIN QUERY PLAN output using the reproducer below.

   The live symptom is workload-dependent; this is not a claim of a deterministic small fresh-home reproduction.

   Fixture construction and limits

   The timing fixture was built from an immutable, closed 2026-10-23 backup, using the official 3617 table/index definitions captured from the current installation:

   • One heavily used thread: 8,846 runs, 8,846 run attempts, 60,234 nodes, 86,621 turn items, 23,662 messages, 3,458 subagent records.
   • All 4,232 provider-thread records and 587 provider-session records were retained because the recovery query can traverse those records globally.
   • Relevant provider turns, session bindings and runtime requests were retained.
   • Completed non-message/non-subagent turn-item payloads were replaced with {}; unfinished payloads and message/subagent payloads were retained. This reduces private-data volume and fixture size while retaining the rows/payloads selected by the tested queries.
   • Baseline size 476,733,440 bytes; indexed size 479,313,920 bytes. All three indexes together added 2,580,480 bytes in this fixture.
   • The fixture is not a complete application/event-replay database and was never started as a T3 server. It is private and is not attached.
   • The baseline used the captured official indexes, not the older backup's custom experimental index. No planner statistics were collected.

   Each query was executed three times per phase; the tables below use the median of those three timings. The recovery total is a sum of 11 per-query medians, not measured end-to-end server/UI latency. Ordered row counts and SHA-256 digests matched across every phase.

   Timing controls

   Run	Without candidate indexes: recovery sum	With candidate indexes	Ratio
   Original fixture harness	866.215 ms	97.767 ms	8.86×
   Remove/reapply control in the same fixture	493.723 ms	74.939 ms	6.59×
   Rerun of the public reproducer below: A → B	530.883 ms	105.307 ms	5.04×
   Same public reproducer: remove indexes A → reapply B	636.461 ms	116.160 ms	5.48×

   The variation is visible; these are same-host measurements under background machine load, not a controlled device-performance claim. The fresh public reproducer still shows about 5× lower recovery-query cost.

   Initial per-query evidence

   Query	Rows	Before median	After median	Ordered results
   recovery_0 — thread metadata	1	0.645 ms	0.631 ms	identical
   recovery_1 — runs/latest/unfinished owners	44	135.230 ms	22.488 ms	identical
   recovery_2 — run attempts	39	0.193 ms	0.147 ms	identical
   recovery_3 — live nodes	45	134.590 ms	10.922 ms	identical
   recovery_4 — live subagents	6	145.268 ms	13.751 ms	identical
   recovery_5 — provider sessions	1	0.182 ms	0.231 ms	identical
   recovery_6 — provider threads	13	254.737 ms	12.508 ms	identical
   recovery_7 — provider turns	0	18.390 ms	8.921 ms	identical
   recovery_8 — runtime requests	0	0.096 ms	0.074 ms	identical
   recovery_9 — streaming messages	0	48.053 ms	27.948 ms	identical
   recovery_10 — pending turn items	6	128.830 ms	0.146 ms	identical
   canonical_pending_control	11	131.311 ms	78.258 ms	identical
   message_order	18712	325.186 ms	142.558 ms	identical

   canonical_pending_control uses status IN (SELECT value FROM json_each(?)). Its query plan did not switch to a candidate partial index. Its timing change is not evidence that this path was fixed; this remains relevant to #23926/#23929.

   Candidate indexes tested only in the fixture

   This is evidence for useful access paths, not a request that these local names or this installation method become the upstream implementation.

   CREATE INDEX t3_local_perf_unfinished_filter_v1 ON orchestration_v2_projection_turn_items(thread_id,status,type,run_id,node_id,provider_thread_id) WHERE status IN ('pending','running','waiting');
   CREATE INDEX t3_local_perf_unfinished_order_v1 ON orchestration_v2_projection_turn_items(thread_id,ordinal,turn_item_id) WHERE status IN ('pending','running','waiting');
   CREATE INDEX t3_local_perf_message_order_v1 ON orchestration_v2_projection_turn_items(thread_id,json_extract(payload_json,'$.messageId'),ordinal) WHERE type IN ('user_message','assistant_message');

   The pending filter index and the pending ordinal index serve different access paths. The message-order index avoids the grouping scan. A status-leading general index was also checked at the schema/query-plan stage and did not remove the ordered pending scan, so it was not part of the measured candidate.

   Representative before/after plans

   runs/latest/unfinished owners
   Before:
   SEARCH item USING INDEX orchestration_v2_projection_turn_items_thread_run_idx (thread_id=? AND run_id>?)
   After:
   SEARCH item USING COVERING INDEX t3_local_perf_unfinished_filter_v1 (thread_id=? AND status=? AND type=? AND run_id>?)
   

   provider threads
   Before:
   SEARCH item USING INDEX orchestration_v2_projection_turn_items_thread_run_idx (thread_id=?)
   SEARCH item USING INDEX orchestration_v2_projection_turn_items_thread_run_idx (thread_id=?)
   After:
   SEARCH item USING COVERING INDEX t3_local_perf_unfinished_filter_v1 (thread_id=? AND status=?)
   SEARCH item USING COVERING INDEX t3_local_perf_unfinished_filter_v1 (thread_id=? AND status=?)
   

   pending turn items
   Before:
   SEARCH item USING INDEX orchestration_v2_projection_turn_items_thread_ordinal_idx (thread_id=?)
   After:
   SEARCH item USING INDEX t3_local_perf_unfinished_order_v1 (thread_id=?)
   

   message_order
   Before:
   SEARCH orchestration_v2_projection_turn_items USING INDEX orchestration_v2_projection_turn_items_thread_run_idx (thread_id=?)
   USE TEMP B-TREE FOR GROUP BY
   After:
   SEARCH orchestration_v2_projection_turn_items USING INDEX t3_local_perf_message_order_v1 (thread_id=?)
   

   The provider-thread outer scan remains in the plan. The change avoids its historical turn-item subquery scans; it does not eliminate every scan.

   Reproducer

   Save the following as t3-reader-index-repro.mjs. Use only an isolated writable database copy, containing a thread with substantial history, plus its thread ID:

   node t3-reader-index-repro.mjs isolated-copy.sqlite THREAD_ID results.json --isolated-copy
   

   It fetches the exact recovery SQL from the pinned release, runs baseline → indexed → rollback-baseline → reindexed, checks every ordered result, and removes its three test indexes before closing. It prints/saves only plans, row counts, hashes and timings, not payloads or thread IDs. The script was executed successfully against a separate copy of the private fixture; its measured results are included above. On a different database/hardware the numbers will differ.

   Complete tested Node.js reproducer

   // Run ONLY against an isolated, writable database copy. Never use a live T3 home.
   // Usage: node t3-reader-index-repro.mjs <isolated-copy.sqlite> <thread-id> <results.json> --isolated-copy
   import fs from 'node:fs';
   import crypto from 'node:crypto';
   import { performance } from 'node:perf_hooks';
   import { DatabaseSync } from 'node:sqlite';
   const [file, threadId, output, consent] = process.argv.slice(2);
   if (!file || !threadId || !output || consent !== '--isolated-copy') throw Error('Use an isolated database copy and the documented arguments.');
   if (!fs.existsSync(file) || fs.existsSync(output)) throw Error('Database must exist; output must be new.');
   const commit = '27e0c684c585f955252d65f06206dd11a143bff5';
   const url = `https://raw.githubusercontent.com/pingdotgg/t3code/${commit}/apps/server/src/orchestration-v2/ProjectionStore.ts`;
   const response = await fetch(url);
   if (!response.ok) throw Error(`Source fetch failed: ${response.status}`);
   const source = await response.text();
   const start = source.indexOf('const getRuntimeRecoveryProjection:');
   const end = source.indexOf('} satisfies ProjectionRuntimeRecoveryState;', start);
   if (start < 0 || end < start) throw Error('Pinned source shape changed');
   const queries = [...source.slice(start, end).matchAll(/sql<PayloadRow>`([\s\S]*?)`/g)].map((m, i) => ({
     name: `recovery_${i}`, sql: m[1].replace(/\$\{threadId\}/g, '?')
   }));
   if (queries.length !== 11 || queries.some(q => q.sql.includes('${'))) throw Error('Unexpected query extraction');
   queries.push({name: 'message_order', sql: `SELECT json_extract(payload_json, '$.messageId') AS message_id, MIN(ordinal) AS ordinal
    FROM orchestration_v2_projection_turn_items WHERE thread_id = ? AND type IN ('user_message', 'assistant_message')
    GROUP BY json_extract(payload_json, '$.messageId')`});
   queries.push({name: 'canonical_pending_control', sql: `SELECT payload_json FROM orchestration_v2_projection_turn_items
    WHERE thread_id = ? AND status IN (SELECT value FROM json_each(?)) ORDER BY ordinal, turn_item_id`,
    args: [threadId, '["pending","running","waiting"]']});
   const indexes = [
    `CREATE INDEX t3_local_perf_unfinished_filter_v1 ON orchestration_v2_projection_turn_items
    (thread_id,status,type,run_id,node_id,provider_thread_id) WHERE status IN ('pending','running','waiting')`,
    `CREATE INDEX t3_local_perf_unfinished_order_v1 ON orchestration_v2_projection_turn_items
    (thread_id,ordinal,turn_item_id) WHERE status IN ('pending','running','waiting')`,
    `CREATE INDEX t3_local_perf_message_order_v1 ON orchestration_v2_projection_turn_items
    (thread_id,json_extract(payload_json,'$.messageId'),ordinal) WHERE type IN ('user_message','assistant_message')`
   ];
   const names = indexes.map(s => s.match(/CREATE INDEX (\w+)/)[1]);
   const db = new DatabaseSync(file);
   db.exec('PRAGMA busy_timeout=1000;');
   if (!db.prepare('SELECT 1 FROM orchestration_v2_projection_threads WHERE thread_id=?').get(threadId)) throw Error('Thread not found');
   for (const name of names) if (db.prepare('SELECT 1 FROM sqlite_schema WHERE name=?').get(name)) throw Error('Candidate indexes already present; use a baseline copy.');
   function measure(label) {
    const result = [];
    for (const q of queries) {
     const args = q.args ?? Array((q.sql.match(/\?/g) ?? []).length).fill(threadId);
     const statement = db.prepare(q.sql);
     const times = []; let reference, rows;
     for (let repetition = 0; repetition < 3; repetition++) {
      const begin = performance.now(); const data = statement.all(...args); times.push(performance.now()-begin);
      rows = data.length; const h = crypto.createHash('sha256');
      for (const row of data) h.update(JSON.stringify(row)+'\n');
      const digest = h.digest('hex');
      if (reference && reference !== digest) throw Error(`Unstable ordered results: ${q.name}`);
      reference = digest;
     }
     result.push({query:q.name, rows, sha256:reference, timesMs:times, medianMs:[...times].sort((a,b)=>a-b)[1],
      plan:db.prepare('EXPLAIN QUERY PLAN '+q.sql).all(...args).map(r=>r.detail)});
    }
    return {label, recoverySumOfMediansMs:result.filter(x=>x.query.startsWith('recovery_')).reduce((a,x)=>a+x.medianMs,0), queries:result};
   }
   const beforeSchema = db.prepare('SELECT type,name,tbl_name,sql FROM sqlite_schema ORDER BY type,name').all();
   const phases = [];
   try {
    phases.push(measure('baseline'));
    for (const sql of indexes) db.exec(sql);
    phases.push(measure('indexed'));
    for (const name of names) db.exec('DROP INDEX '+name);
    phases.push(measure('rollback-baseline'));
    for (const sql of indexes) db.exec(sql);
    phases.push(measure('reindexed'));
    for (const phase of phases) for (let i=0;i<phase.queries.length;i++) {
     const base=phases[0].queries[i], current=phase.queries[i];
     if(base.rows!==current.rows || base.sha256!==current.sha256) throw Error('Results differ: '+current.query);
    }
   } finally {
    for (const name of names) db.exec('DROP INDEX IF EXISTS '+name);
    const afterSchema=db.prepare('SELECT type,name,tbl_name,sql FROM sqlite_schema ORDER BY type,name').all();
    if(JSON.stringify(beforeSchema)!==JSON.stringify(afterSchema)) throw Error('Schema was not restored');
    db.close();
   }
   // Contains query names, row counts, timings, hashes and plans; no payloads or thread IDs.
   fs.writeFileSync(output, JSON.stringify({commit,node:process.version,at:new Date().toISOString(),identicalOrderedResults:true,phases},null,2)+'\n');
   console.log(JSON.stringify({identicalOrderedResults:true,phases:phases.map(p=>({label:p.label,recoverySumOfMediansMs:p.recoverySumOfMediansMs}))}));

   Machine-readable public-reproducer measurements (all phases; no private payloads)

   {
     "commit": "27e0c684c585f955252d65f06206dd11a143bff5",
     "node": "v24.18.0",
     "identicalOrderedResults": true,
     "phases": [
       {
         "label": "baseline",
         "recoverySumOfMediansMs": 530.8829,
         "queries": [
           {
             "query": "recovery_0",
             "rows": 1,
             "timesMs": [
               0.6131,
               0.4423,
               0.5641
             ],
             "sha256": "c5ababb0f5ba0118793efba54247b21fb5107af5e08196ab59bac4dc682f08e4"
           },
           {
             "query": "recovery_1",
             "rows": 44,
             "timesMs": [
               92.8345,
               86.2456,
               86.1931
             ],
             "sha256": "6e3b4b85fe396ac8c0c36b162b4dfe65d11f032e05bf49406665ccaf823a7517"
           },
           {
             "query": "recovery_2",
             "rows": 39,
             "timesMs": [
               0.2377,
               0.1107,
               0.1038
             ],
             "sha256": "90ef9bc383e68f831cc5f415815305e3ba09dbcfdb2982f352f45bf7af010876"
           },
           {
             "query": "recovery_3",
             "rows": 45,
             "timesMs": [
               78.9255,
               75.9954,
               77.8129
             ],
             "sha256": "2bb8e7f3f16d7f50b6d2988992e429b718bc933d8269f7c433e86d99bfcb57a9"
           },
           {
             "query": "recovery_4",
             "rows": 6,
             "timesMs": [
               80.6573,
               75.9124,
               83.9005
             ],
             "sha256": "4bc283c202288cdd9c8b877bea9a134cc692f6184ada1b4b569a6c2cfc40eadc"
           },
           {
             "query": "recovery_5",
             "rows": 1,
             "timesMs": [
               0.2573,
               0.0996,
               0.0786
             ],
             "sha256": "615b323a5607a4decf49b8c8bc56dc78127307b177c8aed93f49f8917bb2e699"
           },
           {
             "query": "recovery_6",
             "rows": 13,
             "timesMs": [
               158.7557,
               169.319,
               172.5948
             ],
             "sha256": "f8bd5b0a4a9b9e48ffb3028e67a675c47c056796c1d0df6e50b50324167b5150"
           },
           {
             "query": "recovery_7",
             "rows": 0,
             "timesMs": [
               10.3385,
               9.6735,
               9.2509
             ],
             "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
           },
           {
             "query": "recovery_8",
             "rows": 0,
             "timesMs": [
               0.1163,
               0.0474,
               0.0456
             ],
             "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
           },
           {
             "query": "recovery_9",
             "rows": 0,
             "timesMs": [
               24.8671,
               25.4204,
               24.8029
             ],
             "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
           },
           {
             "query": "recovery_10",
             "rows": 6,
             "timesMs": [
               81.4857,
               77.702,
               82.6206
             ],
             "sha256": "62b626c7796749117432c04025294984c0cd72bb61209b4537b376594de7d2b5"
           },
           {
             "query": "message_order",
             "rows": 18712,
             "timesMs": [
               239.6799,
               229.5451,
               220.0705
             ],
             "sha256": "441d97035a93a0abae13e744f94b4d863485cbb9cd8d96c6438e80ff9d391f42"
           },
           {
             "query": "canonical_pending_control",
             "rows": 11,
             "timesMs": [
               82.4767,
               78.8608,
               82.1859
             ],
             "sha256": "48ef96f696db4462268b4698f9ce3786c8545ed9078e90111677206a5306b715"
           }
         ]
       },
       {
         "label": "indexed",
         "recoverySumOfMediansMs": 105.3072,
         "queries": [
           {
             "query": "recovery_0",
             "rows": 1,
             "timesMs": [
               0.514,
               0.7055,
               0.6442
             ],
             "sha256": "c5ababb0f5ba0118793efba54247b21fb5107af5e08196ab59bac4dc682f08e4"
           },
           {
             "query": "recovery_1",
             "rows": 44,
             "timesMs": [
               19.7152,
               21.0318,
               22.4767
             ],
             "sha256": "6e3b4b85fe396ac8c0c36b162b4dfe65d11f032e05bf49406665ccaf823a7517"
           },
           {
             "query": "recovery_2",
             "rows": 39,
             "timesMs": [
               0.4032,
               0.2572,
               0.185
             ],
             "sha256": "90ef9bc383e68f831cc5f415815305e3ba09dbcfdb2982f352f45bf7af010876"
           },
           {
             "query": "recovery_3",
             "rows": 45,
             "timesMs": [
               17.3416,
               14.1544,
               12.2148
             ],
             "sha256": "2bb8e7f3f16d7f50b6d2988992e429b718bc933d8269f7c433e86d99bfcb57a9"
           },
           {
             "query": "recovery_4",
             "rows": 6,
             "timesMs": [
               11.2805,
               9.9605,
               11.4954
             ],
             "sha256": "4bc283c202288cdd9c8b877bea9a134cc692f6184ada1b4b569a6c2cfc40eadc"
           },
           {
             "query": "recovery_5",
             "rows": 1,
             "timesMs": [
               0.3114,
               0.1642,
               0.0999
             ],
             "sha256": "615b323a5607a4decf49b8c8bc56dc78127307b177c8aed93f49f8917bb2e699"
           },
           {
             "query": "recovery_6",
             "rows": 13,
             "timesMs": [
               12.297,
               12.0273,
               12.5402
             ],
             "sha256": "f8bd5b0a4a9b9e48ffb3028e67a675c47c056796c1d0df6e50b50324167b5150"
           },
           {
             "query": "recovery_7",
             "rows": 0,
             "timesMs": [
               13.1131,
               12.0477,
               11.27
             ],
             "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
           },
           {
             "query": "recovery_8",
             "rows": 0,
             "timesMs": [
               0.1273,
               0.0961,
               0.0632
             ],
             "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
           },
           {
             "query": "recovery_9",
             "rows": 0,
             "timesMs": [
               33.1803,
               32.5401,
               34.9586
             ],
             "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
           },
           {
             "query": "recovery_10",
             "rows": 6,
             "timesMs": [
               0.3861,
               0.1538,
               0.1274
             ],
             "sha256": "62b626c7796749117432c04025294984c0cd72bb61209b4537b376594de7d2b5"
           },
           {
             "query": "message_order",
             "rows": 18712,
             "timesMs": [
               124.8226,
               112.0123,
               112.7417
             ],
             "sha256": "441d97035a93a0abae13e744f94b4d863485cbb9cd8d96c6438e80ff9d391f42"
           },
           {
             "query": "canonical_pending_control",
             "rows": 11,
             "timesMs": [
               98.5949,
               94.6558,
               103.4883
             ],
             "sha256": "48ef96f696db4462268b4698f9ce3786c8545ed9078e90111677206a5306b715"
           }
         ]
       },
       {
         "label": "rollback-baseline",
         "recoverySumOfMediansMs": 636.4608,
         "queries": [
           {
             "query": "recovery_0",
             "rows": 1,
             "timesMs": [
               0.7391,
               0.7133,
               0.5373
             ],
             "sha256": "c5ababb0f5ba0118793efba54247b21fb5107af5e08196ab59bac4dc682f08e4"
           },
           {
             "query": "recovery_1",
             "rows": 44,
             "timesMs": [
               108.9142,
               115.5612,
               109.7496
             ],
             "sha256": "6e3b4b85fe396ac8c0c36b162b4dfe65d11f032e05bf49406665ccaf823a7517"
           },
           {
             "query": "recovery_2",
             "rows": 39,
             "timesMs": [
               0.2099,
               0.1099,
               0.1003
             ],
             "sha256": "90ef9bc383e68f831cc5f415815305e3ba09dbcfdb2982f352f45bf7af010876"
           },
           {
             "query": "recovery_3",
             "rows": 45,
             "timesMs": [
               96.2767,
               96.1683,
               91.529
             ],
             "sha256": "2bb8e7f3f16d7f50b6d2988992e429b718bc933d8269f7c433e86d99bfcb57a9"
           },
           {
             "query": "recovery_4",
             "rows": 6,
             "timesMs": [
               88.3469,
               95.3638,
               92.9917
             ],
             "sha256": "4bc283c202288cdd9c8b877bea9a134cc692f6184ada1b4b569a6c2cfc40eadc"
           },
           {
             "query": "recovery_5",
             "rows": 1,
             "timesMs": [
               0.3675,
               0.1187,
               0.1102
             ],
             "sha256": "615b323a5607a4decf49b8c8bc56dc78127307b177c8aed93f49f8917bb2e699"
           },
           {
             "query": "recovery_6",
             "rows": 13,
             "timesMs": [
               189.3902,
               206.8118,
               204.9509
             ],
             "sha256": "f8bd5b0a4a9b9e48ffb3028e67a675c47c056796c1d0df6e50b50324167b5150"
           },
           {
             "query": "recovery_7",
             "rows": 0,
             "timesMs": [
               10.4749,
               12.0891,
               12.4979
             ],
             "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
           },
           {
             "query": "recovery_8",
             "rows": 0,
             "timesMs": [
               0.1217,
               0.0682,
               0.0641
             ],
             "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
           },
           {
             "query": "recovery_9",
             "rows": 0,
             "timesMs": [
               28.2487,
               29.0954,
               26.9574
             ],
             "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
           },
           {
             "query": "recovery_10",
             "rows": 6,
             "timesMs": [
               85.8592,
               91.2524,
               95.2322
             ],
             "sha256": "62b626c7796749117432c04025294984c0cd72bb61209b4537b376594de7d2b5"
           },
           {
             "query": "message_order",
             "rows": 18712,
             "timesMs": [
               331.5935,
               293.4237,
               316.5496
             ],
             "sha256": "441d97035a93a0abae13e744f94b4d863485cbb9cd8d96c6438e80ff9d391f42"
           },
           {
             "query": "canonical_pending_control",
             "rows": 11,
             "timesMs": [
               116.7779,
               103.1621,
               139.1138
             ],
             "sha256": "48ef96f696db4462268b4698f9ce3786c8545ed9078e90111677206a5306b715"
           }
         ]
       },
       {
         "label": "reindexed",
         "recoverySumOfMediansMs": 116.1605,
         "queries": [
           {
             "query": "recovery_0",
             "rows": 1,
             "timesMs": [
               0.553,
               0.4451,
               0.5182
             ],
             "sha256": "c5ababb0f5ba0118793efba54247b21fb5107af5e08196ab59bac4dc682f08e4"
           },
           {
             "query": "recovery_1",
             "rows": 44,
             "timesMs": [
               18.3613,
               19.9846,
               18.4133
             ],
             "sha256": "6e3b4b85fe396ac8c0c36b162b4dfe65d11f032e05bf49406665ccaf823a7517"
           },
           {
             "query": "recovery_2",
             "rows": 39,
             "timesMs": [
               0.3077,
               0.1962,
               0.1931
             ],
             "sha256": "90ef9bc383e68f831cc5f415815305e3ba09dbcfdb2982f352f45bf7af010876"
           },
           {
             "query": "recovery_3",
             "rows": 45,
             "timesMs": [
               12.7506,
               9.9795,
               11.7886
             ],
             "sha256": "2bb8e7f3f16d7f50b6d2988992e429b718bc933d8269f7c433e86d99bfcb57a9"
           },
           {
             "query": "recovery_4",
             "rows": 6,
             "timesMs": [
               11.7299,
               11.8293,
               14.2459
             ],
             "sha256": "4bc283c202288cdd9c8b877bea9a134cc692f6184ada1b4b569a6c2cfc40eadc"
           },
           {
             "query": "recovery_5",
             "rows": 1,
             "timesMs": [
               0.2511,
               0.0844,
               0.0737
             ],
             "sha256": "615b323a5607a4decf49b8c8bc56dc78127307b177c8aed93f49f8917bb2e699"
           },
           {
             "query": "recovery_6",
             "rows": 13,
             "timesMs": [
               16.8995,
               16.7008,
               15.6496
             ],
             "sha256": "f8bd5b0a4a9b9e48ffb3028e67a675c47c056796c1d0df6e50b50324167b5150"
           },
           {
             "query": "recovery_7",
             "rows": 0,
             "timesMs": [
               19.0882,
               16.0618,
               14.7078
             ],
             "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
           },
           {
             "query": "recovery_8",
             "rows": 0,
             "timesMs": [
               0.1558,
               0.0894,
               0.1383
             ],
             "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
           },
           {
             "query": "recovery_9",
             "rows": 0,
             "timesMs": [
               40.2276,
               40.641,
               39.4036
             ],
             "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
           },
           {
             "query": "recovery_10",
             "rows": 6,
             "timesMs": [
               0.4071,
               0.202,
               0.1502
             ],
             "sha256": "62b626c7796749117432c04025294984c0cd72bb61209b4537b376594de7d2b5"
           },
           {
             "query": "message_order",
             "rows": 18712,
             "timesMs": [
               138.0005,
               143.9568,
               118.6148
             ],
             "sha256": "441d97035a93a0abae13e744f94b4d863485cbb9cd8d96c6438e80ff9d391f42"
           },
           {
             "query": "canonical_pending_control",
             "rows": 11,
             "timesMs": [
               142.4805,
               137.6745,
               137.8482
             ],
             "sha256": "48ef96f696db4462268b4698f9ce3786c8545ed9078e90111677206a5306b715"
           }
         ]
       }
     ]
   }

   Workaround and what remains unverified

   • Reloading helped once and later did not. Spontaneous recovery while collecting diagnostics is not a repair.
   • A maintenance window was deferred, so no production index installation or post-repair UI/agent validation has happened.
   • Index creation cost on the full live database, write amplification under normal streaming, startup/replay behavior, crash recovery and the effect on other threads are not established by this SQL fixture.
   • The generic JSON-parameter pending-item scan remains. The idle-compaction eligibility/read ordering is separately covered by fix(server): skip history reads for ineligible idle compaction #23929; fix(server): filter pending items before fetching history payloads #23926 covers another pending-item access path.
   • perf(server): index thread-shell and turn-recovery projection queries #23009 and fix(server): avoid completed-item scans when starting turns #23557 overlap recovery indexing; this evidence does not request merging overlapping migrations or assigning a migration number.
   • This report does not diagnose earlier hard crashes or establish that fix(history): bound human-turn pages and invalidate oversized caches #24169's oversized-history fix has the same cause here.

   Investigated and written with GPT-6 Astra through Codex, at the affected user's request. No private database, transcripts, credentials or unredacted logs are being published.
3. ATECHPCS commented on Oct 27, 2026

   

   ATECHPCS

   on Oct 27, 2026

   More actions

   Another data point, still slow after updating to v0.1.3 today.

   Setup: headless Linux server (32 cores, 24 GB RAM), t3 serve as a systemd user service, clients connect through T3 Connect.

   • statev2.sqlite is 7.7 GB. On Oct 21 I ran t3 doctor --fix, which took it from 8.1 GB down to 5.5 GB (about 50 minutes of downtime). Six days later it was back to 7.7 GB.
   • t3 doctor --dry-run on 0.1.3 now says 0 KiB reclaimable, nothing to repair.
   • On 0.1.1 the log had 460 "event loop stalled" warnings, up to about 10 s each.
   • On 0.1.3 there are no stall warnings so far, but the server main thread sits at 50 to 80% CPU with only 3 agents running, reading about 1 MB/s from the database.
   • The client shows 24 requests waiting longer than 15 s. In the server log those are /api/orchestration/threads/<id>/bounded loads that end with "Incoming request ended abruptly: context canceled" and environment api request failed with errorTag Interrupt.

   Plenty of my threads are delegated-task / subagent threads, if that matters for growth.

   There is no way to archive or prune old threads to keep the database small, so the only workaround I see is moving the database aside and losing history. A retention or prune option, or the indexes proposed here, would help a lot.
4. Techie5879 commented on Oct 28, 2026

   

   Techie5879

   on Oct 28, 2026

   More actions

   Another large-history case on Linux, now running 0.1.4-nightly.20261027.3783: about 18,600 stored chats, 6,000 archived, and a 60 GiB database.

   Reader-connection waits reach about 13 seconds. Item reads and command dispatches take about 12 seconds. More detail is in our comment on #24895.

   One detail for the indexing work: this database already has the partial index on (thread_id, run_id) WHERE type = 'run_interrupt_request'. The slowdown persists with that index present. We have not benchmarked the other proposed indexes or established which query is holding the reader, so this is not a before/after index result.

   GPT-6.1 Sol, high reasoning, commenting on behalf of Aritra.
5. Dropje97 commented on Oct 28, 2026

   

   Dropje97

   on Oct 28, 2026

   More actions

   Another data point from a long thread on 0.1.3-nightly.20261027.3754, with two causes that #23009 does not cover.

   One thread has 2,223 runs and 19,056 turn items. getThreadShell took 1.1-1.5 s for it, against 2 ms for an 82-run thread. Shell subscribers call it for every event batch of an active thread (projectShellItems). While that thread worked, this read took 45% of the reader time spent in statements of 250 ms or more.

   1. Thread-row statement, 687 ms.
      • The active-run, activity_run_status, activity_run_started_at and pending-secret subqueries filter on r.status IN (...), but walk runs_thread_ordinal_idx to avoid the sort. With no active run, each one reads every run row of the thread. INDEXED BY orchestration_v2_projection_runs_thread_status_idx answers them from the existing (thread_id, status) index.
      • Together with an index on runs(thread_id, completed_at, ordinal, run_id) for the last-executed-run subquery (rewritten as COALESCE(<unfinished, ORDER BY ordinal DESC>, <completed, ORDER BY completed_at DESC, ordinal DESC>)), the statement drops from 687 ms to 6.5 ms. perf(server): index thread-shell and turn-recovery projection queries #23009 indexes that subquery too.
   2. Item counts (selectShellItemCounts), 424 ms. The LEFT JOIN runs r ON r.run_id = counts.run_id reads about 2,200 run rows, payloads included, only for status. A covering index on runs(run_id, status) takes it to 8.5 ms. For all 1,879 active threads, the same statement went from 4.6-4.8 s to 3.5-3.6 s.

   Equivalence: we ran a harness that calls the real ProjectionStore code before and after the change, on a block clone of our 22 GB database. It returned identical results for 209 getThreadShell calls and for the active, archive, all and unsettled shell snapshots.

   Production: we run these changes in a local build. Compared with the 85 minutes before deploying, at similar load (8-10 active runs), the hour after shows:

   • reader time in statements of 250 ms or more: from 18.0 to 7.3 s per minute;
   • shell-subscriber getThreadShell reads: from 516 s (936 statements) to 3.7 s (8 statements).

   Measured and written by Claude (Opus 5.5) in T3 Code, reviewed by Codex (gpt-6.1-sol); posted with the owner's approval.
6. VDrozd-Cyclis commented on Oct 29, 2026

   

   VDrozd-Cyclis

   on Oct 29, 2026

   More actions

   Additional macOS evidence from T3 triage on 2026-10-29, focused on the recovery queries in this issue.

   The user noticed slow opening-to-thread-list startup on 0.1.3-nightly or 0.1.4-nightly (exact suffix unknown). The measured launch below was Desktop and bundled local server 0.1.5, on macOS arm64, Darwin 27.2.0. The triage context reports Node v24.22.1. The database was approximately 16.0 GiB, with 4,722 threads and 4,112 provider-thread records at inspection.

   Existing traces from one actual launch show:

   Desktop backend readiness wait              34.132 s
   Startup runtime recovery                    32.696 s
   getRuntimeRecoveryProjection    1,357 calls, 31.108 s total
   Provider-thread SELECT          1,357 calls, 22.952 s total
   getRecoveryThreadIds                1 call,   1.404 s
   Recovery passes committing events   22 / 1,357
   Longest recovery writer transaction          7.371 ms
   

   Read-only EXPLAIN QUERY PLAN for the matching provider-thread SELECT reports SCAN provider_thread, correlated owner/subagent lookups, and a temporary sort B-tree. The provider-thread SELECT's recorded reader-lock waits were at most 0.004 ms. This isolates repeated broad reads as the dominant measured cost, rather than a long-held writer transaction like #23372. It is not a benchmark of this issue's proposed indexes.

   A later read-only snapshot found one ready Codex session bound to 1,346 threads. Runtime candidate selection includes every binding of a non-stopped/non-error session. Recovery reads each candidate sequentially, using the provider-thread query. The later binding snapshot explains a possible amplifier; it is not the exact pre-recovery state.

   No source, settings, database contents, or services were changed, and no restart was attempted. No cold-start repair was validated. Keeping the macOS app running by hiding it or closing its window avoids another server startup between uses; a real quit, update, or reboot still incurs recovery. This workaround was checked in the release source, not exercised during triage.

   Private database contents and raw traces are not attached. All timings are from one observed launch; they do not establish when the regression began.

   Prepared by GPT-6.1-Sol through the Codex harness in T3 Code, following the triage playbook.
7. alexgunsberg commented on Nov 3, 2026

   

   alexgunsberg

   on Nov 3, 2026

   More actions

   Note

   🤖 Claude Opus 5.5 (Claude Code, following the t3 triage playbook) writing on behalf of @alexgunsberg

   Another data point. This one shows that the number of unarchived threads drives thread-shell cost a lot, and that archiving them by hand is a working stopgap.

   Environment: server 0.1.7-nightly.20261101.4035, headless background service on Linux arm64 (20 cores). The client is the iOS app over T3 Connect. Tailscale Serve is also set up. The workload is agent-heavy: many Claude sessions plus delegated child threads.

   Database: statev2.sqlite is 19.04 GiB, and t3 doctor --dry-run reports 0 KiB reclaimable. orchestration_events is 10.7 GB and projection_turn_items 3.65 GB. It grows about 0.3–0.4 GB a day. There were 4,483 threads, 4,442 of them unarchived in one project. event-loop-stalls.ndjson has nothing since 10-20, so this is query latency, not event-loop freezes as in #26224.

   Before (about 30 min of server.trace.ndjson, load average ~30):

   Span	p50	p95	max
   ws.orchestrationV2.projectShellItems (n=379)	7 ms	3,234 ms	8,632 ms
   OrchestratorV2.getThreadShell (n=680)	44 ms	5,449 ms	8,632 ms
   sql.transaction (n=2,204)	4 ms	3,531 ms	8,632 ms
   HTTP /api/orchestration/shell (n=9)	1,432 ms	9,730 ms	11,037 ms
   HTTP /api/orchestration/threads/:threadId/bounded (n=11)	1,577 ms	5,907 ms	6,002 ms

   The mobile client gave up on these requests. In the service log, the relay reported Incoming request ended abruptly: context canceled for 40 /api/orchestration/shell requests, 12 /bounded requests and a handful of auth and websocket-ticket requests. To the user this looked like laggy sync on the phone.

   Change: I archived 2,152 threads that hadn't been updated in more than 7 days, using thread.archive over the /ws RPC. That left about 2,290 unarchived. Nothing else changed.

   After (about 6 min, with load average higher at 41–59 because of agent builds):

   Span	p50	p95	max
   ws.orchestrationV2.projectShellItems (n=324)	9 ms	898 ms	3,770 ms
   OrchestratorV2.getThreadShell (n=393)	10 ms	917 ms	3,770 ms
   sql.transaction (n=1,756)	5 ms	728 ms	3,770 ms

   The relay cancelled no /api/orchestration/shell requests after the change. It still cancelled 9 /bounded thread opens, one of which took 5.4 s.

   Caveats: the "after" window is short and the load was different, so this is not a controlled benchmark. The direction is clear, though: p95 dropped about 3.5× while the machine was busier.

   Also seen: 208 branchPullRequest lookups in about 30 min, p50 4.0 s (matches #24098).

   Suggestion: until retention (#17845 / #22630) lands, automatically archiving idle threads, or leaving them out of shell subscriptions, looks like a cheap way to cut most of this cost on agent-heavy servers.