Background service exhausts V8 heap and aborts after ~1-2 days of continuous uptime under sustained session load #12584

Activity

1. juliusmarminge commented on Sep 19, 2026

   

   juliusmarminge

   on Sep 19, 2026

   Member

   More actions

   Triage

   Confirmed as a real background-service V8 heap abort, not a one-off OOM from a single oversized payload and not a misconfigured unit. Reproduced twice on v0.0.42 (719a76ca1d) with the same main-thread stack. Current main @ dfbb11bdd still has the long-lived retention paths that fit this shape. I did not find a duplicate.

   Related, not the same ticket:

   • [Bug]: Desktop server private memory climbs ~25-30 MB/h while idle (immortal Effect parent spans + always-on tracing) #5410 / open fix(server): detach immortal parent spans from idle background work #9824 — idle desktop Private Bytes from immortal Effect parent spans (~25–30 MB/h with no provider work). That leak is still unfixed on main (forkParked still inherits the caller span). It is a contributor, not this crash: idle growth alone does not reach a V8 ceiling in 30–48h.
   • fix(server): bound provider event log records before serialization #12305 (on main, after 0.0.42) — bound provider event-log serialization.
   • fix(server): release oversized pull request diff cache entries #12523 (on main, after 0.0.42) — drop oversized PR diff cache entries.
   • fix(server): keep tool payloads out of completion queues #12345 — keep tool payloads out of completion queues (merged to t3code/codex-turn-mapping, not on main).
   • fix(server): bound getThreadProjection history with explicit paging #11514 — bound getThreadProjection with paging (open).
   • fix(server): forget a deleted thread's provider binding #8796 — delete provider_session_runtime rows when a thread is deleted (open).

   None of those is a multi-day busy-service OOM under hundreds of live sessions.

   What the report shows

   Two SIGABRTs, both v8::internal::V8::FatalProcessOutOfMemory on the server thread during Heap::CollectGarbage / AllocateRawWithLightRetrySlowPath:

   crash 1   2026-09-18 05:48:00Z   uptime 48.0h   cgroup memory peak 18.4 G
   crash 2   2026-09-19 11:35:36Z   uptime 29.8h   cgroup memory peak 15.4 G
   

   RSS 230 MB after 90 minutes, then fatal after a day-plus. That is retention, not a single huge allocation. The 15–18 GiB cgroup peaks include children (claude, git, extra node). The abort itself is the server process V8 heap.

   Workload matches the expensive path: many long-lived provider sessions, frequent turn churn, lots of git/worktree work, ~16.7k orchestration events, ~430 live session bindings. 13h57m CPU / 29.8h wall on the second window.

   The generated user unit still ships with no heap cap:

   t3code/apps/server/src/cloud/bootService.ts

   Lines 95 to 125 in dfbb11b

   	export function renderBootServiceUnit(plan: BootServicePlan): string {
   	// The user manager has no reliable network-online target; server networking retries itself.
   	return [
   	"[Unit]",
   	"Description=T3 Code server",
   	"StartLimitIntervalSec=300",
   	"StartLimitBurst=5",
   	"",
   	"[Service]",
   	"Type=simple",
   	"WorkingDirectory=%h",
   	`Environment=T3CODE_HOME=${quoteSystemdValue(plan.baseDir)}`,
   	`Environment=${BOOT_SERVICE_UNIT_ENV}=${BOOT_SERVICE_UNIT_FILE}`,
   	`ExecStart=${plan.program.map(quoteSystemdValue).join(" ")}`,
   	// Let the launcher mark an explicit stop before it signals the server.
   	// systemd still SIGKILLs the whole cgroup if graceful shutdown times out.
   	"KillMode=mixed",
   	// Agent tool calls run as children of the server, so they share this cgroup.
   	// With the systemd default of OOMPolicy=stop, the kernel killing one greedy
   	// child stops the whole unit: the server, every live agent, and the user's
   	// connection. Keep running and let Restart=always cover the main process.
   	"OOMPolicy=continue",
   	"Restart=always",
   	"RestartSec=5",
   	`StandardOutput=append:${escapeSystemdSpecifiers(plan.logPath)}`,
   	`StandardError=append:${escapeSystemdSpecifiers(plan.logPath)}`,
   	"",
   	"[Install]",
   	"WantedBy=default.target",
   	"",
   	].join("\n");

   Environment= is only T3CODE_HOME and the unit name. No NODE_OPTIONS, no --max-old-space-size. KillMode=mixed + Restart=always is why recovery is ~25s and why the cgroup teardown SIGKILLs in-flight provider/git children. OOMPolicy=continue only covers kernel OOM of a child, not a V8 abort of the main process.

   What the process still retains

   Several independent sites can grow for the life of t3 serve. Together they fit “small at 90 minutes, dead after 1–2 days” on a busy orchestration host.

   1. Adapter session maps keep every completed turn.

   Claude’s in-memory session is a process-lifetime Map, and each finished turn is appended — not replaced:

       context.turns.push({
         id: turnState.turnId,
         items: [...turnState.items],
       });

   items is the turn’s tool/content payload. The array is only rewritten on rollback. Codex / Cursor / OpenCode / Grok / Antigravity keep the same kind of Map<ThreadId, …SessionContext>. stopSession deletes the entry; if the session is still considered live, the history stays.

   2. The inactivity reaper will not touch a “busy” or background-live session.

   t3code/apps/server/src/provider/Layers/ProviderSessionReaper.ts

   Lines 42 to 95 in dfbb11b

   	for (const binding of bindings) {
   	if (binding.status === "stopped") {
   	continue;
   	}
   	const lastSeenMs = Date.parse(binding.lastSeenAt);
   	if (Number.isNaN(lastSeenMs)) {
   	yield* Effect.logWarning("provider.session.reaper.invalid-last-seen", {
   	threadId: binding.threadId,
   	provider: binding.provider,
   	lastSeenAt: binding.lastSeenAt,
   	});
   	continue;
   	}
   	if (now - lastSeenMs < inactivityThresholdMs) {
   	continue;
   	}
   	const thread = yield* projectionSnapshotQuery
   	.getThreadShellById(binding.threadId)
   	.pipe(Effect.map(Option.getOrUndefined));
   	// Ingestion updates this timestamp alongside activeTurnId when a turn
   	// settles. Long turns must get a full idle window after that transition,
   	// even though the binding was last touched when the turn was sent.
   	const lastActivityMs = Math.max(
   	lastSeenMs,
   	Date.parse(thread?.session?.updatedAt ?? binding.lastSeenAt),
   	);
   	const idleDurationMs = now - lastActivityMs;
   	if (idleDurationMs < inactivityThresholdMs) {
   	continue;
   	}
   	if (thread?.session?.activeTurnId != null) {
   	yield* Effect.logDebug("provider.session.reaper.skipped-active-turn", {
   	threadId: binding.threadId,
   	activeTurnId: thread.session.activeTurnId,
   	idleDurationMs,
   	});
   	continue;
   	}
   	// The turn can settle while background work runs on (subagent
   	// fleets, workflow runs, Monitor watch loops). Those live inside the
   	// provider process, so stopping the session would kill them silently,
   	// and nothing bumps lastSeenAt between turns.
   	if (thread?.backgroundLiveness != null) {
   	yield* Effect.logDebug("provider.session.reaper.skipped-background-work", {
   	threadId: binding.threadId,
   	backgroundLiveness: thread.backgroundLiveness,
   	idleDurationMs,
   	});
   	continue;
   	}

   Default idle window is 30 minutes. It skips stopped, anything with activeTurnId, and anything with backgroundLiveness (working / monitoring — subagent fleets, workflow runs, Monitor loops). On this host that is the intended workload, so ~430 live bindings can be real in-process sessions, not leftover SQLite rows.

   ThreadBackgroundLiveness is in-memory and only clears on task.completed (terminal status) or session.exited. A monitor/subagent that never emits a terminal event keeps the session off-limits to the reaper.

   Open #8796 is the other half: deleted threads stop the session but leave the provider_session_runtime row. That inflates the binding count; it does not by itself keep adapter maps warm.

   3. Live event / worker queues are unbounded.

   Every adapter publishes on Queue.unbounded<ProviderRuntimeEvent>(). Ingestion, checkpoint, PR, and command reactors use makeDrainableWorker (TxQueue.unbounded). A slow git/diff/ingestion worker holds every queued event, including diffs and tool payloads, until it drains. #12345 is the same class of bug and is not on main.

   4. Immortal parent spans (#5410) are still on main.

   forkParked still does Effect.forkScoped(effect) with the caller’s span:

   t3code/apps/server/src/serverActivation.ts

   Lines 12 to 26 in dfbb11b

   	export const forkParked = <A, E, R>(
   	effect: Effect.Effect<A, E, R>,
   	): Effect.Effect<void, never, Scope.Scope | R> =>
   	Effect.gen(function* () {
   	const activation = yield* ServerActivation;
   	if (activation === undefined) {
   	yield* Effect.forkScoped(effect);
   	return;
   	}
   	const parked = yield* Deferred.make<void>();
   	yield* Effect.forkScoped(
   	Deferred.succeed(parked, undefined).pipe(Effect.andThen(activation), Effect.andThen(effect)),
   	);
   	yield* Deferred.await(parked);
   	});

   Local file tracing is always on (server.trace.ndjson, Info + timing). Under idle load this was 25–30 MB/h; under this host’s turn/git churn it will be worse. #9824 is the proposed detach. Not merged.

   5. Full thread projections are still unbounded (#11514).

   orchestration.getThreadProjection still materializes every turn/payload/checkpoint for a long thread. A reconnect or snapshot refresh of many busy threads is a heap spike on top of the leak.

   16.7k orchestration events in SQLite is not large by itself. It only hurts if those rows (or their payloads) are decoded and held — replay, projection, live catch-up, or the completion-queue path above.

   What this is not

   • Not kernel OOM of a child. The stack is V8 FatalProcessOutOfMemory on the server thread.
   • Not “raise the heap and we are done.” --max-old-space-size only delays the abort. The generated unit also has no place to put it without a systemd drop-in.
   • Not already fixed in a release newer than 0.0.42. fix(server): bound provider event log records before serialization #12305 / fix(server): release oversized pull request diff cache entries #12523 help log/cache spikes; they do not bound session maps, reaper skips, unbounded workers, or parent spans.

   Suggested direction

   Do not start with a larger heap. Bound the process-lifetime sets:

   1. Drop completed-turn payloads from adapter session maps (or keep a small rolling window). Resume/rollback should read history from the provider store, not from an ever-growing turns[].
   2. Land fix(server): detach immortal parent spans from idle background work #9824 so parked reactors do not pin startup spans for the life of the service.
   3. Bound the runtime-event / DrainableWorker queues (drop or coalesce when the consumer is behind; bring fix(server): keep tool payloads out of completion queues #12345 onto main).
   4. Land fix(server): bound getThreadProjection history with explicit paging #11514 so projection reads cannot materialize a whole thread.
   5. Land fix(server): forget a deleted thread's provider binding #8796 so deleted threads do not accumulate bindings. Separately: sample provider.session.reaper.skipped-background-work / skipped-active-turn on this class of host — stuck liveness is how 430 “live” sessions stay in the Map.

   Graceful restart is a follow-up, not the fix. KillMode=mixed will still SIGKILL the cgroup if the main process aborts.

   What would help on the next occurrence

   The offer of --heapsnapshot-near-heap-limit=1 is the right next artifact. Also useful, if cheap:

   • process.memoryUsage() / v8.getHeapStatistics() on a 15–30 min timer (heapUsed vs external vs rss).
   • Counts: provider_session_runtime by status, adapter.listSessions() vs that table, and reaper skip reasons from the server log.
   • Whether server.trace.ndjson is rotating (traceMaxBytes is 10 MiB) or the file is huge.

   Raising the heap via a systemd drop-in is a valid delay, not a diagnosis:

   # ~/.config/systemd/user/t3code.service.d/heap.conf
   [Service]
   Environment=NODE_OPTIONS=--max-old-space-size=8192 --heapsnapshot-near-heap-limit=1
   

   Then systemctl --user daemon-reload && systemctl --user restart <unit>. Snapshots land in the service cwd (%h). Redact before attaching.

   No workaround is required for the reporter — Restart=always is already holding. Filing this is correct; a long-running t3 serve should not hit the V8 ceiling under normal sustained use.
2. juliusmarminge

   added

   acceptedfeature request acceptedfeature request accepted

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Sep 19, 2026
3. OscarsOrbit commented on Oct 15, 2026

   

   OscarsOrbit

   on Oct 15, 2026

   More actions

   Data from another long-running host, from T3's own trace spans and a source read. No heap contents are shared.

   Runtime

   • 0.0.46-nightly.20261008.2833 (a6ec88f), SEA linux-x64, Node 26.8.2, V8 14.6.202.34, in a container. Our build carries a small local patch (MCP wiring and provider instructions only); it touches none of the files named below.
   • Since a graceful restart on 2026-10-15 at 08:00 UTC: NODE_OPTIONS=--max-old-space-size=12288 --heapsnapshot-near-heap-limit=1 --diagnostic-dir=<dir>. RSS was about 1.0 GiB shortly after.
   • At our next rebuild we move to a current nightly, which includes perf(shared): cached SQLite statements no longer hold large parameters #19305, and raise the cap to about 20 GiB.

   What we saw

   • Snapshot residue (an earlier process). One snapshot from the in-app SIGUSR2 handler at RSS 2926 MiB. The server.heapSnapshot span took 8.3 s and succeeded, writing a 267 MB file. The next server.eventLoop.stall span, 23 s later, had rssMb 4337. RSS settled near 5.5 GiB (high-water mark about 5.9 GiB) and stayed there. Stalls of 3-5 s recurred from then on.
   • Slope (a clean process, 6 GiB cap, no snapshot). A sawtooth of about ±500 MB of anonymous memory. The troughs rose about 0.3 GiB/h; trough-to-peak was up to about 0.9 GiB/h. RSS alone cannot separate retained objects from heap headroom.

   Source note

   v8.writeHeapSnapshot leaves V8's HeapObjectsMap allocated and object-move tracking on for the life of the process. HeapProfiler::TakeSnapshot sets is_tracking_object_moves_ = true (deps/v8/src/profiler/heap-profiler.cc:167 at nodejs/node v26.8.2, f2f2c2f2). Only ClearHeapObjectMap (:344-353) turns it off, and Node's writeHeapSnapshot path never calls it. Every later scavenge and compaction then reports each moved object to the profiler. So the SIGUSR2 handler (apps/server/src/observability/HeapSnapshot.ts:24, :46-49 at 26fffe5) can permanently raise RSS and per-GC cost on a long-running server. This is from source; we have not measured how much of our rise is the map and how much is the allocator keeping memory.

   Suggestions

   1. Take the in-app snapshot through an in-process node:inspector Session: HeapProfiler.takeHeapSnapshot, then HeapProfiler.disable, which calls ClearObjectIds (deps/v8/src/inspector/v8-heap-profiler-agent-impl.cc:333-341, same node commit). Otherwise, document that the residue lasts until restart.
   2. Put heapUsed, heapTotal, external, arrayBuffers and the heap limit on the server.eventLoop.stall span next to rssMb (apps/server/src/observability/EventLoopMonitor.ts:146, :151 at 26fffe5). That is the triage ask's data, taken at the moment that matters, and it separates GC-near-limit stalls from sync-I/O stalls without a snapshot.

   Disk

   Disk, not heap. Our statev2.sqlite is 10.24 GB; orchestration_events is 6.48 GB of it (63%) over about 1.70M rows, and orchestration_v2_events is empty. Since migration 055 orchestration_events also holds every v2 event, so we cannot yet say how much is v1 history. compactEventStore (apps/server/src/orchestration-v2/ProjectionMaintenance.ts:242 at 5a4e998) has no caller outside tests, which matches #18559.

   Offer

   We will take no further manual snapshot. If the near-limit snapshot fires, we will post a summary here: retainer classes and their sizes, never heap contents.
4. juliusmarminge commented on Oct 22, 2026

   

   juliusmarminge

   on Oct 22, 2026 – with GitCafe Production

   Member

   More actions

   Checked current main 89fe509232 on Linux x86_64 / Node 24.21.0 with the real web/server dev app and isolated copied state (3 stopped threads, 341 events). Browser-driven reads completed 200 shell/empty-thread requests and 200 populated-thread requests, all HTTP 200, without a V8 abort. This is a short read-load check, not a reproduction of 430 live sessions over 30–48 hours; it does not establish that the bug is fixed.

   Runtime transcript, commands, results and limitations.

   No matching duplicate or existing fix PR found. Main includes #19305, but its native SQLite retention fix does not establish a fix for this V8 heap abort.

   Does this still occur on a current nightly? Please include the exact build, provider mix, approximate concurrent active sessions and turns/hour, and whether growth occurs while idle too. For a recurrence, a summary of heapUsed/heapTotal/external/arrayBuffers versus RSS and the largest retained object classes/retainer paths would distinguish the suspected causes. Please do not post raw heap snapshots; they can contain conversation content and secrets. The follow-up host report already notes snapshot-related overhead, so no additional manual snapshot is requested.
5. juliusmarminge

   added

   needs more infoInitial triage showed no bug. Awaiting more infoInitial triage showed no bug. Awaiting more info

   on Oct 22, 2026
6. athammer commented on Oct 27, 2026

   

   athammer

   on Oct 27, 2026

   More actions

   Still happening on t3 v0.1.0: SEA build dated Oct 15, linux-x64, Node v26.8.2, bundling effect 4.0.4, headless user service. I haven't tried a current nightly yet. There have been five V8 heap aborts since Oct 24, and I have a reproduction for what looks like the main cause on this host: a request whose client disconnects early keeps its whole encoded JSON response in memory.

   Process start	Uptime at abort
   Oct 18	161.7 h
   Oct 24	14.9 h
   Oct 25	9.2 h
   Oct 25	13.4 h
   Oct 26	10.2 h

   At every abort, the last full Mark-Compact freed almost nothing (for example 4091.1 -> 4088.0 MB) against the default 4,192 MB limit.

   Heap breakdown on the live server

   I collected these through the inspector: process.memoryUsage() and V8 heap space statistics every minute. No snapshot was taken.

   Uptime	heapUsed	heapTotal	large_object_space	old_space	external	arrayBuffers	RSS
   64 min	2,265 MB	2,345 MB	1,978 MB	235 MB	1,131 MB	1,110 MB	3,547 MB
   94 min	2,704 MB	2,799 MB	2,383 MB	277 MB	1,361 MB	1,339 MB	3,058 MB
   154 min	3,236 MB	3,438 MB	2,967 MB	230 MB	1,693 MB	1,665 MB	3,066 MB

   Almost all of the growth is in large_object_space (7-13 MB/min) and ArrayBuffers (4-8 MB/min); old space stays flat at about 230 MB.

   The sampling heap profiler (128 KiB interval) attributes 802 of 909 MB (88%) of still-live sampled allocations over its first hour to the JSON encoder in effect@4.0.4 http-api/HttpApiBuilder.js, getResponseEncode, which calls JSON.stringify(e) and then text(...). That is the allocation site, not a retainer path. I didn't take a heap snapshot, so what keeps these objects reachable is still open.

   Reproduction

   I forced a full GC (HeapProfiler.collectGarbage) before and after each step:

   Step	large_object_space	arrayBuffers
   One minute with no extra requests	+2 MB	+1 MB
   One completed GET /api/orchestration/shell (5 MB body, 43.6 s)	+1 MB	0
   The same request, aborted by the client after 2 s	+14 MB	+7 MB
   Two more aborted requests, after 1 s and 5 s	+31 MB	+16 MB

   Completed responses are released. For aborted requests, the string from JSON.stringify and its UTF-8 buffer stay reachable, and they were still held after later GCs.

   What it isn't

   I couldn't reproduce it in a standalone Effect 4.0.4 server. The test served a ~6 MB JSON body after an uninterruptible 1.5 s delay and aborted 10 requests at 500 ms. I ran it with a plain router and with HttpApi schema responses, with and without a recording tracer, and with the global compression middleware T3 installs, on Node 24 and on Node 26. Every variant released everything after GC.

   So the retention looks specific to T3's server layers around these endpoints, rather than to Effect's HTTP stack. Candidates are the auth middleware, annotateEnvironmentRequest, traceLocalHandlerWork and the trace exporter. I haven't narrowed it further.

   Why it compounds

   /api/orchestration/shell returns every unarchived thread. I have 1,744, of which only 9 are archived. It took a median of 6.4 s earlier, and 44 s once the heap was large.

   1. Clients with timeouts abandon most calls; mine used 8 s and 15 s.
   2. Each abandoned call leaks about 20 MB.
   3. The heap grows, so GC and the endpoint both slow down.
   4. Slower responses mean more calls are abandoned, and the cycle repeats.

   That matches my time-to-abort dropping from 161.7 h, with about a dozen threads, to 9-15 h once I had 1,100+ threads.

   Answers to the questions above

   • Build: t3 v0.1.0 (Oct 15 build, effect 4.0.4).
   • Current nightly: not tested. main now uses effect 4.0.5, but its Node HTTP server and response changes are only URL redaction in logs, so the client-disconnect path looks unchanged.
   • Providers: Claude Code (through a local API gateway) and Codex.
   • Load: in the 10 h before the last abort, 478 turns (average 48/h, peak 122/h) and a peak of 21 concurrent runs.
   • Idle growth: not measured, because the server was never idle.
   • Retained objects: large strings and their UTF-8 ArrayBuffer backing stores from response encoding, almost all in large_object_space. Retainer paths are unknown.
   • Snapshots: no raw heap snapshot was taken or shared.

   Probably separate, from T3's own trace spans

   • A repository and pull request discovery sweep runs every 2-4 minutes. Each pass is about 13,200 spans and 4,100 git commands across 542 worktrees.
   • Two traces never end and keep gaining child spans: one at about 1,400/min (Codex provider retry and SQLite) and one at about 575/min (ThreadLiveEventCoalescer).

   On my side, I'm removing the client timeouts and most of my shell calls, which should stop the growth for me. The server-side retention on client disconnect would remain.
7. juliusmarminge

   removed

   needs more infoInitial triage showed no bug. Awaiting more infoInitial triage showed no bug. Awaiting more info

   on Oct 27, 2026
8. juliusmarminge commented on Oct 27, 2026

   

   juliusmarminge

   on Oct 27, 2026 – with GitCafe Production

   Member

   More actions

   Thanks for the abort reproduction. I followed it on current main 68339a0846 (effect 4.0.5) and on your build, v0.1.0 81493efb71 (effect 4.0.4). I ran v0.1.0 under both Node 24.22.1 and Node 26.8.2. I could not reproduce the retention.

   Setup: t3 serve from source on a fresh base dir with --inspect, 1 project, and 1,744 unarchived threads created through the server. GET /api/orchestration/shell returned 2.6–2.8 MB in 0.3–0.4 s, with a bearer token from t3 auth session issue. I forced a full GC (HeapProfiler.collectGarbage, twice) before each reading.

   Step	main, Node 24	v0.1.0, Node 24	v0.1.0, Node 26.8.2
   20–40 completed requests	+0.3 MB large objects	0	+2.3 MB, released by the next reading
   20–40 aborted before the response (50–300 ms)	0	0	0
   20–40 dropped mid-body (raw socket)	0	0	0
   160 aborted at 1 s, 16 concurrent, so requests queue	0	—	0
   The same with accept-encoding: gzip, br (br response)	0	—	0 (96 at 200 ms)

   arrayBuffers stayed at 0.5 MB after every abort phase. Transcript, drivers: seed-threads.mjs, measure-aborts.mjs, measure-bursts.mjs, issue-mcp-token.ts.

   So the path that keeps the response alive is probably something on your host that this setup lacks. To close the gap:

   1. How the aborting clients reach the server. Loopback, LAN or tailnet IP, Tailscale Serve, or T3 Connect. Also their auth: a bearer from t3 auth session issue, a browser session cookie, or a pairing token. And the client itself (curl, fetch, your own tool), with the exact command or code that aborts.
   2. Telemetry. Is OTLP trace or metrics export configured (the telemetry export settings, or T3CODE_OTLP_TRACES_URL / T3CODE_OTLP_METRICS_URL / T3CODE_OTLP_LOGS_URL)? If so, roughly what endpoint (local collector or remote)?
   3. One retainer path. On a test process, not the live one: take one heap snapshot after a few aborted requests and open it in Chrome DevTools. Pick one of the large retained strings (~5 MB) and give the class and property names along its Retainers chain up to the first GC root. Names only, no contents. That chain is what tells us which layer holds it.
   4. A fresh base dir. Does it still reproduce with t3 serve --base-dir <empty dir> --port <free port> on the same host and binary, with your aborting client pointed at that server?

   The PR discovery sweep and the two never-ending traces you list look like separate issues. #25172 already tracks the PR sync sweep's memory growth.
9. juliusmarminge

   added

   needs more infoInitial triage showed no bug. Awaiting more infoInitial triage showed no bug. Awaiting more info

   on Oct 27, 2026
10. athammer commented on Oct 27, 2026

    

    athammer

    on Oct 27, 2026

    Last edited by athammer

    More actions

    Thanks for running it. I found the retainer, and I think it explains why your runs stayed clean. The leak needs the client's disconnect to be noticed after the handler has returned, while the response is being compressed and sent. Every abort in your runs landed while the handler was still running, so the fiber was interrupted cleanly and answered 499.

    Retainer path, from the live process

    I read this from the running v0.1.0 server through the inspector, not a snapshot. I used Runtime.queryObjects on ServerResponse, then the closure of makeHandler's close listener to reach each request fiber. Names and sizes only.

    After a forced GC, 9 ServerResponses were alive with writableEnded: true, writableFinished: false and destroyed: true:

    • 5 × GET /api/orchestration/threads/:threadId:
      • status 200, content-encoding: gzip, content-length about 7.3 MB;
      • none of the body reached the socket, and the socket closed without an error;
      • one end callback is still queued;
      • the request fiber isn't done, has interruptible: false with an interrupt pending, and is suspended on that callback under toHandled's sendResponse continuation.
    • 4 × GET /api/orchestration/shell with status 499: their fibers completed. I don't know what keeps these four response objects reachable, and they don't hold a body.

    The chain for the 200s:

    1. The server scope. makeHandler runs each request with Fiber.runIn(..., options.scope).
    2. The request fiber.
    3. sendResponse's mapEager continuation, which closes over the original, uncompressed response.
    4. Its Uint8Array body: a 54.4M-character JSON string and its 54.4 MB of UTF-8 bytes. The 7.3 MB gzip output is held by the pending end call.

    That's about 116 MB per stuck request, which matches the large_object_space and arrayBuffers growth in my earlier comment.

    Why it happens

    In effect 4.0.4, toHandled sends the response uninterruptibly, including pre-response handlers such as the compression middleware. Suppose the socket's close is handled in that window:

    1. makeHandler requests an interrupt, which is deferred.
    2. handleResponse calls nodeResponse.end(body, cb) on the destroyed response. That's the Uint8Array branch for 1 MB or more, or Raw.
    3. A destroyed response never emits finish, so Node never calls cb, and the fiber waits forever.

    A disconnect during the synchronous end of the handler (JSON.stringify of the full snapshot) is only noticed at the next I/O turn, and that is the async zlib call. Here the responses are 54 MB before compression, on a busy event loop, so the window lasts seconds and my client's 15 s timeout lands in it. Your shell responses were 2.7 MB in 0.3 s, so the window was a few milliseconds.

    I've opened a fix in Effect: Effect-TS/effect#10004. writeResponse now skips the write and releases the body when the response is already destroyed. Writes that wait on end's callback also resume when the response closes, which covers a disconnect during the send itself. Its deterministic test is an uninterruptible route that responds after the client's socket closes. On Effect's main, the Raw case hangs and the Uint8Array case fails with a content-length mismatch. T3 would pick up the fix with an Effect bump.

    To reproduce it in T3, request a thread whose snapshot is tens of MB with GET /api/orchestration/threads/:threadId and accept-encoding: gzip. Time a completed request first, then abort a little before the response would arrive. The standalone version is the test in Effect-TS/effect#10004.

    Your questions

    1. Clients: loopback 127.0.0.1:3773, HTTP/1.1. The client is Node fetch (undici, accept-encoding: gzip, deflate) with AbortSignal.timeout(15000). Auth is a bearer token obtained by exchanging a t3 auth pairing create credential.
    2. Telemetry: no OTLP export. None of the T3CODE_OTLP_* or OTEL_EXPORTER_OTLP_* variables is set, and there are no telemetry settings. Only the local server.trace.ndjson is written.
    3. Retainer path: above, from the live process instead of a test process.
    4. Fresh base dir: not tried. The retention depends on response size and timing rather than base-dir state, and the Effect test reproduces it without T3.

    Separately, my client was requesting /api/orchestration/threads/:threadId?turnLimit=1. That route has no query schema, so turnLimit is ignored and the full snapshot comes back, which is why these responses were 54 MB.
11. juliusmarminge

    removed

    needs more infoInitial triage showed no bug. Awaiting more infoInitial triage showed no bug. Awaiting more info

    on Oct 27, 2026