[Bug]: Effect span exporter writes to disk unconditionally with no way to disable it

[morgaesis]

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

1. Start the server with t3 serve.
2. Run an interactive session that produces terminal output.
3. Inspect ~/.t3/userdata/logs/server.trace.ndjson and its rotated siblings.

Expected behavior

Span export to disk is off by default in a long-running server, or can be disabled through a documented flag or environment variable.

Actual behavior

The server writes Effect tracing spans to ~/.t3/userdata/logs/server.trace.ndjson for the whole life of the process. The file rotates through 11 files of 10 MB, a new one roughly every 12 minutes under ordinary use, indefinitely.

Records are marked "sampled":true with no sampling reduction, and spans are emitted per terminal event, so the volume tracks terminal output rather than anything the operator configures.

There is no supported way to turn it off:

• T3CODE_TRACE_FILE changes the path but does not disable the exporter, and the directory creation for it runs unconditionally.
• T3CODE_LOG_LEVEL defaults to Info and governs the logger, not this file.
• t3 serve --help exposes no tracing or verbosity control.

Impact

Major degradation or frequent failure

Version or commit

0.0.34 and 0.0.36

Environment

Platform-independent; reproduced on Linux with the published npm releases.

Logs or stack traces

{"type":"effect-span","name":"terminal.queuePersist","sampled":true,...}
{"type":"effect-span","name":"terminal.drainProcessEvents","sampled":true,...}

Workaround

Point T3CODE_TRACE_FILE at a path on a tmpfs, so the writes never reach persistent storage.

Related

• [Bug]: terminal history can grow without a byte bound and repeatedly rewrites the full log #7991 overlaps: the spans above come from the terminal persistence path it describes. That issue is about the terminal history file growing and being rewritten; this one is about the span exporter being unconditional, which stands regardless of how history persistence is bounded.
• [Bug]: Remote server updates accumulate every immutable runtime and consume disk space #8345 is the same theme of unbounded disk growth from a different cause.

[SkiTee3000]

Numbers from a Windows desktop install, 0.0.41-nightly.20260908.1414, 12 projects, idle: ~200 MB of server.trace.ndjson* per day.

• 88% of the bytes come from the two one-minute background sweeps (ThreadPullRequestReactor.synchronize, ThreadSettlementReactor.sweep; Thread PR discovery and settlement sweeps run every minute without client demand, and their caches expire before the next sweep #11220).
• Almost all of the rest is empty spans that carry no information: PortDiscovery.pollTick every 3 s (1514 spans in a 76-minute file; the probe itself is [Bug]: Windows port discovery re-spawns a doomed PowerShell/WMI probe every 3 s (the port-scanner half of #4182, not fixed by #2679) #5900), ServerSecretStore.get every 5 s (909), and one createTrace2Monitor per git command (1407).
• With the default rotation of 10 × 10 MiB that is about 13 h of history, so the trace cannot be used to look at anything older than half a day.
• The CLI reads T3CODE_TRACE_MAX_BYTES, T3CODE_TRACE_MAX_FILES, and T3CODE_TRACE_MIN_LEVEL (apps/server/src/cli/config.ts:80-88), which the report above does not mention. There is still no off switch, and the desktop app exposes none of them.
• For diagnosis the spans are also too thin: runGitCommand / processRunner.runProcessCore record no cwd or args, so the trace cannot show which command in which repository hung or failed, and the Windows taskkill runs from [Bug]: Frequent cmd.exe / conhost flashes on Windows from provider probe & VCS process kill paths #2537 do not appear at all.

[bompus]

Another data point, from a 21-minute window of a Linux/WSL desktop install (nightly 0.0.46-20261011, about seven agents streaming, 212,821 spans, 111 MB). The 10 × 10 MiB window covered about 20 minutes.

By bytes: database spans (sqlite, sql.transaction, EventSink.write) were 48.5%, git/process spans 17%, project and PR lookups 13%, ThreadLiveEventCoalescer 4%, other 17%. StorageCleanup.sweep descendants were 11% and PR polling about 16%, so unlike the earlier numbers here the sweeps weren't the main source on this install; streaming agent events were. Each event write is one sql.transaction plus about five sqlite spans (about 4.9 KB of trace per write in a local benchmark).

Tracing cost on that path measured about 0.1 ms per write, around 0.05% of a core, so this is about disk and history length, not CPU.

I opened a PR for this: #18106. It stops tracing the statements inside event-write transactions (the transaction span still times each write), keeps startup attributes off the effect worker's spans, and untraces the three ProjectEnrichmentService cache helpers. In the same sample that is about 30% fewer bytes. Per CONTRIBUTING.md, the first drops per-statement timing, so I'd like a maintainer's view on that direction; happy to split the commits if you'd rather have them separately.

[josephv123]

Current-nightly trace-retention data, consolidated from three audits rather than posting each measurement separately. Installed build was 0.0.46-nightly.20261014.3132.

Linux captured approximately 112.6 MB / 256k spans in 2.75 hours (about 41 MB/hour), so the default rotating files retained only a few hours of diagnostic history. By serialized span bytes in that window:

• sqlite: about 23.9 MB (21%), including SQL text.
• ThreadSettlementServiceV2.settleThread: about 21.6 MB (19%); approximately 56.8k spans with roughly 0.02 ms average work.
• sql.transaction: about 4.7 MB.
• Workspace path, favicon and project enrichment calls supplied much of the remaining small-span volume.

The exact split shifts with workload and capture cutoff; these are not additive categories across machines. Hidden child candidates explain a large part of settlement volume on this host (151 of 183 candidates; #19657).

The Mac desktop capture also showed repeated topology sendSync polling behind about 89% of desktop spans, although the handlers were cheap on that machine. That evidence belongs to #12689 as well.

No material tracing CPU stall was established. The demonstrated problem is short useful diagnostic retention from large numbers of cheap/repeated spans. Normal Mac sleep/wake reconnects and machine-wide swap pressure were not counted as T3 regressions.