Repository navigation
docs build: next build holds ~7.6 GB in ONE process — the docs deploy's next memory ceiling, and no in-repo knob bounds it #12683
Description
Found while fixing #12677 (spec's DTS pass was declaring a 12 GB heap ceiling inside the build container). That fix lands separately and is not affected by this card; this records the next thing that will take the docs deploy down, because it is now the largest single consumer in the pipeline and nothing in this repo bounds it.
Measured
pnpm turbo run build --filter=@objectstack/docs is two sequential tasks — @objectstack/spec#build then @objectstack/docs#build (the docs app's only workspace dependency is spec). Measured as peak anonymous RSS of the whole process tree, inside a cgroup capped at the stated size and a mount namespace whose /proc/meminfo reports the same figure, so tools that size themselves from reported memory see a real container:
| phase | peak | dominant process |
|---|---|---|
@objectstack/spec#build, after the #12677 fix |
5654 MB | tsup DTS worker |
@objectstack/docs#build |
~7.6 GB | a single next-build (v16.3.1) process at 7592 MB |
The whole pipeline in an 8192 MB container: exit 137, oom_kill=1, peak 8143 MB, killed in the docs phase. In a 10240 MB container: completes, 2/2 tasks, 5m02s, peak 8135 MB.
The turbopack worker-pool children are not the problem — they measure 100–430 MB each. One process holds everything.
Why the existing controls do not bound it
Both knobs already in apps/docs/next.config.mjs and apps/docs/package.json were put there for earlier OOMs, and neither reaches this:
NODE_OPTIONS='--max-old-space-size=4096'bounds V8's old space only. Next 16 builds with Turbopack, which is Rust — its allocations sit outside V8 entirely. Measured: the process reaches 7592 MB while nominally capped at 4096.experimental.cpus: 2bounds the static-generation worker fan-out. Measured above: the workers are small; capping them cannot move a 7.6 GB main process.
And the demand is fixed, not adaptive — it does not shrink when given less. Under smaller containers the docs build grows until it is killed rather than sizing down:
| container | result |
|---|---|
| 8192 MB | exit 137, oom_kill=1, limit_failcnt=11073 |
| 6144 MB | exit 137, oom_kill=1, killed at 75s |
| 4096 MB | exit 137, oom_kill=1, killed at 34s |
Next 16.3.1 exposes no memory-limit option: its config schema carries no memoryLimit for turbopack (the only match anywhere in its dist/ is TURBO_TRACE_DEFAULT_MEMORY_LIMIT, which is tracing, not the build).
Why this is not already an outage
The docs phase has been building successfully on Vercel at roughly this size for weeks — the experimental.cpus: 2 comment records a previous OOM fought at this same seam. So the Vercel build container's real envelope is at or above ~8 GB usable. What broke in #12677 was the other phase crossing the line, not this one. This card is the standing risk, not a live failure.
Why it is a real risk anyway
The margin is thin and only one side of it is under this repo's control. The docs site prerenders 400+ MDX pages; that count grows with the documentation. When this phase crosses, it produces exactly the #12677 signature — exit 137, no diagnostic, every production deploy failing — and unlike #12677 there is no in-repo number to lower.
Options, none of them free
- Raise the Vercel build container (dashboard/maintainer action, not versioned in
apps/docs/vercel.json). Buys headroom, changes no code, costs money. - Build the docs app with webpack instead of Turbopack. Different memory profile, but it is a bundler change for a 400-page site — an architecture decision, and
next.config.mjsdocuments that Vercel and CI both currently run Turbopack. - Shard the prerender so no single process holds the whole page set. Largest change; also the only one that scales with the page count.
Not filed with a fix attached on purpose: (1) is a maintainer decision and (2)/(3) are architecture decisions, none of which belongs inside the #12677 memory fix.
Reproducing
Peak was read from a cgroup-v1 memory.stat total_rss sampler with per-process attribution at each new peak; the harness was validated against a positive control (600 MB allocation under an 8 GB cap completes) and a negative control (2 GB allocation under a 1 GB cap reproduces exit 137 with oom_kill=1). The container emulation is two halves — the cgroup cap and a faked /proc/meminfo in a private mount namespace; a cap alone is not a container, because it bounds what a process may use without changing what it believes it has.
Generated by Claude Code
Activity
os-zhuang commentedon Aug 27, 2026
① 项目长远合理性:C(预渲染分片)是唯一随页面数增长而扩展的形状;A(调大构建机)只买间距不改曲线;B(换 webpack)换了打包器也没换增长曲线,还引入一次 400 页站点的打包器迁移。⚠️ 检测延迟有实测:#12333 这次断流两天才被发现。
② 实际业务拉动:零当前故障——该阶段数周来一直构建成功;但实测间距 ~0.5 GB(8192 容器内峰值 8135 MB),且每新增一页文档都在消耗它,一次大型文档落地即可能触线。
③ 防 AI 犯错:触线时的形态 = exit 137 无诊断、全部生产部署静默失败——与 #12677 同签名,但这次仓库里没有可降的数字;卡内已刻好指路(「再见 137 先查本卡」),响亮性只能靠记录补。
④ 创业阶段不扩散:A 零代码零新义务;B/C 都是架构级手术——默认不为尚未发生的故障动刀。
推荐:A(维护者面板动作,分钟级)现在做,C 记录为页面数真正增长后的既定方向;B 排除。备选 D(记录后 defer,什么都不做):真实代价 = 某天 docs 生产部署再次全停,以实测两天的检测延迟计,停机窗口可能又是「悄悄的两天」,修复仍是去面板点 A。
一行置信缺口:本分析看不见 Vercel 的真实容器规格与价格档(标准/增强构建机的内存与月费),也看不见未来文档页的增长速率。
Generated by Claude Code
os-litant commentedon Aug 27, 2026
Deferred by ruling — not adjudicated this round. Director seat, batch #1, 2026-08-27 (session
session_01MnijPVVDakqK2J335JoJtq). Maintainer's one-line ruling on the batch, verbatim: 「12683 本轮不处理，其他同意」.Card stays parked in the decision inbox (
needs-user-decision, labels untouched) and rolls into the next Director batch. No option is chosen; the standing-risk analysis and the option set (A raise the Vercel build container / B webpack / C shard the prerender / D defer) remain as presented. Related but independent: #12698 (pm:awaiting-maintainer, P1 — dashboard Redeploy + Ignored Build Step fix) was NOT deferred by this ruling and remains on the 待您 ledger.Generated by Claude Code
build(docs): stop emitting the 348 MB of server source maps the OOM-k…
hotlong commentedon Aug 28, 2026
Adjudicated by events: option A was taken, and it worked
This card was deferred on 2026-08-27 («12683 本轮不处理») on the reading that there was «零当前故障». That premise expired the same day: production docs deploys began failing with
exit 137/errorCode: "out_of_memory", the site served a stale deployment for two days, and the team build queue reached 92 deployments with the oldest 34 hours old (#12711).The maintainer has since moved the docs project to an Enhanced build machine (8 vCPU / 16384 MB, project-scoped; the team's other six projects stay on
standard). That is this card's option A.out_of_memory, thenBUILD_EXCEEDED_MAXIMUM_TIMENote for ④ («A 零代码零新义务»): it also turned out cheaper per build, not more expensive. All three machine tiers bill the same $0.0035/CPU-minute, and doubling the cores cut wall time ~13x because the slowness was reclaim pressure near the ceiling, not work.
What this card got right
Measured and confirmed:
--max-old-space-sizedoes not bound this peak — lowering it to 2048 moved the peak by 23 MB, i.e. noise.experimental.cpusdoes not either — and more decisively than stated here: the kill lands in the compile phase, before static-generation workers are spawned at all.What needs correcting
The attribution. This card explains the 7.6 GB as «The docs site prerenders 400+ MDX pages». Measured on both failing production builds, the kill lands between
Creating an optimized production build ...andCompiled successfully, with zero output in between — the prerender phase never executes. The consumer is Turbopack compile. This also clears the 403 build-time OG images and 403 markdown routes, which look like obvious suspects and are not: they aregenerateStaticParamswork in a phase the process dies before.The growth curve this card wanted (② asked for it, could not state it)
This card independently measured 7592 MB. Two independent measurements, 1 MB apart.
Consequences:
On option C (shard the prerender): the fixed cost is a floor. Even at 100 pages per shard the peak is ≈4652 MB on Vercel, barely better than 200/shard, for a rewrite of the build architecture. ①'s reasoning that C is «唯一随页面数增长而扩展的形状» holds in principle, but the constant term makes it a poor trade at this scale. B and C are not worth doing.
Free knobs, for the record
Only one existed, and it is merged (#12712):
experimental.turbopackSourceMaps: false, −434 MB, plus 348 MB less build output. It now banks margin rather than spending it. Everything else measured as noise (±200 MB):turbopackScopeHoisting,--max-old-space-size,turbopackFileSystemCacheForBuild. Rejected:turbopackMinify: false(−972 MB but client JS 5.8 → 16 MB). Non-starter: splitting large MDX files — cost is per page, not per byte; truncating the 292.9 KBv17.mdxmoved the peak +191 MB, i.e. noise. Also note Turbopack's build filesystem cache raises peak (4591 → 6488 MB) while cutting compile 20s → 6.1s — it trades away exactly the resource that was scarce, though it may be attractive now that headroom exists.Separately, #12743 / PR #12746 now skips docs builds that cannot change the output (80% of pushes in production so far), so the ceiling is approached far less often.
Closing: option A adjudicated by events, executed, and verified. Refs #12711, #12743, #12698, #12677, #12333.