Repository navigation
docs deploy: production builds are OOM-failing now — the peak is the Turbopack compile phase, not the prerender; turbopackSourceMaps is a free knob #12683's option set missed #12711
Description
Production docs deploys are failing right now — five consecutive production builds OOM-killed. This is the live-outage counterpart to #12683's standing-risk card, and it corrects the attribution in both #12683 and #12698.
Measured (Vercel API + build logs, 2026-08-27)
Vercel's own API on the failed deployment:
errorCode = "out_of_memory"
buildMachine = { purchaseType: "standard", cores: 4, memory: 8192 }
The peak is the Turbopack COMPILE phase, not the prerender
Both failures die between Creating an optimized production build ... and Compiled successfully — with zero output in between:
11:31:03.206 Creating an optimized production build ...
11:42:26.615 ERROR ... exited (137) <- 11m23s of silence
11:43:01.851 Creating an optimized production build ...
11:45:03.086 ERROR ... exited (137) <- 2m of silence
The build never reaches Collecting page data / Generating static pages. So #12683's attribution — "the docs site prerenders 400+ MDX pages", offered as the reason one process holds 7.6 GB — cannot be the mechanism: that phase does not execute. Same for the 403 build-time OG PNGs (app/og/docs/[...slug]/route.tsx) and the 403 markdown routes (app/llms.mdx/docs/[[...slug]]/route.ts) — both are generateStaticParams work in a phase the build dies before.
Two of #12683's negative findings are confirmed by measurement below: --max-old-space-size and experimental.cpus do not bound this peak. experimental.cpus additionally cannot matter here for a structural reason — its static-generation workers have not spawned yet when the kill lands.
Experiment matrix (local cold builds, 403 MDX, peak single-process RSS)
| config | peak | compile | verdict |
|---|---|---|---|
| baseline | 5191 MB | 22.6s | — |
experimental.turbopackSourceMaps: false |
4757 MB | 19.5s | ✅ -434 MB (-8.4%), no downside |
turbopackScopeHoisting: false |
4806 MB | 19.1s | ❌ no effect |
NODE_OPTIONS=--max-old-space-size=2048 |
4734 MB | 18.0s | ❌ no effect (peak is Rust-side, confirming #12683) |
turbopackFileSystemCacheForBuild: true |
4779 MB | 22.2s | ❌ no effect |
+ turbopackMinify: false |
3785 MB | 16.2s | |
includeProcessedMarkdown: false |
— | — | ❌ build fails (getLLMText / /llms.txt depend on it) |
Why turbopackMinify: false is rejected: client JS goes 5.8 MB -> 16 MB (+176%) — that is real user download weight. Restricting it to the server side (where the memory actually goes: 589 MB of server chunks vs 5.8 MB client) is not available: experimental.serverMinification is consumed only by dist/build/webpack-config.js and is never read on the Turbopack path.
Why sourcemaps are free here
The build emits 260 sourcemap files totalling 348 MB into .next/server. Production serverless functions have no use for them. Next's documented build-time default for turbopackSourceMaps follows productionBrowserSourceMaps (false), but the server-side maps are emitted regardless — setting the flag explicitly is what suppresses them.
Side benefits beyond the memory: 348 MB smaller build output, and compile time drops 22.6s -> 19.5s locally.
Proposed change
// apps/docs/next.config.mjs
experimental: {
cpus: 2,
turbopackSourceMaps: false,
}Honest limitation: local measurement is macOS/arm64; Vercel is Linux/x86_64, and the absolute figures differ substantially (local 4.7 GB vs the 7.6 GB #12683 measured). The ratio is the transferable part, and even it is not a guarantee. Scaling -8.4% onto 7592 MB gives ~6950 MB against an 8192 MB container — roughly 15% headroom. That is thin. It is worth trying first because the cost of being wrong is one build cycle, and #12683's option A (bigger build machine) stays available and unaffected.
Also worth noting: Vercel takes 11 minutes to compile what takes 20s locally. A 4-vs-10 core gap does not explain 30x; the rest is thrashing near the container ceiling. Lowering the peak should cut build time substantially as a side effect.
Correction to #12698
#12698's action 2 ("fix the Ignored Build Step — it skips dependency-only changes") rests on a misreading. The project's actual rule is:
if [ "$VERCEL_ENV" == "production" ]; then exit 1; else exit 0; fi
exit 1 = build, exit 0 = skip. So production always builds and Preview always skips — it is not a path-scoped rule. The Canceled by Ignored Build Step that #12698 read off PR #12684's head was that PR's Preview deployment being skipped, which is the intended behaviour. Pushes to main do trigger production builds; they have simply been OOM-failing. No dashboard change is needed for the ignore step.
Re-check
curl -s https://objectstack.ai/ | grep -o 'data-dpl-id="[^"]*"'
Refs: #12683 (standing-risk card — this adds an option E its set did not contain) · #12698 (ignore-step correction) · #12677 · #12333 · epic #12243.
Activity
os-zhuang commentedon Aug 27, 2026
Claimed. session
3100934e-af9d-4104-9145-a37b8280a504· branchclaude/issue-12711-docs-build-sourcemaps· worktreesort-dot-path-fallback-fc72e9.Scope for this claim: the
experimental.turbopackSourceMaps: falsechange toapps/docs/next.config.mjsplus its changeset — nothing else. The build-machine decision (#12683 option A) stays with the maintainer and is untouched by this PR; it remains available if the measured headroom turns out insufficient on Linux.build(docs): stop emitting the 348 MB of server source maps the OOM-k…
os-zhuang commentedon Aug 27, 2026
Result of the merged fix: the failure mode changed, the outage did not end
PR #12712 merged as
18b6d89c. The production build it triggered (dpl_BF8NovndAJa5ezngzUJPphn41hXB) failed, but not the way its predecessors did:errorCodeout_of_memoryBUILD_EXCEEDED_MAXIMUM_TIMEThe flag is confirmed applied on Vercel — the build log prints
⨯ turbopackSourceMapsalongside· cpus: 2, byte-identical to the local run where source maps measurably went 260 files/348 MB → 0 and peak RSS dropped 434 MB.Reading
This is the signature of a process that stopped crossing the OOM line but is still pressed against it. It entered
Creating an optimized production build ...at 13:28:53 and emitted nothing for the next 27+ minutes; the same compile takes 20s locally. Reclaim pressure near the container ceiling is the only thing that accounts for a 100x slowdown on work that is not I/O bound, and it is exactly what the earlier 11m23s-of-silence-then-SIGKILL was already showing in weaker form.So the −8.4% was real and it was not enough: it bought just enough headroom to avoid the kill and not enough to let the compiler run. The ~15% projected margin this card flagged as "thin" turned out to be thin in the direction that matters.
What this retires
turbopackScopeHoisting,--max-old-space-size,turbopackFileSystemCacheForBuildmeasured as noise (matrix in the issue body);turbopackMinify: falsewould give another 972 MB but inflates client JS 5.8 MB → 16 MB (+176%) and cannot be confined to the server side;includeProcessedMarkdown: falsebreaksgetLLMText. There is no further free number to lower.What it confirms
Memory is the binding constraint, and the container is undersized for this build — now demonstrated from both sides: below the line it is killed, at the line it is too slow to finish. #12683 option A (a larger build machine) is the remaining route, and this run is the evidence its decision was waiting on. Current machine:
standard, 4 cores / 8192 MB, on aproteam.The merged change stays: it is a net win independently of the machine (348 MB less build output, −434 MB peak, faster compile), and on a larger container it keeps that margin rather than spending it.
dpl_2nfWjGjSwZjakVEmUG1kBWD6697r, unchanged since before the outage.hotlong commentedon Aug 27, 2026
Resolved — the outage is over, and the fix was the machine
The maintainer moved the docs project to an Enhanced build machine (8 vCPU / 16384 MB, project-level only; the other six projects stay on
standard). Vercel had independently reached the same conclusion, recordingbuildMachineElasticReason: "build-timeout-failure"on the project.Result, measured:
out_of_memory, thenBUILD_EXCEEDED_MAXIMUM_TIMEhttps://objectstack.ai/now servesdpl_3YG2asYDmqBzoQt9eiicVLgZYivP— off the pinneddpl_2nfWjGjSwZjakVEmUG1kBWD6697rfor the first time since the outage began. HTTP 200,/docsrenders. The re-check in #12698 is satisfied.A larger machine turned out to be cheaper per build, not more expensive. Same $0.0035/CPU-minute across tiers, and doubling the cores cut wall time by ~13x because the slowness was reclaim pressure, not work. The earlier projection in this thread ($110–164/mo) was pessimistic for the same reason.
What the evidence chain established
next buildholds ~7.6 GB in ONE process — the docs deploy's next memory ceiling, and no in-repo knob bounds it #12683 and clears OG images / markdown routes as suspects.2360 MB + 6.21 MB × pageslocally, which extrapolates to≈3682 MB + 9.7 MB × pageson Vercel and predicts 7591 MB at 403 pages against docs build:next buildholds ~7.6 GB in ONE process — the docs deploy's next memory ceiling, and no in-repo knob bounds it #12683's independently measured 7592 MB. 403 pages was at 92.7% of an 8192 MB container — past the safe water line, which is why it had been fine for weeks and then was not.turbopackSourceMaps: false(merged, −434 MB, kept — it now banks margin instead of spending it) was the only real one.turbopackScopeHoisting,--max-old-space-size,turbopackFileSystemCacheForBuildall measured as noise (±200 MB).turbopackMinify: falsegives −972 MB but inflates client JS 5.8 → 16 MB. Splitting large MDX files does nothing — cost is per page (6.21 MB), not per byte; truncating the 292.9 KBv17.mdxmoved the peak by +191 MB, i.e. noise.Capacity, going forward
At
≈3682 MB + 9.7 MB/pageagainst 16384 MB, the 75% water line sits near 870 pages; the site is at 403. That is the number to re-check against, and #12743 (avoiding unnecessary builds) reduces how often the ceiling is approached at all.Closing this card. Residual work is tracked in #12743.
build(docs): stop rebuilding the docs site on every push to main (#12746