Repository navigation
build: build milo through the same WASM build infrastructure as llhttp #5938
Copy link
Copy link
Open
Description
#5492 added milo as an experimental HTTP/1.1 parser. Its build doesn't go through the same pipeline as llhttp, and that pipeline is also what Node.js uses when it updates undici in nodejs/node. Before milo can go beyond experimental (and ideally before it ships in a Node.js release), we need to bring the two in line.
@ShogunPanda, can you take a look?
How llhttp is built today
- The llhttp C sources are vendored in
deps/llhttp. build/wasm.jscompiles them with clang/wasi-sdk inside the pinnedghcr.io/nodejs/wasm-buildercontainer (npm run build:wasm), runswasm-opt, and writes the base64-wrappedlib/llhttp/llhttp-wasm.js/llhttp_simd-wasm.js.- The build is configurable through
WASM_CC,WASM_CFLAGS,WASM_LDFLAGS,WASM_LDLIBS,WASM_OPTandEXTERNAL_PATH. Distros building with--shared-builtin-undici/undici-pathrely on this, and we test it in.github/workflows/nodejs-shared.ymlwith the system clang and wasi-libc. - In Node.js,
tools/dep_updaters/update-undici.shdeleteslib/llhttp/llhttp*.*, runsnpm run build:wasmand saves the build environment tolib/llhttp/wasm_build_env.txt. The WASM that ships in Node.js is therefore always rebuilt from source in a known, reproducible environment.
How milo is built today
build/milo.jsrunsnpm pack @perseveranza-pets/milo-cjs@latestand extracts the tarball intolib/milo.- No milo sources are in the repo.
lib/milo/binary/{simd,no-simd}.wasmand the base64 blobs inlib/milo/src/*/index.jsare prebuilt artifacts taken from npm. - The version is
@latest, not pinned, sonpm run build:miloisn't reproducible. update-undici.shin Node.js doesn't know about milo. It would vendor those prebuilt binaries intodeps/undicias they are, without rebuilding them and without recording a build environment.--shared-builtin-undicibuilds andnodejs-shared.ymlhave no way to rebuild milo with the system toolchain.
What needs to be done
- Vendor the milo sources at a pinned version (e.g.
deps/milo, likedeps/llhttp) instead of downloading the published npm package. Add an update path, either a script or a workflow, that bumps the pinned version and opens a PR. - Build milo in
build/wasm.js, or a script it calls, so thatnpm run build:wasmproduces both the llhttp and milo WASM in the samewasm-buildercontainer with the samewasm-optflags and the same SIMD / no-SIMD variants. If milo needs a Rust toolchain, we need to either add it tonodejs/wasm-builder(and bump the pinned digest), or have milo emit C sources that build with the existing clang/wasi-sdk setup. The second option is better for distros, which already have clang + wasi-libc but may not have a wasm32 Rust target. - Honour the existing env overrides (
WASM_CC/WASM_CFLAGS/WASM_LDFLAGS/WASM_LDLIBS/WASM_OPT, or the Rust equivalents if we go that way). Extendnodejs-shared.ymlso the shared-builtin job also rebuilds milo from source. - Generate the JS wrapper the same way as llhttp. Embed the WASM through
writeWasmChunk(lazy base64 →Buffer) and load it fromlib/dispatcher/parser-h1.jslikellhttp-wasm.js. Only keep the files undici actually uses: right now we vendorunbundled.jsfor both variants plus the raw.wasmfiles, and.npmignoredoesn't excludelib/milo/binary/*.wasm, so the parser ships twice in the npm tarball. - Make the output removable and rebuildable by Node.js.
update-undici.shshould be able to delete the milo artifacts and regenerate them withnpm run build:wasm, with the build environment recorded the same way aswasm_build_env.txt. Once undici is set up this way, the corresponding change totools/dep_updaters/update-undici.shin nodejs/node should be minimal or not needed. - Add a CI check that rebuilds the WASM from the vendored sources and fails if the committed artifacts don't match. This covers both llhttp and milo.
Once this is done, Node.js can pick up milo the same way it picks up llhttp: from source, through the same build infrastructure, with no prebuilt binaries from npm.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity