Skip to content

build: build milo through the same WASM build infrastructure as llhttp #5938

Open

Description

@mcollina

#5492 added milo as an experimental HTTP/1.1 parser. Its build doesn't go through the same pipeline as llhttp, and that pipeline is also what Node.js uses when it updates undici in nodejs/node. Before milo can go beyond experimental (and ideally before it ships in a Node.js release), we need to bring the two in line.

@ShogunPanda, can you take a look?

How llhttp is built today

  • The llhttp C sources are vendored in deps/llhttp.
  • build/wasm.js compiles them with clang/wasi-sdk inside the pinned ghcr.io/nodejs/wasm-builder container (npm run build:wasm), runs wasm-opt, and writes the base64-wrapped lib/llhttp/llhttp-wasm.js / llhttp_simd-wasm.js.
  • The build is configurable through WASM_CC, WASM_CFLAGS, WASM_LDFLAGS, WASM_LDLIBS, WASM_OPT and EXTERNAL_PATH. Distros building with --shared-builtin-undici/undici-path rely on this, and we test it in .github/workflows/nodejs-shared.yml with the system clang and wasi-libc.
  • In Node.js, tools/dep_updaters/update-undici.sh deletes lib/llhttp/llhttp*.*, runs npm run build:wasm and saves the build environment to lib/llhttp/wasm_build_env.txt. The WASM that ships in Node.js is therefore always rebuilt from source in a known, reproducible environment.

How milo is built today

  • build/milo.js runs npm pack @perseveranza-pets/milo-cjs@latest and extracts the tarball into lib/milo.
  • No milo sources are in the repo. lib/milo/binary/{simd,no-simd}.wasm and the base64 blobs in lib/milo/src/*/index.js are prebuilt artifacts taken from npm.
  • The version is @latest, not pinned, so npm run build:milo isn't reproducible.
  • update-undici.sh in Node.js doesn't know about milo. It would vendor those prebuilt binaries into deps/undici as they are, without rebuilding them and without recording a build environment.
  • --shared-builtin-undici builds and nodejs-shared.yml have no way to rebuild milo with the system toolchain.

What needs to be done

  1. Vendor the milo sources at a pinned version (e.g. deps/milo, like deps/llhttp) instead of downloading the published npm package. Add an update path, either a script or a workflow, that bumps the pinned version and opens a PR.
  2. Build milo in build/wasm.js, or a script it calls, so that npm run build:wasm produces both the llhttp and milo WASM in the same wasm-builder container with the same wasm-opt flags and the same SIMD / no-SIMD variants. If milo needs a Rust toolchain, we need to either add it to nodejs/wasm-builder (and bump the pinned digest), or have milo emit C sources that build with the existing clang/wasi-sdk setup. The second option is better for distros, which already have clang + wasi-libc but may not have a wasm32 Rust target.
  3. Honour the existing env overrides (WASM_CC/WASM_CFLAGS/WASM_LDFLAGS/WASM_LDLIBS/WASM_OPT, or the Rust equivalents if we go that way). Extend nodejs-shared.yml so the shared-builtin job also rebuilds milo from source.
  4. Generate the JS wrapper the same way as llhttp. Embed the WASM through writeWasmChunk (lazy base64 → Buffer) and load it from lib/dispatcher/parser-h1.js like llhttp-wasm.js. Only keep the files undici actually uses: right now we vendor unbundled.js for both variants plus the raw .wasm files, and .npmignore doesn't exclude lib/milo/binary/*.wasm, so the parser ships twice in the npm tarball.
  5. Make the output removable and rebuildable by Node.js. update-undici.sh should be able to delete the milo artifacts and regenerate them with npm run build:wasm, with the build environment recorded the same way as wasm_build_env.txt. Once undici is set up this way, the corresponding change to tools/dep_updaters/update-undici.sh in nodejs/node should be minimal or not needed.
  6. Add a CI check that rebuilds the WASM from the vendored sources and fails if the committed artifacts don't match. This covers both llhttp and milo.

Once this is done, Node.js can pick up milo the same way it picks up llhttp: from source, through the same build infrastructure, with no prebuilt binaries from npm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions