Strategic initiative: future of Node.js build toolchain

This was brought up several times, most recently on the Build IRC channel. Our current build toolchain is based on GYP, which is not the default toolchain for V8 anymore and was discontinued by Google. As a result, we have to maintain our own GYP fork, and keeping dependencies (especially V8) up to date requires considerable manual labor. Furthermore, the build experience for Windows and Unix is extremely different, with different config files for those platforms (which makes it hard to keep them in sync feature wise).

Switching to a modern, widely used build toolchain like CMake has been brought up a few times. There were also suggestions to move to GN, which comes with a completely different set of tradeoffs. In the recent discussion on the Build chat I suggested we should investigate an hybrid mode where V8 is built with GN and the rest of the project is built with CMake* (this also would come with a completely different set of tradeoffs). And this is only talking about building Node.js itself, we also need to evaluate how each toolchain will impact native modules, and we'll need to provide gyp + something else for some time on the native modules side.

I suggest we create a strategic initiative to start planning, exploring options and working on the future of our build toolchain. This could potentially become a working group in the future depending on the amount of work we have ahead as well as the number of collaborators interested in helping. I'm willing to champion this initiative.

* I would really like to explore this one because if it works, we can provide pre-built V8 binaries which can be downloaded by collaborators, reducing significantly the build time of the project

cc @nodejs/build-files @nodejs/gyp @nodejs/tsc

@mmarchini

Thanks for taking this up, @mmarchini. I suppose a good first step would be to list down the pros and cons of each approach so that everyone would have a good idea of what exactly are the tradeoffs involved. This has always been a tough pill for me to swallow personally because of the fact that:

Building V8 with CMake doesn't really fix the problem of "considerable manual labor", we'd just end up having to maintain CMakeLists for V8 instead of gypfiles. It might be easier due to various reasons, but it would not go away completely IIUC, unless we switch to GN.
Building with GN comes with its own set of problems, since IIUC, GN supports a much smaller set of platforms than Node. I don't know for sure what the Chromium team feels about it, but adding support for the missing platforms in GN would be a significant undertaking in itself which would come with attached maintenance costs.

I had never considered your suggestion of using GN to build V8 and CMake to build everything else, but it might just work. That said, I am afraid it would still require us to add support for platforms like SmartOS to GN.

I'll need to dig deeper on the whole "GN unsupported platforms" story. My understanding is that, as long as we can target some of those platforms, we should be fine, right? Especially platforms which are more common on servers.

Totally agree with you that changing from gyp to cmake doesn't solve the issues with upgrading V8, which is why I would like to think of it as a last resort. I think there's a lot of investigation and experimentation to do before making any decisions on the path forward.

I'll need to dig deeper on the whole "GN unsupported platforms" story. My understanding is that, as long as we can target some of those platforms, we should be fine, right? Especially platforms which are more common on servers.

I mean, if GN doesn't support a platform that we currently do, either we would have to consider dropping support or continue maintaining gypfiles, right?

I think there's a lot of investigation and experimentation to do before making any decisions on the path forward.

Absolutely agreed. Let's figure this one out, shall we? 😄

I mean, if GN doesn't support a platform that we currently do, either we would have to consider dropping support or continue maintaining gypfiles, right?

I mean, that's probably the case, but I'm not 100% sure 🤷‍♀️. From what I can tell GN just generates Ninja files, so as long as we can use it to generate Ninja files to build on SmartOS and friends we should be fine.

I wonder if we could, as a starting point, use CMake as a frontend for gyp, replacing configure and vcbuild (assuming we can use CMake to generate build files on Windows).

+1 for cmake, we could vendor stuff so much more easily.

+1 for cmake, we could vendor stuff so much more easily.

Except for V8 😅

Regarding "GN unsupported platforms": GN is an extensible build system, even if there are platforms that currently unsupported, one can also add new scripts and config files to include a new toolchain. https://gn.googlesource.com/gn/+/master/docs/reference.md#func_toolchain

Re. gn, it’s not just whether gn supports all of our supported platforms but by implication whether ninja support for those platforms is available.

Feedback from Google in the past is that they were not looking to support GN for use outside of V8/Chrome. They had other suggestions instead. I'm hesitant to bet on GN (except possibly just for V8 itself) without Google recommending/supporting it as a good choice and be willing to help if we run into issue.

@mhdawson

@mhdawson yes, that's definitely going to weight in our decision. The ideal build toolchain for us is still IMO GN for V8 and insert-new-tool-but-probably-cmake for everything else. Biggest challenge is to keep defines and compiler flags in sync between the V8 build and everything else, I have some ideas and am working on a proof-of-concept for GN+GYP hybrid build with synced defines and without depot_tools. This hybrid approach might turn out to be unfeasible or too hacky, but it's hard to know without trying.

I'm also in contact with Google to come up with a solution that is good for both projects (since they also have CI bots for Node.js with updated V8), and I want to include embedders in the conversation to make sure whichever path we choose also work for them.

regarding v8 and cmake: I think that the people at vcpkg and conan could join forces with nodejs to maintain the cmake script...

@gocarlos

Thanks @gocarlos, if we move in the direction of porting V8 to cmake we'll loop those projects in, although I'm still hopeful that we can avoid that (since porting V8 to any other build system will result in burden to maintain that port, which is quite hard considering V8's development speed).

You might be interested in https://github.com/bnoordhuis/v8-cmake, which I think already works.

I would like to vendor getdns, which is practically impossible because we don't support cmake. Additionally, most of our deps already have cmake configs. I think it would be unfortunate if we moved to some new system and still didn't support cmake.

@eli-schwartz

@eli-schwartz maybe you see it out of a other view but since nodejs exists i got build problems with it on diffrent devices and platforms and arch thats nothing new and i needed to learn a lot to solve any of that issues and to even understand how all this systems work.

When we would code the most parts in JS we would create something flexible that a JS Developer could maintain and we would even teach him how to integrate with other lanaguges and software.

As you mentioned already every build system introduces also additional a DSL Domain Specific Language that your free to learn on top :D so i see no point in learning all that details.

Thats my point we can reduce a lot of complexity here and get it right once and for all as we can Easy create good readable Conditional code when we write that build Logic in ESM or if we would even use something like gulp and add if missing plugins.

Everything that reduces layers makes it also later more easy to switch to other build systems without changing as much code because we can then use all the JS Stuff like Injection and Loader Interception patterns.

Whether we use python or nodejs, building systems that depend on external runtimes can introduce more uncertainty and problems. Even if a previous project can be compiled successfully, a python version update that introduces a bug can break meson/scons. like this case: xmake-io/xmake-repo#1013

but xmake has a lightweight builtin lua runtime. So if the current xmake version is stable, it will at least not break xmake builds due to lua runtime version updates and stability issues.

@waruqi

@waruqi and we would have a light lightweight v8 runtime https://github.com/just-js/just/i as light as that for example but with v8 builds for more platform arch combos

i already use JS for system programming and it is the best choice i ever took in the last years the v8/JS/ECMAScript event loop replicates exactly what Zircon (FuchsiaOS Kernel) Does and in general it matches what i do in the cloud. I know out of expirence that every big Cloud Architecture is using the event bus pattern and all this concepts are not new.

but today we see that they are working. I am a big fan of the Overall Movement of the Ecosystem but there are some blockers like the idea of reusing existing stuff and fear of reinventing it.

All that systems got designed out of a 1970 or maybe older view its time to change that. Today we got CPU's with 128+ cores thats simply something total diffrent then it was 10 years back.

just-js by the way ranks 1 in a well designed benchmark suite that tests frameworks https://www.techempower.com/benchmarks/#section=data-r21&test=composite

i can tell you this is amazing my build speed is nice and the results are stable also all this is at last fast understandable!

Thats my point we can reduce a lot of complexity here and get it right once and for all as we can Easy create good readable Conditional code when we write that build Logic in ESM or if we would even use something like gulp and add if missing plugins.

Everything that reduces layers makes it also later more easy to switch to other build systems without changing as much code because we can then use all the JS Stuff like Injection and Loader Interception patterns.

In fact, by reducing layers you make it harder to switch to other build systems, and simultaneously move all complexity into your own project instead of letting a dedicated "build system" project handle the complexity for you.

This has been specifically rejected in this ticket. In fact, it's the entire point of the ticket.

The plan is to steer away from custom solutions (like we currently have).

It's eminently reasonable to want to avoid custom stuff you don't specialize in, and the nodejs team specializes in javasacript runtimes, not cross-platform build systems.

i already use JS for system programming

Build systems are not system programming. If build systems were system programming, the nodejs team would write their own build system where all configuration is done in config.cpp.

I am confident this is not what you want.

@eli-schwartz

@eli-schwartz i guess i understand now where your view comes from but i do not agree while i know many people will agree with you at last it does not matter the future is clear at last for me and i see the whole ecosystem Polyglot moving into the right direction i see overlap in functionality everywhere and i see a OS Getting it finaly right via simply offering a Universal ABI to hardcode interfaces.

All this concepts solve so many problems that they will get adopted it is only a question of time and i only wanted to speed up the overall process at the end it will never matter for the future what the nodejs core project does or does not do. There is a reason why NodeJS gets forked and forked all over and it is not contributing back to NodeJS Core. There are Fundamental Problems that the People will address and they will sideStep RFC Processes as they got simply the ability to do so.

thats all i have to say. At the end You and me are Correct at the same time:

Your point is never touch a running system
My point is build to change or be legacy even before you created the PoC

I need to dig deeper into the replies here, but just FYI, I don't think we're considering switching from ninja or make here. We're just considering switching from gyp-next to a better meta build system.

Another important thing to note is that speed is not our biggest priority here IIUC, our biggest priority is cutting down the work needed to:

Maintain our own bespoke meta build system in Python.
Translate V8's GN build files into gypfiles.

And while there's arguments to be made in favor of writing and maintaining our own in JavaScript, I don't believe it would make our lives much easier as compared to the current situation but will come at a huge cost.

@ryzokuken

@ryzokuken do no worry about that once i am done i will do a PR if you like it take it if not it is total ok

I am Coding a Whole Kernel in ECMAScript so i need some tooling to do custom v8 builds and optional other builds anyway as part of that i will come up with something that is gyp compatible and a drop in replacement.

I will take as much of the XMake implementations as needed. and incremental come up with own ways.

I did code a lot of parser stuff in ECMAScript already parsing the lua code of XMake to ECMAScript is a nobrainer. That should be low Hanging Fruits.

@frank-dspeed

we would have a light lightweight v8 runtime https://github.com/just-js/just as light as that for example but with v8 builds for more platform arch combos

@frank-dspeed I saw it, I don't think it's lightweight, as long as it still relies on v8, because no matter how you trim v8, it's still heavyweight. it's still 50M, and the whole binary of just after linking is 17M.

But the whole xmake binary is only 1M. If lto is turned on, it is only 670K.

I did code a lot of parser stuff in ECMAScript already parsing the lua code of XMake to ECMAScript is a nobrainer. That should be low Hanging Fruits.

I don't really think it's necessary to implement a build system based on js for this, and right now just-js only supports linux.

@waruqi

@waruqi your partial correct just js out of the box does static linking so the resulting binary can be smaller when we do dynamic linking like XMake does overall i guess you get the situation a bit wrong let me explain more.

There is a reason why it is Linux only because it made it simple there is no technical reason for that. The Concepts are Portable. In Fact it is already a Compose able Linux Only NodeJS there are repos that hold all neeeded modules for that.
and it is easy to add more modules as this is simply C and Some Conventions to build v8

thats it!

NodeJS Example

Download Source
run our own make script that builds v8 or downloads prebuild
build our build environment (creating some C code not much that throws our .js files into the v8::isolate) using fast v8 ffi calls to integrate into the Host system v8 includes a own FFI Call Engine so that no round trips are needed to the host environment that embeddeds v8 (Filesystem, network access, exec .....)
build NodeJS with the same v8

Be Happy! No Extra Dependencies no Extra waisted Space as reused the same v8 in this scenario

I do not understand why we are 21 comments deep into a discussion about taking a bespoke python meta-build system and basically bespoke gyp meta-build files and... reimplementing it in bespoke javascript...

... when the topic of this discussion is about moving away from bespoke build systems on the grounds that it "doesn't make things better, but does add a huge cost".

Writing a nodejs-specific build system in any language does not sound like it's on the table. Writing one that depends on having a previously built copy of nodejs or v8 is just additonal pain by way of bespoke "own make script that builds v8", which is exactly the thing that was declared to be a problem to be moved away from.

I think it probably makes quite a bit of sense to stop talking about something that's known to be a non-goal and will not happen.

@mmarchini

We haven't reached a decision, so it's not late

@mmarchini is it now? If not, I'd in turn suggest looking into Meson. I find it odd Meson is mentioned everywhere in the discussion, however nobody has put a suggestion for it.

I never worked with XMake, GN and Bazel, so can't say anything useful here. However, I see there is a suggestion for CMake, and I can tell for sure Meson is better. Basically, you can do all the same things, however it's more automated, and it has waaay nicer syntax which is easier to read (IMO that's the most important difference, meson.build files a way more understandable than CMakeLists.txt ones). Also, it has an implicit type-system, which allows to give much more useful error messages when one screws up something in the build system files.

Meson is very popular, these days it's used by lots of well known projects: Gnome-related ones, Mesa graphics drivers, XServer, libinput, i3wm, etc… Mesa is an interesting user, because it has lots of code-generators, they have different langs (e.g. there's an assembly code, which has to be compiled without lto, otherwise it won't link), so I'd say they use it in non-trivial ways.

@Hi-Angel

@Hi-Angel I guess your correct at last i am working on a ECMAScript written MESON implementation already as the syntax is easy parse able and transpile able. But i also throw GN into the Mix and then the user can choose what he wants to use to build his NodeJS Distrubtion.

We are in a good time periot where you can transpile anything fast. GN, MESON that makes not much difference or Python (Gyp).

At the end the only importent thing is feeding text into gcc or cmake or any esoteric compiler stack that you choose and call it a day.

i am working on a ECMAScript written MESON implementation already as the syntax is easy parse able and transpile able.

That's super cool. Also, we (Meson) have intentionally designed the syntax to be capable of reimplementing in other languages without undue difficulty, there's a FAQ entry about it even. :) So I'm happy to hear that that is working out for you.

Whenever you feel it's ready, I'd love to take a look at your implementation. We could list it in the Meson FAQ as well. Currently we list a couple alternative implementations of Meson, the most advanced one being a c99 version.

feeding text into gcc or cmake or any esoteric compiler stack that you choose

CMake isn't a compiler stack! :p

But yes, indeed, build systems are really just a pile of conventions for calling a compiler, with a few frills added on at the end.

ah sorry cmake is always automaticly assigned with llvm some how in my head

Navigation Menu

Repository navigation

Strategic initiative: future of Node.js build toolchain #901

72 remaining items

NodeJS Example

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Strategic initiative: future of Node.js build toolchain #901

Description

Activity

mmarchini commented on Aug 4, 2020

ryzokuken commented on Aug 5, 2020

mmarchini commented on Aug 5, 2020

ryzokuken commented on Aug 5, 2020

mmarchini commented on Aug 5, 2020

mmarchini commented on Aug 6, 2020

devsnek commented on Aug 6, 2020

mmarchini commented on Aug 8, 2020

victorgomes commented on Aug 10, 2020

richardlau commented on Aug 10, 2020

mhdawson commented on Aug 11, 2020

mmarchini commented on Aug 11, 2020

gocarlos commented on Aug 20, 2020

mmarchini commented on Aug 20, 2020

devsnek commented on Aug 20, 2020

72 remaining items

frank-dspeed commented on Aug 15, 2022

waruqi commented on Aug 15, 2022

frank-dspeed commented on Aug 15, 2022

frank-dspeed commented on Aug 15, 2022

eli-schwartz commented on Aug 15, 2022

frank-dspeed commented on Aug 15, 2022

ryzokuken commented on Aug 16, 2022

frank-dspeed commented on Aug 16, 2022

waruqi commented on Aug 17, 2022

frank-dspeed commented on Aug 17, 2022

NodeJS Example

eli-schwartz commented on Aug 17, 2022

Hi-Angel commented on Oct 25, 2022

frank-dspeed commented on Oct 25, 2022

eli-schwartz commented on Oct 25, 2022

frank-dspeed commented on Oct 25, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions