Repository navigation
Strategic initiative to move to cmake for building Node.js? #648
Description
@bnoordhuis had done some experimentation to move Node.js to use cmake but I think we need more consensus/momentum to be able to make a move like that.
I'd like to restart discussion in the context of: nodejs/node-addon-api#422. Before we start to experiment with building addons with a different tool like cmake, it would be good to better understand if it might align with what we using in Node.js in the future.
Activity
ofrobots commentedon Jan 2, 2019
This seems related to #642.
For some background there is some related discussion in #464.
To keep the discussion clear and focussed, I think this is about how Node.js-core should be built rather than ecosystem native addons, right? The constraints are quite different between the two problem spaces, so let clarify that this is not about native addons.
Before we can have a strategic initiative for building with cmake, I think we will need to explicitly agree that
cmakeis the build system we are going to use.At the core, the choice of build tool is about modularity. C++, unlike Go, rust, and JavaScript, does not have a standard "assemble-dependencies-and-build" system. Whatever choice of build system we have will have to deal with the fact that we depend on some fairly large projects (e.g. V8) and that we have to make those projects buildable with whatever build tool we chose for building Node.js itself. Most of our other C/C++ dependencies are simple enough that the main challenge we need to solve is how our build tool choice will affect how we incorporate V8. It is not a matter of migrating V8 to cmake once. We have to maintain this build system, and V8 notoriously moves very fast.
For diagnostics we (I?) want to pick up the Perfetto project as a dependency to replace the Trace Events infrastructure that we are maintaining in V8 and Node. This and other future dependencies, and how we consume them, will be affected by our choice of build tool.
I am not necessary against cmake, I just haven't seen consensus around it yet.
gabrielschulhof commentedon Jan 2, 2019
FWIW there's some momentum in nodejs/node-addon-api#422 to move to cmake. Do we need to agree to move the whole project to CMake before we deploy cmake on our CI? I'd like to ask in @nodejs/build that CMake be added to the tools available on the CI (if not already there).
mhdawson commentedon Jan 2, 2019
@ofrobots that is exactly the discussion I wanted to get re-started "how Node.js-core should be built. @bnoordhuis for some reason I was thinking that you might have been building everything except v8 with cmake in your work, but I may not be remembering correctly.
mhdawson commentedon Jan 2, 2019
@gabrielschulhof I mentioned nodejs/node-addon-api#422 above as I wanted to see if there was any consensus on what we should do for Node.js core as that would make the choice for add-ons easier. I don't think it's a prereq but I think its worth exploring as it would be good to avoid having too many different build systems required.
ofrobots commentedon Jan 2, 2019
As a point of comparison, from another notable V8 embedder, deno seems to be using GN to build.
mhdawson commentedon Jan 2, 2019
I think when we asked the Google team last time they specifically said GN was not "supported" for use in a case like ours and recommended other options.
ofrobots commentedon Jan 2, 2019
Adding @nodejs/v8 for clarity about GN and level of support. It is ultimately the V8 team's choice to use GN that puts this cost on their embedders like us.
mhdawson commentedon Jan 2, 2019
@ofrobots thanks, its good to re-confirm on that front. Things may even have changed over time.
mhdawson commentedon Jan 2, 2019
While not the same topic I think it would be good to add to the discussion whether the build system for Node.js core and add-ons should be the same? I wanted to restart the discussion in core to have as good as data as possible for nodejs/node-addon-api#422 but if the concerns can be/should be separated it's good to figure that out as well.
joyeecheung commentedon Jan 3, 2019
It is possible to build v8 with gn but use cmake for the rest of the project? V8’s fork has been using the configure option —build-v8-with-gn to trigger an action and build v8 directly with gn, if we can somehow host gn in-tree and bootstrap by compiling it with cmake (if gn is not available in PATH) then we won’t take on the burden of porting v8 build files to cmake?
Trott commentedon Jan 3, 2019
Same for electron. (Not saying we should use GN. Just another data point.)
ryzokuken commentedon Jan 3, 2019
I had been compiling a list of pros/cons regarding build systems in our case, but had been reluctant to spark off this discussion myself since there's a lot of conflicting opinions.
Some objective truths that are important to consider IMO:
If we want to build using GN, we will not only have make all contributors install depot_tools, we'd also have to use them for various other purposes including fetching dependencies (which is something electron does as well). Important things to consider: another step for new and returning contributors as well as atleast a minor loss in productivity while everyone gets used to the new workflow.
Right now, @targos and I take turns at updating our gypfiles for building V8 whenever canary-base is broken, which on bad days is nearly daily. While CMake is great in the way that we do not have to maintain it by ourselves (Refael has taken up the burden to maintain gyp pretty much all by himself recently, but I wouldn't mention him since he's staying away from the project right now) we will have to do the same for CMake, porting the changes in the latest GN configs to the CMake format.
Pros of CMake/Cons of gyp:
It's easily one of the most widely used build systems out there. Since there's a ton of resources, documentation and knowledge regarding CMake out there in the open, we'd get rid of the current situation where apart from a small group of collaborators, nobody really understands how gyp really works etc.
It's well maintained, and not going anywhere anytime soon.
Pros of gyp/Cons of CMake:
There's a small upfront cost in porting all our gyp configs to CMake, but that should probably be the least of our worries.
CMake is completely unsupported by the V8 team, so it would be futile to expect for a lot.
Nobody in the ecosystem (either upstream at V8, nor downstream at Electron, for example) really uses CMake. We make everyone else's lives hard (including people who help embed Node.js into Electron, /cc @codebytere).
Game changer: Over at Electron, the exact opposite is required to be done (porting our changes in gyp to GN), and they've hacked up a clever script that automatically does most of the work, if not all for them. I've wanted to write a similar script for us vs V8 for quite some time, which should take care of ~70-80% of minor changes to V8's build. If we do manage to cover that many changes, it would be a serious pro in favor of gyp.
(Note: A similar script could also be written down for CMake, tbh. The only difference is that we would not have much to refer to.)
If the official support (neither of the above two are, really) seems to be a major issue, we have two officially supported alternatives (these would cut down the amount of work we need to put in on a near daily basis): GN and Bazel.
Last I discussed this with @hashseed, the Chromium team was not very happy with any outside project using GN, but that might have changed: https://gn.googlesource.com/gn/.
They seemed to be (atleast at that time) more interested in the idea of us using Bazel since Google uses Blaze internally anyway and it was likely that they could share the currently internal build files with us, making our lives easier.
Ofcourse, the GN build files are already public so the sky's the limit for that one.
EDIT: Removed way too many occurrences of the phrase "unless I'm wrong". Please note that I'm by far not an authority in any of these matter and may be colossally wrong in multiple points.
gengjiawen commentedon Feb 3, 2019
I want list some pros of CMake:
CMake has better IDE support
I think CMake is very friendly to new cpp users. Both clion and visual studio support cmake.
With great IDE support we can attract more developers contribute to nodejs.
I have been use https://github.com/lev-kazakov/node-cmake-generator to generate CMake files for me.
Pretty pleasant developer exerience with clion.
If we can go to other build system in the end, maybe still provide an option to generate cmake.
CMake has better cpp tools integration
With CMake, it will be easier for us to integrate tools like
clang-format,clang-tidy.I recently see https://github.com/bugwelle/cpp-starter-project.
refack commentedon Feb 3, 2019
@gengjiawen in that respect GYP has two redeeming qualities. It can generate CMake files (specifically
CMakeLists.txt), and it can generatecompile_commands.json.gengjiawen commentedon Feb 4, 2019
@refack Yeap, that's
node-cmake-generatorinternally using. But it doesn't support the latest cmake.11 remaining items
hashseed commentedon Nov 19, 2019
Note that nosnapshot build has already been removed upstream.
frutiger commentedon Nov 20, 2019
Looks like Google's spam filter was being overly aggressive. I have added you to my contacts list so hopefully there are no further issues.
The https://github.com/frutiger/ncg repo is able to satisfy the following of your bullet points (on the
v10.xbranch):Some notes:
ncgthat would work forv12.x, but I have not completed it. The main issues are minor, and related to new usages ofgypthat were not anticipated when I worked on thev10.xbranch.configure.pystep, but it does some essential things (e.g. downloading ICU) that I do not fully understand or did not want to replicate in CMake just yet.bnoordhuis commentedon Nov 22, 2019
@frutiger Thanks, got your email!
It'd be nice if we could leverage ncg to (re)generate the sources lists and, if possible, generate the list of
torqueinputs and outputs for V8's snapshot mode, since those change frequently. We can commit them whenever V8 is upgraded.ncg would need to work with the master branch for that. I'll see if I can make the necessary tweaks unless that's something you'd like to pick up.
It does but that's something we can punt on.
The bundled ICU (the one in the increasingly inaptly named
deps/icu-smalldirectory) is now full-icu as of nodejs/node#29522 so I expect the download option to become a lot less important for people that build from source.frutiger commentedon Nov 22, 2019
I have just ported
ncgto work withv12.xin no snapshot mode. I will trysnapshotandmasterin the coming days. I'm also mainly usingconfigureas a wrapper to launch GYP.ncgworks in two steps:CMakeLists.txtfileOnce that is done, we don't have much need for the
configurescript. If I have it all working, I will propose the changeset as a strawman againstnode'smasterbranch.sam-github commentedon Dec 16, 2019
I just took a quick look at ncg, and I'm not totally clear on what the plan here is.
Currenty, the build is:
gyp+.gyp filesgenerates-->Makefileinput to -->make(which does the build)The ncg flow looks to be:
ncg+.gyp filesgenerates-->CMakeLists.txt-->cmake(which does the build)I've never used cmake, unless it was quietly behind a
./configure; make, so perhaps I misunderstand, but this would be literally replacing gyp as a tool, but is the intention to continue using .gyp files to control our build configuration, but now as a way of generating cmake files instead of make (or ninja) files ?bnoordhuis commentedon Dec 17, 2019
@sam-github The end goal is to completely move away from gyp. I think ncg or a tool like it will come in handy during the transitional period, when the gyp files are still leading.
V8 and openssl updates usually add and remove a lot of source files. Manually updating only one build system instead of two will save a ton of work.
The cmake build will probably exist of a manually maintained part (setup, configuration, etc.) and auto-generated source lists. The second part is where ncg comes in.
devsnek commentedon Jun 8, 2020
what is the status on this? can i do anything to help?
bnoordhuis commentedon Jun 9, 2020
@devsnek I've kind of given up on this. Things move too fast to keep up.
If you have a few hours to kill, have a look at https://github.com/bnoordhuis/v8-cmake - ICU support in particular is one of the bigger blockers.
The V8 8.4 upgrade is currently also a bit of a pain point but I hope to have that done by the end of the week. (edit: bnoordhuis/v8-cmake#26)
devsnek commentedon Jun 9, 2020
@bnoordhuis to be clear, building node as a whole with cmake looks iffy but building v8 with cmake is within reach?
bnoordhuis commentedon Jun 9, 2020
@devsnek In a nutshell:
gengjiawen commentedon Jun 10, 2020
So recently libuv got android build easily with cmake build system https://github.com/libuv/libuv/blob/2967557b6651ee10dd2f9e0b4baeff35df0ab684/.github/workflows/CI.yml#L17.
I guess we can easily build android Node.js if the toolchain has migrated to cmake.
But as @bnoordhuis has stated, we have to figure out an automatic way on V8 considering we have https://github.com/nodejs/node-v8. (I think openssl won't be a problem as I have seen community using cmake to build it)
mhdawson commentedon Jan 4, 2021
I'm going to close this out. It's stalled and @mmarchini has #901 which should cover it.