Repository navigation
The future of GYP in Node.js #2
Description
Inspired by a discussion with @indutny, but this has been brewing for a long time now:
It's clear that Chromium has effectively deprecated GYP and stopped active development on it, with effort moving over to GN. @domenic reports that this is being finished up "this month".
GYP is stuck on Python ^2.7 and we get a regular stream of issues filed against node-gyp because of the pain people feel when using a default installed Python 3. In fact, the strict limited range of Python support (must be at least 2.7) causes pain on our build infra, where very old EL (EL5 / CentOS 5) comes with an older version and much of its system tools are tied to that version so introducing a new Python becomes a bit hacky.
We don't "support" GYP, we regularly get pull requests against node-gyp and nodejs/node that adjust the GYP source and our normal response is to take it upstream, where it sometimes succeeds through the Chromium gauntlet and is sometimes dropped. We've been (generally) unwilling to float patches, which puts us, and therefore our users, entirely at the mercy of Chromium developers. This is obviously an increasing concern as they move to GN and GYP becomes a legacy concern that they need to detach themselves from.
So, we need to come up with a plan for how to deal with GYP, or at least a general goal that we can encourage effort on where appropriate. If we are not proactive about this we're going to be left with a lot of pain, in fact, we've probably left it far too late as it is.
There are two very distinct issues to discuss and we need to make sure we keep them both in mind when discussing:
- Ecosystem: we have a very large ecosystem that depends on node-gyp, it's built into every version of npm, there's a ton of knowledge about how to wrangle it for Node.js addons and it's vital (IMO, up for discussion of course), that we consider longer-term support for GYP and node-gyp to ensure ecosystem stability—even if we plan on migrating to another tool.
- Core: we use GYP in core (libuv does too, but it's optional). One of the primary reasons for doing so is that our most complex dependency (by far), V8, uses it so it's straightforward to extend that into the rest of the project. When Chromium's support for GN starts to progress further than GYP or GYP is completely dropped, we need to be prepared for it. This doesn't necessarily mean dropping GYP as our primary tool but there are obvious ramifications if we're supporting it while Chromium isn't.
Some beginning talking points:
- @chrisdickinson has collected some useful links on GN @ collecting useful links chrisdickinson/nojs#3 and that project uses GN
- This thread on Twitter discusses how difficult it is to extract GN from the Chromium codebase and reuse it elsewhere, it's not a separate project we can just pick off the shelf although @chrisdickinson is doing some work on that.
- We could consider taking over GYP from Chromium, either officially or by forking it. This would be particularly helpful with ecosystem fork and maybe we can finally get Python 3 support into it! Maybe we can even make it more pleasant for our needs. I know we have some quality Python hackers on the team that could step up.
- CMake often gets proposed as an alternative, I'm not a huge fan but I also don't have a ton of experience so can't be objective. It's worth discussing since it's a mature solution.
- There have been multiple experiments in creating something new, including a bunch of JavaScript-based tools (@indutny proposes bootstrapping with duktape for this purpose). This feels a little like the Jenkins-replacement effort, it's a lot of work that someone needs to do and we may not have anything to show for a long time.
Activity
saghul commentedon Jun 9, 2016
require('this_comment');
I think replacing GYP (the build file format thingy) would break things too much. We can keep GYP (the syntax) and replace the project with a pure JS thing which just generates Ninja files. GYP and GN files are similar enough that even if Google drops it Node can keep up. And be independent.
I did that, and called it GYN and I was able to compile iojs with it. Original post. I still volunteer to finish the work, if that's the approach everyone agrees upon.
Note that this is not orthogonal to replacing GYP with a pure JS thing. Since it's the same codebase, GYP could be replaced with GYN, which gives us the advantage of working on Python 3 and switching to Ninja. Then if GYN is replaced with a pure JS implementation, the Python dependency would go away.
/saghul's 2 cents
seishun commentedon Jun 9, 2016
We've already survived migration from waf to gyp. Migrating the ecosystem from gyp to gn is going to painful, but in the long term it's better to use a build system that's actually used outside of Node.js.
So in the end it depends on what V8 does, doesn't it? I'm not sure how much V8 team cares about third-party users, but if they do, I find it unlikely that they would switch to
gnin its current state.Not a huge fan of introducing another prerequisite for building Node.js.
bnoordhuis commentedon Jun 9, 2016
The ecosystem was a lot smaller those days, though.
But only in one other project (Chromium), just like GYP in 2012.
There was debate back then about whether to move to GYP or CMake. I supported GYP but, with the benefit of hindsight, that was the wrong move.
seishun commentedon Jun 9, 2016
What about V8?
Do you mean addons or Node.js itself? I'm not sure what would be worse, using different build systems for addons and Node.js itself, or using a different build system for Node.js than for V8.
indutny commentedon Jun 9, 2016
I'm working on a JS port of GYP at this very moment. Once it is done - it should be easier for the contributors to support it, and we could possibly modify it to fit our needs better.
ofrobots commentedon Jun 9, 2016
Some other discussions on this: nodejs/node#6089 and nodejs/NG#24.
eljefedelrodeodeljefe commentedon Jun 9, 2016
For reference I was working on this week and have a working compiler abstraction here: https://github.com/eljefedelrodeodeljefe/platform-tools
I am able to build C/C++ source files, native addons (haven't pushed win support yet) across all platforms and will head to compiling node itself after the weekend.
Also I disagree that we need a port of GYP or integrating Ninja or GN, since the task of building sources gets overcomplicated and overabstracted in all of those tools, imo, after looking at their sources and having been able to build a prototype really quick. There is a little bit more justification for a build tool on windows, since VS hides a lot of information like include paths in macros, which you have to dig out really. After that a callout to
cl.exeis as easy as one togcc.saghul commentedon Jun 9, 2016
@eljefedelrodeodeljefe Thanks for working on this! What I suggested was always based on the assumption that the migration would be slow, and changing one component at a time, trying to limit the risk.
I'm happy to hear you are able to compile addons calling the compiler directly, that's great news! FWIW Python does the same, you just create an instance of Extension, list some sources, call python setup.py build_ext and magic happens.
Keep up the good work!
indutny commentedon Jun 12, 2016
FWIW, https://github.com/indutny/gyp.js
indutny commentedon Jun 17, 2016
Update from gyp.js: node.js builds on windows with ninja and gyp.js, it also builds on mac/linux/freebsd as well, and possible on SmartOS with some tweaks to the .gyp file.
Should we consider investing some love into this project and/or using it in node-gyp?
eljefedelrodeodeljefe commentedon Jun 17, 2016
I hope not. It's good engineering, don't get me wrong, but it is really bad management, sorry. This is 5000+ sloc of a port of a build systems that is by all means mediocre and chromium specific.
I am really unemotional about it, but ...sorry... this .... this is really bad news.
Fishrock123 commentedon Jun 17, 2016
Fwiw, with a JS port of GYP, people will likely no longer require python to build node addons if we use it in
node-gyp, could be a big win, there's a good amount of non-compiler python issues that crop up there.Fishrock123 commentedon Jun 17, 2016
What's bad news? Supporting an entire existing ecosystem without a another required language dependancy that we can better maintain ourselves? Perhaps not perfect, but that's a lot of wins IMO.
evanlucas commentedon Jun 17, 2016
I'm pretty +1 on gyp.js. My only concern is what will we do once v8 switches to gn? IIRC that was happening this year sometime?
eljefedelrodeodeljefe commentedon Jun 17, 2016
That you wanna do something really easy with 5000+ sloc someone needs to maintain and users need to be able to use without the fewest amount of bugs possible. With the rule being: the more sloc the bigger the surface error for bugs. It's specifically bad news, since I don't feel that node has a good track record on rather managerial issues.
57 remaining items
hashseed commentedon Mar 28, 2017
Maintenance of V8's GYP configs will stop by the end of year. Node.js has been the only reason it is still being maintained so far, after Chrome has switched to GN more than half a year ago. Maintaining two build systems long-term doesn't sound particularly appealing.
The proposed plan decouples V8's GYP deprecation from Node.js. Obviously, if Node.js wants to move away from GYP, work has to be invested, regardless of whether Node.js wants to adopt GN or something else. I understand that this does not please everyone, but there are no viable and better alternatives on the horizon to my knowledge.
eljefedelrodeodeljefe commentedon Mar 28, 2017
There easily could be one if @nodejs/ctc would create a policy for one. I wouldn't have expected something creative from your (Google's) corner...which is alright and no offense.
refi64 commentedon Mar 28, 2017
FWIW, I may be alone here, but I like the plan. Seems like it would have the minimal amount of potential breakage and workflow issues.
Trott commentedon Mar 28, 2017
Could you elaborate on this? I suspect I'm misunderstanding the meaning because that doesn't make any sense to me at all. (EDIT: What I mean is: I don't understand how a policy "easily" creates options that aren't there now. But there's a lot I could be missing...)
fhinkel commentedon Mar 29, 2017
The gyp config is already in Node.js' repository as part of deps/v8. The difference is, that changes to it must be copied and pasted (or slightly modified) from V8's GN config files instead of from V8's gyp config files.
hashseed commentedon Mar 30, 2017
Previously mistakenly dumped the meeting notes for the meeting between CTC and V8 team here. Moved them to where they belong here.
Sorry for the confusion.
gibfahn commentedon Mar 30, 2017
@hashseed
Edited to @ everyone who was mentioned
indutny commentedon Mar 30, 2017
Did I mention dukgyp already? Needs some love from Windows, otherwise is able to bootstrap node.
seishun commentedon Mar 30, 2017
Could someone describe what kind of work it would involve? Anything other than rewriting
node.gyp,node.gypiandcommon.gypiand modifyingMakefileandvcbuild.bat?eljefedelrodeodeljefe commentedon Mar 30, 2017
@Trott what I meant is that we are not leveraging the possibility of writing something better than exiting solutions. Gyp, make, CMake and all the likes are often bound to specific environments, like the dev environment at Google, and/ or overcomplicate things tremendously, because they turn DSLs. A viable option would just to make our own, suited for the dev environment of the Node.js ecosystem and being more simple, namely not requiring anything, but JS and a compiler. Other big software projects utilise authoring sub projects very well by having a plan / policy / vision someone can grab, setting out to spend a couple of months just programming what has been previously discussed as requirement.
xml commentedon Apr 2, 2017
Please forgive an outsider to this project. But... it sounds like you have a great idea @eljefedelrodeodeljefe, and nobody seems to dispute the value of the ambition. I think people simply have lots of good experience with the gap between ambition and reality, and all the intervening edge cases. Until there's a tangible, viable alternative (at least a nascent one) that people could examine and rally around, the risk of implementation and adoption can't even be assessed, let alone solved. A strong community are going to quite rightly emphasize and invest in an MVP migration path until that's the case. (An MVP migration path might be a necessary parallel/backup option in any case.)
If you could show a baseline project and say 'let's use this, or something like it', it's a much stronger case. You mentioned you had a working build. Can you package that, name it, document it with a To-Do list, and share it to give people a sense of what would be involved? Once people can count lines, examine interface complexity, react to what's supported and what's not supported (especially around Windows)... it becomes a live option.
Until then, folks are almost certainly going to revert to the known possible.
refack commentedon Apr 10, 2017
[edited to be less opinionated]
@eljefedelrodeodeljefe let me quote the scripture:
But seriously, I think a node-native platform-tools/autotools solution is great (wink wink), and perfectly suited for small/simple addons, and I'm willing to pitch in, if you're willing to commit to your
platform-toolswithout apriori support.If we have a PoC that shows it can compile 'n' arbitrary existing addons (CITGM style), we could convince the community.
As for building the whole node tree, I feel we're stuck with
.gypfiles for the time being (speaking from experience nodejs/node#12218 which is a spin off of nodejs/node#11217 and took ~2man-weeks, just to deobfuscate one half of one rather small.gypfile).IMHO as a first step we should concentrate on the addons/node-gyp use case.
hashseed commentedon Apr 27, 2017
As previously mentioned, V8 team is going to stop supporting the GYP build by the end of 2017. Here's the public announcement.
Trott commentedon Sep 8, 2017
This issue has been inactive for a while and this repository is now obsolete. I'm going to close this, but feel free to open another issue in a relevant active repository (TSC perhaps?) and include a link back to this issue if this is a subject that should receive continued attention.