node-gyp Python 3 support

Considering Python 2 will not be maintained past 2020 and node-gyp Python 3 support looks unclear, any thoughts here how building N-API modules could look like in a year from now? Any alternatives to start exploring now?

@sam-github

Thanks @sam-github. With EXPERIMENTAL_NODE_GYP_PYTHON3="yes" and master branch it works on Linux and fails on macOS and Windows:

macOS (Xcode and CLT are installed)

$ node-pre-gyp configure

node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using node-pre-gyp@0.13.0
node-pre-gyp info using node@10.16.3 | darwin | x64
gyp info it worked if it ends with ok
gyp info using node-gyp@5.0.3
gyp info using node@10.16.3 | darwin | x64
gyp info find Python using Python version 3.7.4 found at "/Users/d037732/.pyenv/versions/py374/bin/python"
gyp info spawn /Users/d037732/.pyenv/versions/py374/bin/python
gyp info spawn args [ '/Users/d037732/src/NG-APPS/node-rfc/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/d037732/src/NG-APPS/node-rfc/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/d037732/src/NG-APPS/node-rfc/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/d037732/Library/Caches/node-gyp/10.16.3/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/Users/d037732/Library/Caches/node-gyp/10.16.3',
gyp info spawn args   '-Dnode_gyp_dir=/Users/d037732/src/NG-APPS/node-rfc/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/Users/d037732/Library/Caches/node-gyp/10.16.3/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/Users/d037732/src/NG-APPS/node-rfc',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.' ]
No receipt for 'com.apple.pkg.DeveloperToolsCLILeo' found at '/'.
No receipt for 'com.apple.pkg.DeveloperToolsCLI' found at '/'.
gyp: No Xcode or CLT version detected!

Windows

$ node-pre-gyp configure
node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using node-pre-gyp@0.13.0
node-pre-gyp info using node@10.15.3 | win32 | x64
gyp info it worked if it ends with ok
gyp info using node-gyp@5.0.3
gyp info using node@10.15.3 | win32 | x64
gyp info find Python using Python version 3.7.4 found at "C:\Tools\Python374\python.exe"
gyp info find VS using VS2017 (15.7.27703.2026) found at:
gyp info find VS "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools"
gyp info find VS run with --verbose for detailed information
gyp info spawn C:\Tools\Python374\python.exe
gyp info spawn args [ 'C:\\src\\node-rfc\\node_modules\\node-gyp\\gyp\\gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'msvs',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\src\\node-rfc\\build\\config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\src\\node-rfc\\node_modules\\node-gyp\\addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   'C:\\Users\\d037732\\AppData\\Local\\node-gyp\\Cache\\10.15.3\\include\\node\\common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=C:\\Users\\d037732\\AppData\\Local\\node-gyp\\Cache\\10.15.3',
gyp info spawn args   '-Dnode_gyp_dir=C:\\src\\node-rfc\\node_modules\\node-gyp',
gyp info spawn args   '-Dnode_lib_file=C:\\\\Users\\\\d037732\\\\AppData\\\\Local\\\\node-gyp\\\\Cache\\\\10.15.3\\\\<(target_arch)\\\\node.lib',
gyp info spawn args   '-Dmodule_root_dir=C:\\src\\node-rfc',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'C:\\src\\node-rfc\\build',
gyp info spawn args   '-Goutput_dir=.' ]
Traceback (most recent call last):
  File "C:\src\node-rfc\node_modules\node-gyp\gyp\gyp_main.py", line 44, in <module>
    sys.exit(gyp.script_main())
  File "C:\src\node-rfc\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 554, in script_main
    return main(sys.argv[1:])
  File "C:\src\node-rfc\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 547, in main
    return gyp_main(args)
  File "C:\src\node-rfc\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 532, in gyp_main
    generator.GenerateOutput(flat_list, targets, data, params)
  File "C:\src\node-rfc\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 2040, in GenerateOutput
    version=msvs_version)
  File "C:\src\node-rfc\node_modules\node-gyp\gyp\pylib\gyp\MSVSNew.py", line 208, in __init__
    self.Write()
  File "C:\src\node-rfc\node_modules\node-gyp\gyp\pylib\gyp\MSVSNew.py", line 233, in Write
    all_entries = sorted(all_entries)
TypeError: '<' not supported between instances of 'MSVSFolder' and 'MSVSProject'
gyp ERR! configure error

@bsrdjan

@bsrdjan I'm a bit confused. You are using node-pre-gyp, not node-gyp, and node-pre-gyp is using gyp info using node-gyp@5.0.3 according to your output, not node-gyp master. I think you might need to wait until node-gyp is published, and the downstreams use it, unless you want to manually patch node-gyp@master into your deps, just to give feedback on what is broken with py3 (and there are surely things broken).

The Python stack trace is saying that all_entries contains at least one MSVSFolder and at least one MSVSProject and that is like trying to do sort(apples, oranges). This is probably caused by __cmp__() being removed in Python 3... I will dig in but it is probably an old gyp.

The changes in nodejs/node-gyp#1793 should have fixed this one. It would be helpful to have a repeatable test case that fails. Perhaps you are spot on Sam that a new release might fix this.

The macOS issue should be fixed in nodejs/node-gyp#1890

The No module named 'compiler' issue #445 (comment) was fixed in nodejs/node-gyp#1820

@sam-github

@sam-github it shows 5.0.3 but it is the master branch, here my package.json.zip. Testing with pre-gyp is convenient for me because no changes in my binding.gyp needed.

I re-tested with macOS and the error message is now different:

$ export EXPERIMENTAL_NODE_GYP_PYTHON3=yes                                                                                                                                    
$ node-pre-gyp configure
node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using node-pre-gyp@0.13.0
node-pre-gyp info using node@10.16.3 | darwin | x64
gyp info it worked if it ends with ok
gyp info using node-gyp@5.0.3
gyp info using node@10.16.3 | darwin | x64
gyp info find Python using Python version 3.7.4 found at "/Users/d037732/.pyenv/versions/py374/bin/python"
gyp info spawn /Users/d037732/.pyenv/versions/py374/bin/python
gyp info spawn args [ '/Users/d037732/src/NG-APPS/node-rfc/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/d037732/src/NG-APPS/node-rfc/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/d037732/src/NG-APPS/node-rfc/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/d037732/Library/Caches/node-gyp/10.16.3/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/Users/d037732/Library/Caches/node-gyp/10.16.3',
gyp info spawn args   '-Dnode_gyp_dir=/Users/d037732/src/NG-APPS/node-rfc/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/Users/d037732/Library/Caches/node-gyp/10.16.3/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/Users/d037732/src/NG-APPS/node-rfc',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.' ]
Traceback (most recent call last):
  File "/Users/d037732/src/NG-APPS/node-rfc/node_modules/node-gyp/gyp/gyp_main.py", line 44, in <module>
    sys.exit(gyp.script_main())
  File "/Users/d037732/src/NG-APPS/node-rfc/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 554, in script_main
    return main(sys.argv[1:])
  File "/Users/d037732/src/NG-APPS/node-rfc/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 547, in main
    return gyp_main(args)
  File "/Users/d037732/src/NG-APPS/node-rfc/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 532, in gyp_main
    generator.GenerateOutput(flat_list, targets, data, params)
  File "/Users/d037732/src/NG-APPS/node-rfc/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 2215, in GenerateOutput
    part_of_all=qualified_target in needed_targets)
  File "/Users/d037732/src/NG-APPS/node-rfc/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 802, in Write
    self.WriteCopies(spec['copies'], extra_outputs, part_of_all)
  File "/Users/d037732/src/NG-APPS/node-rfc/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 1145, in WriteCopies
    env = self.GetSortedXcodeEnv()
  File "/Users/d037732/src/NG-APPS/node-rfc/node_modules/node-gyp/gyp/pylib/gyp/generator/make.py", line 1885, in GetSortedXcodeEnv
    additional_settings)
  File "/Users/d037732/src/NG-APPS/node-rfc/node_modules/node-gyp/gyp/pylib/gyp/xcode_emulation.py", line 1616, in GetSortedXcodeEnv
    additional_settings)
  File "/Users/d037732/src/NG-APPS/node-rfc/node_modules/node-gyp/gyp/pylib/gyp/xcode_emulation.py", line 1527, in _GetXcodeEnv
    if XcodeVersion() >= '0500' and not env.get('SDKROOT'):
TypeError: '>=' not supported between instances of 'tuple' and 'str'
gyp ERR! configure error

@bsrdjan

@bsrdjan That makes sense, I forgot that master's package has 5.0.3 in it :-)

@cclauss might have ideas on the specific error.

You package isn't on github or otherwise publically available/reproducible, is it? If the build was reproducible, it's probably be faster to fix than round-tripping through here.

modify XcodeVersion() to convert "4.2" to "0420" and "10.0" to "1000"

gyp: modify XcodeVersion() to convert "4.2" to "0420" and "10.0" to "…

Yes, it would simplify the testing and conversation.

The github project is sap/node-rfc and the working branch, built with Python2, is napi. The releases of this branch are linked to npm node-rfc package pre-releases (npm install node-rfc@next).

Above mentioned issues should be reproducible with gyp3 branch, used for Python3 builds.

The cmake branch has no issues with Python 3 but others: #445 (comment).

CMake.js is a viable alternative to node-gyp for building Node N-API native addons. The N-API Team has created an example project that shows how this can be accomplished.

In addition, prebuild now supports CMake.js as a build backend for N-API native addons. The specifics can be found in the documentation. prebuild is of interest to Node native addon maintainers using GitHub as it facilitates the building and uploading of native addon binaries as GitHub releases. This permits users of the native addon to download the binary from GitHub instead of needing to have the necessary build tools installed on their development and deployment machines.

@jschlight

Thanks @jschlight, for the the prebuild hint.

My project works with CMake now, on Linux, Windows and macOS and this issue can be closed AFAIAC.

@bsrdjan

@bsrdjan thanks for the update, closing

@bsrdjan

@bsrdjan You might consider adding an N-API Badge to your README file. The details are here.

thanks, done.

Navigation Menu

Repository navigation

node-gyp Python 3 support #445

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

node-gyp Python 3 support #445

Description

Activity

bsrdjan commented on Sep 25, 2019

sam-github commented on Sep 25, 2019

cclauss commented on Sep 25, 2019

cclauss commented on Sep 25, 2019

bsrdjan commented on Sep 26, 2019

sam-github commented on Sep 26, 2019

bsrdjan commented on Sep 27, 2019

jschlight commented on Sep 27, 2019

bsrdjan commented on Oct 1, 2019

mhdawson commented on Oct 1, 2019

jschlight commented on Oct 7, 2019

bsrdjan commented on Oct 7, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions