Allow explicitly building without mpi4py?

Version: 1.5.8
System: A Cray with SLES 15.3, but I expect this problem will occur on any system
Description of problem: The netcdf4-python installer auto-detects mpi4py by trying to load the module, and will build with it if available. This is a problem for us because we are trying to install through Spack, which requires having some Python version loaded, and our Python installation happens to include mpi4py but we do not want netcdf4-python to use it (partly because we don't need it, and partly because if we wanted it, we would want it to point to an mpi4py copy inside our Spack-based installation, not the Python installation that's only used at build time to run Spack). We are working on finding a way to modify our $PYTHONPATH to work around this issue, but would it be possible to have with/without mpi4py as an explicitly configurable build option as opposed to the current auto-detection scheme?

Actually, setup.py looks for #define NC_HAS_PARALLEL in netcdf_meta.h to determine whether to add parallel support to the build. If it finds it, it then imports mpi4py to find out where the mpi4py header files are. So, if you configure the spack-stack netcdf C library to not have parallel support then netcdf4-python won't have it either.

Thanks for the info. I think what makes our case a bit unusual is that we are compiling NetCDF with MPI support, but the Python module without it. It sounds like maybe the Spack recipe should automatically determine MPI support based on whether NetCDF-C was built with it rather than having it as an independently configurable option.

it would be relatively simple to disable parallel support by patching the setup.py file for netcdf4-python, if you wanted to apply this diff in the spack recipe

--- a/setup.py
+++ b/setup.py
@@ -392,7 +392,8 @@ if 'sdist' not in sys.argv[1:] and 'clean' not in sys.argv[1:] and '--version' n
        (netcdf_lib_version > "4.4" and netcdf_lib_version < "4.5"):
         has_cdf5_format = True

-    has_parallel_support = check_has_parallel_support(inc_dirs)
+    #has_parallel_support = check_has_parallel_support(inc_dirs)
+    has_parallel_support = False
     has_has_not = "has" if has_parallel_support else "does not have"
     print(f"netcdf lib {has_has_not} parallel functions")

@AlexanderRichert-NOAA

@AlexanderRichert-NOAA FWIW, I ended up applying this patch to force building without parallel features, because I ran into this problem: nanoporetech/ont_fast5_api#21 (with py-netcdf4/1.7.1-post2 and mpi4py 4.0.1). I think we need to apply the patch when ~mpi is set for py-netcdf4 (and we need to change our variant request in common packages.yaml).

@jswhit auto-detecting and guessing what user wants is more often than not not the best idea. There should be an explicit flag for py-netcdf4 that the user can set. If they want parallel capabilities, then check that everything is built with those options. Otherwise, don't care and build w/o parallel features.

@jswhit

@jswhit @AlexanderRichert-NOAA I wanted to check if there is any update on this issue. We still need to apply the patch if we want to build without MPI (and we only have a patch for version 1.7.2 in Spack). I am wondering if the original issue still exists (auto-detecting mpi4py in the latest develop code)?

@AlexanderRichert-NOAA

@AlexanderRichert-NOAA Re-reading this a few times ... I think this problem is resolved as long as we want to build with MPI support enabled, because we build Python with Spack and the Spack-built Python does have mpi4py in the build environment for py-netcdf4.

As far as I'm aware our needs on this as far as operations haven't changed, i.e., we want the option of building without MPI but ideally without needing a separate NetCDF C spec. I agree that auto-guessing without the option to configure is not an ideal approach, but of course that's ultimately the developers' call. From my perspective, the Spack patch based on what Jeff provided is sufficient to work around the issue.

I could have setup.py check for an envar (DISABLE_PARALLEL_SUPPORT) to over-ride the autodetection. Would that work?

Yes, that sounds reasonable-- It's easy enough to set a build-time environment variable within Spack.

@AlexanderRichert-NOAA

@AlexanderRichert-NOAA PR #1470 adds this. I have not tested it yet.

I tested building with Spack and I can confirm it disables the parallel support. Once this is in a release, I'm happy to update or help update the Spack recipe (which will need a little tweaking anyway, as it stills requires matching of mpi enablement with netcdf-c and hdf5 even with the patching).

Navigation Menu

Repository navigation

Allow explicitly building without mpi4py? #1389

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Allow explicitly building without mpi4py? #1389

Description

Activity

jswhit commented on Nov 20, 2024

AlexanderRichert-NOAA commented on Nov 20, 2024

jswhit commented on Nov 21, 2024

climbfuji commented on Jan 18, 2025

climbfuji commented on Feb 17, 2026

climbfuji commented on Feb 17, 2026

AlexanderRichert-NOAA commented on Feb 17, 2026

jswhit commented on Feb 18, 2026

AlexanderRichert-NOAA commented on Feb 18, 2026

jswhit commented on Feb 19, 2026

AlexanderRichert-NOAA commented on Feb 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions