Repository navigation
Save number attribute as scalar and not as array? #1445
Description
With hdf5, one can save in attributes a number and an array of size 1. These are two different things:
ATTRIBUTE "a_int" {
DATATYPE H5T_STD_I64LE
DATASPACE SCALAR
DATA {
(0): 10
}
}
ATTRIBUTE "a_list1" {
DATATYPE H5T_STD_I64LE
DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
DATA {
(0): 10
}
}
Unfortunately, netcdf4-python saves a number as an array of size 1, so it is not possible to differentiate these two different things and one can easily obtain bad behavior.
This script demonstrates the issue:
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.12"
# dependencies = [
# "h5netcdf",
# "h5py",
# "netcdf4",
# ]
# ///
"""
# use netCDF4 (bug)
$ ./bad_behavior.py net
# use h5py (works fine)
$ ./bad_behavior.py
"""
import sys
import subprocess
if len(sys.argv) > 1:
package = sys.argv[-1]
else:
package = "h5py"
if package not in ["h5py", "h5netcdf"] and not package.startswith("net"):
raise ValueError(f"wrong option '{package}'")
if package == "h5py":
from h5py import File
path = "data.h5"
elif package == "h5netcdf":
import h5netcdf
File = h5netcdf.File
print(f"h5netcdf {h5netcdf.__version__}")
path = "data.nc"
else:
import netCDF4
File = netCDF4.Dataset
print(f"netCDF4 {netCDF4.__version__}")
path = "data_netCDF4.nc"
number = 10
if package.startswith("h5"):
with File(path, "w") as file:
file.attrs["a_int"] = number
file.attrs["a_list1"] = [number]
file.attrs["a_list2"] = [number, 2 * number]
else:
with File(path, "w") as file:
file.a_int = number
file.a_list1 = [number]
file.a_list2 = [number, 2 * number]
subprocess.run(["h5dump", str(path)], check=True)
if package.startswith("h5"):
with File(path, "r") as file:
attrs = dict(file.attrs)
else:
with File(path, "r") as file:
attrs = {name: file.getncattr(name) for name in file.ncattrs()}
print(attrs)
assert len(attrs["a_list2"]) == 2
assert attrs["a_list2"][0] == attrs["a_int"]
# TypeError: object of type 'numpy.int64' has no len()
assert len(attrs["a_list1"]) == 1
# IndexError: invalid index to scalar variable.
assert attrs["a_list1"][0] == attrs["a_int"]It would be great if numbers could be saved as scalars and lists of size 1 saved as SIMPLE { ( 1 ) / ( 1 ) }.
Related to h5netcdf/h5netcdf#301
CC @kmuehlbauer, @shoyer
Activity
jswhit commentedon Nov 7, 2025
netcdf4-python uses nc_put_att from the netcdf-c API to write attributes. There doesn't seem to be any other way to write attributes in netcdf-c.
jswhit2 commentedon Nov 10, 2025
This discussion in netcdf-c is relevant. In the last comment @DennisHeimbigner suggests that the netcdf spec requires all attributes to be treated as a vector of values, even for a single element vector. This is consistent with the behavior of the the
nc_put_att*functions. The exception seems to benc_put_att_textwhich writes a text attribute as a scalar.ChrisBarker-NOAA commentedon Feb 18, 2026
which may be because back in the day, there was no string type, only an array of char.
kmuehlbauer commentedon Feb 19, 2026
@ChrisBarker-NOAA That might just be the cause. From your experience, do you know of any solutions workarounds to suggest to the OP? It looks like there is no immediate solution to the OP's problem.
Would you think it might be possible for NetCDF4 to adopt scalar attributes? AFAIK, scalar variables exist, so it is at least not totally off.
jswhit commentedon Apr 25, 2026
I don't think we can unless the netcdf-c api provides a way to do it.
ChrisBarker-NOAA commentedon Apr 26, 2026
From that issue on the netCDF-C project:
In short -- netCDF-4 is the netcdf API (spec) encoded as HDF5 -- but it is a more restricted data model than HDF5 -- so users should not expect that it can maintain all details of an HDF5 file.
So it's not just the netcdf-c API, but the spec itself, there is no scalar for numerical attributes in netCDF.
I can't quickly see what the
h5netcdfdoes in this case from the OP's script, but if it is trying to be netCDF, then it should always use an array of length one, rather than a scalar.That is,
netCDF4andh5netcdfshould behave identically.As I noted, I think the scalar string thing is due to netCDF adding an additional string type, so it has a separate API for strings vs array of char.
Note that netcdf could add scalar attributes in a future version, but that would be another distinction between the netcdf4 and "classic" data models -- maybe not worth it?
I've been surprised how many people are still using netCDF3 ....