Repository navigation
NDMeta and NDCollection can not be pickled #847
Description
Describe the bug
When trying to use NDMeta (which is the default with 2.3) with dask distributed you get errors because NDMeta can not be pickled.
Same for NDCollection.
To Reproduce
import ndcube
import pickle
meta = ndcube.NDMeta({"hello": "world"})
print(meta)
pik = pickle.dumps(meta)
meta2 = pickle.loads(pik)---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
File <string>:8
File ~/Git/ndcube/ndcube/meta.py:298, in NDMeta.__setitem__(self, key, val)
297 def __setitem__(self, key, val):
--> 298 axis = self.axes.get(key, None)
299 if axis is not None:
300 if _not_scalar(val):
File ~/Git/ndcube/ndcube/meta.py:227, in NDMeta.axes(self)
225 @property
226 def axes(self):
--> 227 return self._axes
AttributeError: 'NDMeta' object has no attribute '_axes'
linewidth_data = np.random.rand(4, 4) / 2 # dummy data
linewidth_wcs = astropy.wcs.WCS(naxis=2)
linewidth_wcs.wcs.ctype = 'HPLT-TAN', 'HPLN-TAN'
linewidth_wcs.wcs.cunit = 'deg', 'deg'
linewidth_wcs.wcs.cdelt = 0.5, 0.4
linewidth_wcs.wcs.crpix = 2, 2
linewidth_wcs.wcs.crval = 0.5, 1
linewidth_cube = ndcube.NDCube(linewidth_data, linewidth_wcs)
ncol = ndcube.NDCollection([("test", linewidth_wcs)])
pik = pickle.dumps(ncol)
ncol2 = pickle.loads(pik)---------------------------------------------------------------------------
NotImplementedError Traceback (most recent call last)
File <string>:25
File ~/Git/ndcube/ndcube/ndcollection.py:335, in NDCollection.__setitem__(self, key, value)
334 def __setitem__(self, key, value):
--> 335 raise NotImplementedError("NDCollection does not support __setitem__. "
336 "Use NDCollection.update instead")
NotImplementedError: NDCollection does not support __setitem__. Use NDCollection.update instead
Screenshots
No response
System Details
No response
Installation method
No response
Activity
[-]`NDMeta` can not be pickled[/-][+]`NDMeta` and `NDCollection` can not be pickled[/+]DanRyanIrish commentedon Jun 12, 2025
I don't understand this bug for
NDMeta. It DOES have aself._axesattribute...DanRyanIrish commentedon Jun 12, 2025
Regarding
NDCollection, I suppose this means we would have to support__setitem__. Instead of raising an error, we could raise a warning saying that using__setitem__is discouraged, andNDCollection.updateis recommended.Cadair commentedon Jun 12, 2025
I suspect both of these are about subtitles in how unpickling works, I don't think it calls the
__init__. We can probably fix both of them by implementing things from the pickle interface: https://docs.python.org/3/library/pickle.html#pickle-inst or just refactoring weird things we do.DanRyanIrish commentedon Jun 12, 2025
So by implementing some (all?) of those methods, we need to somehow capture/handle the attributes set in the
__init__:self.meta,self._aligned_axes, andself.n_aligned_axes?If I'm right so far, it's not clear to me which of those methods we would need, or how we need to implement them.
Cadair commentedon Jun 12, 2025
I suspect the best thing to do would be to refactor the
__init__a bit. I'm not really sure how with out looking into it more.