Repository navigation
NaN-sized chunks #2801
Description
It would be nice to have support for NaN-sized dask chunks, e.g. x[x > 2].
There are two problems:
x[x > 2]silently resolves the dask graph. It definitely shouldn't. There needs to be some discussion on what needs to happen to indices on the NaN-sized dimension; I can think of 3 options:
- silently drop any index that would become undefined
- drop any index that would become undefined and issue a warning
- hard crash if there is any index that would become undefined
- redesign IndexVariable so that it can contain dask data (probably much more complicated than the 3 above).
The above design decision is anyway for when there is an index; dims without indices should just work.
- This crashes:
>>> xarray.DataArray(a.data[a.data > 2]).compute()
ValueError: replacement data must match the Variable's shape
I didn't investigate but I suspect it should be trivial to fix. I'm not sure why there is a check at all? Any such health check should be in dask only IMHO.
Activity
fujiisoup commentedon Mar 5, 2019
Currently, xarray's indexing does not work for dask-indexers.
At least, there is a TODO
xarray/xarray/core/variable.py
Lines 555 to 560 in c33dab2
but I am not sure if it is the only one thing to prevent this.
I am not sure whether it is our issue or that in upstream.
does not clash, indicating dask can not take xr.DataArray as an index (note,
a < 2is an xr.DataArray).shoyer commentedon Mar 5, 2019
Yes, agreed.
The simple answer is to not allow undefined indices on NaN sized dimensions for now ("hard crash"). I don't think silently dropping indexes is a good idea.
Long term (after #1603 is complete), I can imagine supporting some sort of "lazy index" class.