Repository navigation
DataArray.drop_isel / .drop_sel with duplicated initial time stamp - InvalidIndexError #6605
Description
What is your issue?
I have a DataArray ds with 7098 time steps in cftime format. One of these time steps is a duplicate (pos: 7087 & 7088).
I tried to drop the first instance using both ds.drop_isel(time=[7087]) and ds.drop_sel(ds.time[7087]) but get the error:
pandas.errors.InvalidIndexError: Reindexing only valid with uniquely valued Index objects.
The method works if I do not have the duplicate in it. Maybe this is the wrong method in the first place but it seemed to be the way to go. I am confused on how to get rid of the duplicate, and, if the method is the correct method, if this might be an issue.
ds
Out[247]:
<xarray.DataArray 'LST_Night_CMG' (time: 3, y: 2, x: 2)>
array([[[14357., 14357.],
[14342., 14358.]],
[[14409., 14409.],
[14435., 14388.]],
[[14409., 14409.],
[14435., 14388.]]])
Coordinates:
* time (time) object 2019-09-18 00:00:00 ... 2019-09-19 00:00:00
* x (x) float64 67.98 68.03
* y (y) float64 40.12 40.07
spatial_ref int32 0
Attributes:
coordinates: spatial_ref band
scale_factor: 1.0
add_offset: 0.0
ds.time
Out[249]:
<xarray.DataArray 'time' (time: 3)>
array([cftime.DatetimeProlepticGregorian(2019, 9, 18, 0, 0, 0, 0, has_year_zero=True),
cftime.DatetimeProlepticGregorian(2019, 9, 19, 0, 0, 0, 0, has_year_zero=True),
cftime.DatetimeProlepticGregorian(2019, 9, 19, 0, 0, 0, 0, has_year_zero=True)],
dtype=object)
Coordinates:
* time (time) object 2019-09-18 00:00:00 ... 2019-09-19 00:00:00
spatial_ref int32 0
ds.drop_isel(time=1)
Traceback (most recent call last):
File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\IPython\core\interactiveshell.py", line 3444, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-248-0f89b7589e90>", line 1, in <module>
ds.drop_isel(time=1)
File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\xarray\core\dataarray.py", line 2413, in drop_isel
dataset = dataset.drop_isel(indexers=indexers, **indexers_kwargs)
File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\xarray\core\dataset.py", line 4565, in drop_isel
ds = ds.loc[dimension_index]
File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\xarray\core\dataset.py", line 563, in __getitem__
return self.dataset.sel(key)
File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\xarray\core\dataset.py", line 2505, in sel
self, indexers=indexers, method=method, tolerance=tolerance
File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\xarray\core\coordinates.py", line 422, in remap_label_indexers
obj, v_indexers, method=method, tolerance=tolerance
File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\xarray\core\indexing.py", line 120, in remap_label_indexers
idxr, new_idx = index.query(labels, method=method, tolerance=tolerance)
File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\xarray\core\indexes.py", line 240, in query
indexer = get_indexer_nd(self.index, label, method, tolerance)
File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\xarray\core\indexes.py", line 142, in get_indexer_nd
flat_indexer = index.get_indexer(flat_labels, method=method, tolerance=tolerance)
File "C:\Users\einfa\anaconda3\envs\climate\lib\site-packages\pandas\core\indexes\base.py", line 3442, in get_indexer
raise InvalidIndexError(self._requires_unique_msg)
pandas.errors.InvalidIndexError: Reindexing only valid with uniquely valued Index objects
Activity
dcherian commentedon May 13, 2022
Can you try drop_duplicates please?
pohleric commentedon May 13, 2022
This works, thanks.
dcherian commentedon May 13, 2022
Thanks this looks like a bug to me.
I think the issue is we should not treat unindexed dims the same as an indexed dim.
Also
ds.sel(x=2)works, so it makes senseds.drop_sel(x=2)should also work though this will drop both elements withx=2.max-sixty commentedon May 13, 2022
Great find @dcherian !