Skip to content

Implement sparse.COO.pad #438

Closed
Closed

Description

@khaeru

Is your feature request related to a problem? Please describe.
I would like to be able to use the xarray.DataArray API when the data is sparse.COO.

Instead—as mentioned at #1 (comment) —it appears I currently need to write a custom subclass of DataArray that handles cases where sparse doesn't implement a method expected by xarray.

Describe the solution you'd like
For sparse.COO.pad and the other methods mentioned in the linked comment to be implemented so that, in general, it is possible to use the xarray.DataArray API, or at least all a certain advertised set of basic functions, with sparse data.

Describe alternatives you've considered
As mentioned, I am currently handling this with wrapper code that converts an xr.DataArray to dense data, performs operations, then re-converts the data to sparse.

Another mitigation would be for the sparse documentation to describe which DataArray operations are not supported.

Some example code:

import pandas as pd
import xarray as xr
from numpy import nan

idx = dict(
    columns=pd.Index(["x1", "x2", "x3", "x4"], name="x"),
    index=pd.Index(["y1", "y2", "y3", "y4"], name="y"),
)

# Simplified example data
s1 = pd.DataFrame(
    [
        [1.0, nan, nan, nan],
        [nan, nan, 2.0, nan],
        [nan, nan, nan, 3.0],
        [nan, 4.0, nan, nan],
    ],
    **idx
).stack()

print(s1)

da1 = xr.DataArray.from_series(s1, sparse=True)

print(da1)

# I would like to produce data like the following by using DataArray.shift()
s2 = pd.DataFrame(
    [
        [nan, 1.0, nan, nan],
        [nan, nan, nan, 2.0],
        [nan, nan, nan, nan],
        [nan, nan, 4.0, nan],
    ],
    **idx
).stack()

print(s2)

da2 = xr.DataArray.from_series(s2)

print(da2)

# This line raises an exception, below
da1.shift(x=1)

The exception raised:

Traceback (most recent call last):
  File "test.py", line 42, in <module>
    da1.shift(x=1)
  File "/home/khaeru/.local/lib/python3.8/site-packages/xarray/core/dataarray.py", line 3077, in shift
    variable = self.variable.shift(
  File "/home/khaeru/.local/lib/python3.8/site-packages/xarray/core/variable.py", line 1205, in shift
    result = result._shift_one_dim(dim, count, fill_value=fill_value)
  File "/home/khaeru/.local/lib/python3.8/site-packages/xarray/core/variable.py", line 1166, in _shift_one_dim
    data = duck_array_ops.pad(
  File "/home/khaeru/.local/lib/python3.8/site-packages/xarray/core/duck_array_ops.py", line 56, in f
    return wrapped(*args, **kwargs)
  File "<__array_function__ internals>", line 5, in pad
TypeError: no implementation found for 'numpy.pad' on types that implement __array_function__: [<class 'sparse._coo.core.COO'>]

Activity

  1. khaeru commented on May 11, 2021

    @khaeru
    Author

    Thanks @H4R5H1T-007 !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementIndicates new feature requests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions