How should xarray use/support sparse arrays?

I'm looking forward to being easily able to create sparse xarray objects from pandas: #3206

Are there other xarray APIs that could make good use of sparse arrays, or could make sparse arrays easier to use?

Some ideas:

to_sparse()/to_dense() methods for converting to/from sparse without requiring using .data
to_dataframe()/to_series() could grow options for skipping the fill-value in sparse arrays, so they can round-trip MultiIndex data back to pandas
Serialization to/from netCDF files, using some custom convention (see Sparse arrays #1375 (comment))

This is very exciting! In energy-economic research (unlike, e.g., earth systems research), data are almost always sparse, so first-class sparse support will be broadly useful.

I'm leaving a comment here (since this seems to be a meta-issue; please link from wherever else, if needed) with two example use-cases. For the moment, #3206 seems to cover them, so I can't name any specific additional features.

MESSAGEix is an energy systems optimization model framework, formulated as a linear program.
- Some variables have many dimensions, for instance, the input coefficient for a technology has the dimensions (node_loc, technology, year_vintage, year_active, mode, node_origin, commodity, level, time, time_origin).
  - In the global version of our model, the technology dimension has over 400 labels.
  - Often two or more dimensions are tied, eg technology='coal power plant' will only take input from (commodity='coal', level='primary energy'); all other combinations of (commodity, level) are empty for this technology.
  - So, this data is inherently sparse.
- For modeling research, specifying quantities in this way is a good design because (a) it is intuitive to researchers in this domain, and (b) the optimization model is solved using various LP solvers via GAMS, which automatically prune zero rows in the resulting matrices.
- When we were developing a dask/DAG-based system for model results post-processing, we wanted to use xarray, but had some quantities with tens of millions of elements that were less than 1% full. Here is some test code that triggered MemoryErrors using xarray. We chose to fall back on using a pd.Series subclass that mocks xarray methods.
In transportation research, stock models of vehicle fleets are often used.
- These models always have at least two time dimensions: cohort (the time period in which a vehicle was sold) and period(s) in which it is used (and thus consumes fuel, etc.).
- Since a vehicle sold in 2020 can't be used in 2015, these data are always triangular w.r.t. these two dimensions. (The dimensions year_vintage and year_active in example Added setup.py which runs unit tests as necessary. #1 above have the same relationship.)
- Once multiplied by other dimensions (technology; fuel; size or shape or market segment; embodied materials; different variables; model runs across various scenarios or input assumptions) the overhead of dense arrays can become problematic.

+1 for the introduction of to_sparse() / to_dense(), but let's please avoid the mistakes that were done with chunk(). DataArray.chunk() is extremely frustrating when you have non-index coords and, 9 times out of 10, you only want to chunk the data and you have to go through the horrid

a = DataArray(a.data.chunk(), dims=a.dims, coords=a.coords, attrs=a.attrs, name=a.name)

Exactly the same issue would apply to to_sparse().

Possibly we could define them as

class DataArray:
    def to_sparse(
        self,
        data: bool = True,
        coords: Union[Iterable[Hashable], bool] = False
    )

class Dataset:
    def to_sparse(
        self,
        data_vars: Union[Iterable[Hashable], bool] = True,
        coords: Union[Iterable[Hashable], bool] = False
    )

same for to_dense() and chunk() (the latter would require a DeprecationWarning for a few release before switching the default for coords from True to False - only to be triggered in presence of dask-backed coords).

As already mentioned in #3206, unstack(sparse=True) would be extremely useful.

As for NetCDF, instead of a bespoke xarray-only convention, wouldn't it be much better to push a spec extension upstream?

netCDF has a pretty low-level base spec, with conventions left to higher level docs like CF conventions. Fortunately, there does seems to be a CF convention that would be a good fit for for sparse data in COO format, namely the indexed ragged array representation (example, note the instance_dimension attribute). That's probably the right thing to use for sparse arrays in xarray.

Would it be feasible to use the contiguous ragged array spec or the gathering based compression when the COO coordinates are sorted? I think this could be very helpful for read efficiency, though I'm not sure if random writes were a requirement here.

I like the indexed ragged array representation because it maps directly into sparse’s COO format. I’m sure other formats would be possible, but they would also likely be harder to implement.

That's fair. I just think it would be useful to have an assurance that indices are sorted you read them. I don't see how to express this within the CF specs while still looking like a COO array though.

Yes, it would be useful (eventually) to have lazy loading of sparse arrays from disk, like we want we currently do for dense arrays. This would indeed require knowing that the indices are sorted.

I would be interested in trying to add PDAL (https://pdal.io/) support, is this something of interest in a similar way to the support for rasterio for dense arrays?

@jeliashi

Tagging @jeliashi for visibility/collaboration

Would it be possible that pd.{Series, DataFrame}.to_xarray() automatically creates a sparse dataarray - or we have a flag in to_xarray which allows controlling for this. I have a very sparse dataframe and everytime I try to convert it to xarray I blow out my memory. Keeping it sparse but logically as a DataArray would be fantastic.

We have a new "sparse=True" option in xarray.Dataset.from_dataframe for exactly this use case. Pandas's to_xarray() method just calls this method, so it would make sense to forward keyword arguments, too.

…

On Fri, Aug 30, 2019 at 11:53 AM firdaus janoos ***@***.***> wrote: Would it be possible that pd.{Series, DataFrame}.to_xarray() automatically creates a sparse dataarray - or we have a flag in to_xarray which allows controlling for this. I have a very sparse dataframe and everytime I try to convert it to xarray I blow out my memory. Keeping it sparse but logically as a DataArray would be fantastic — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3213?email_source=notifications&email_token=AAJJFVTD2IWWPE6RTSWPVLDQHFUDTA5CNFSM4ILGYGP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5SPPNI#issuecomment-526710709>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAJJFVWEOGEBGXV62QFYU6DQHFUDTANCNFSM4ILGYGPQ> .

I cloned the master branch and installed it using 'python setup.py develop'.

When I try to use the sparse data loading functionality as per

oo = xa.Dataset.from_dataframe( my_df, sparse=True )

I get the following error:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-9-fce0ca6bc4c2> in <module>
----> 1 oo = xa.Dataset.from_dataframe( poly_df.iloc[:10000], sparse=True )

/mnt/local/xarray/xarray/core/dataset.py in from_dataframe(cls, dataframe, sparse)
   4040 
   4041         if sparse:
-> 4042             obj._set_sparse_data_from_dataframe(dataframe, dims, shape)
   4043         else:
   4044             obj._set_numpy_data_from_dataframe(dataframe, dims, shape)

/mnt/local/xarray/xarray/core/dataset.py in _set_sparse_data_from_dataframe(self, dataframe, dims, shape)
   3936         self, dataframe: pd.DataFrame, dims: tuple, shape: Tuple[int, ...]
   3937     ) -> None:
-> 3938         from sparse import COO
   3939 
   3940         idx = dataframe.index

ModuleNotFoundError: No module named 'sparse'

Any suggestions on what I need to do ?

conda install -c conda-forge sparse

Basically you need to install https://sparse.pydata.org/en/latest/ using either pip or conda.

According to test_sparse.py it looks like XArray already supports sparse, even though the XArray docs doesn't mention this support. Can we expect sparse to gain first-class support in future? Or is the absence of mention in the docs a hint we shouldn't rely on this functionality?

that's mostly an oversight, I think. However, to be really useful we'd need to get a sparse-xarray library which makes working with sparse and xarray easier (going from dense to sparse or the reverse still requires something like da.copy(data=sparse.COO.from_numpy(da.data)), which is not user-friendly).

Anyways, the docs you're looking for is working with numpy-like arrays, even though there's no explicit mention of sparse there, either.

Thanks so much! Appreciate it.

@stavrospapadopoulos

I would prefer to retain the dense representation, but with tricks to keep the data of sparse type in memory.

Look at the following example with pandas multiindex & sparse dtype:

The dense data uses ~40 MB of memory, while the dense representation with sparse dtypes uses only ~0.5 kB of memory!

And while you can import dataframes with the sparse=True keyword, the size seems to be displayed inaccurately (both are the same size?), and we cannot examine the data like we can with pandas multiindex + sparse dtype:

Besides, a lot of operations are not available on sparse xarray data variables (i.e. if I wanted to group by price level for ffill & downsampling):

So, it would be nice if xarray adopted pandas’ approach of unstacking sparse data.

In the end, you could extract all the non-NaN values and write them to a sparse storage format, such as TileDB sparse arrays.
cc: @stavrospapadopoulos

For ffill specifically, you would get a dense array out anyway, so there's no point to keeping it sparse, unless one did something like run-length-encoding or similar.

As for the size issue, PyData/Sparse provides the nbytes attribute which could be helpful in determining size.

I know. But having sparse data I can treat as if it were dense allows me to unstack without running out of memory, and then ffill & downsample the data in chunks:

It would be nice if xarray automatically converted the data from sparse back to dense for doing operations on the chunks just like pandas does.

The picture shows that I'm already using nbytes to determine the size.

Hi all! As part of a research project, I'm looking to contribute to xArray's sparse capabilities, with an emphasis on sparse support for use-cases in the geosciences. I'm wondering if anyone in the geosciences (or adjacent disciplines!) has encountered problems with xArray's current level of sparse support, and what kinds of improvements they'd like to see to address those issues. From playing around, it seems the current strategy of backing DataArrays with COO sparse arrays takes care of a lot of use cases, but I have the following ideas that may (or may not) be useful to implement further:

Functions/methods to convert from geopandas GeoDataframes of vector data to rasterized, potentially sparse ndarrays in an xArray Dataset/DataArray (reverse direction too); this is related to the issue of converting from sparse arrays back to multi-indexed pandas objects at the top of this issue (which I believe has yet to be solved)
Loading sparse data from a netcdf() file directly into a Dataset/Array backed by sparse ndarray(s) (seems like the only way to get sparse backings is to either unstack or call '.from_dataframe()/series()' with the sparse flag set to True?)
Support for other sparse array conventions (for ex, GCXS in the sparse package for better memory efficiency; I can't find any improvements to make on the current COO backing in terms of supported arithmetic operations, merges/joins, etc.)

I'd appreciate any feedback on these ideas, as well as any other things that would be nice to have implemented!

@JDButler

Hi @JDButler and welcome! We would welcome this sort of contribution eagerly.

I would characterize our current support of sparse arrays as really just a proof of concept. When to use sparse and how to do it effectively is not well documented. Simply adding more documentation around the already-supported use cases would be a great place to start IMO.

My own exploration of this are described in this Pangeo post. The use case is regridding. It touches on quite a few of the points you're interested in, in particular the integration with geodataframe. Along similar lines, @dcherian has been working on using opt_einsum together with sparse in pangeo-data/xESMF#222 (comment) and #7764.

I'd also suggest catching up on what @martinfleis is doing with vector data cubes in xvec. (See also Pangeo post on this topic.)

Of the three topics you enumerated, I'm most interested in the serialization one. However, I'd rather see serialization of sparse arrays prototyped in Zarr, as its much more conducive to experimentation than NetCDF (which requires writing C to do anything custom). I would recommend exploring serialization from a sparse array in memory to a sparse format on disk via a custom codec. Zarr recently added support for a meta_array parameter that determines what array type is materialized by the codec pipeline (see zarr-developers/zarr-python#1131). The use case there was loading data direct to GPU. In a way sparse is similar--it's an array container that is not numpy or dask.

@jbbutler

@jbbutler please also see this comment et seq. pydata/sparse#1 (comment) and related pydata/sparse#438.

To add to @rabernat's point about sparse support being "not well documented", I suspect (but don't know, as I'm just a user of xarray, not a developer) that it's also not thoroughly tested. I expected to be able to use e.g. DataArray.cumprod when the underlying data was sparse, but could not.

IMHO, I/O to/from sparse-backed objects is less valuable if only a small subset of xarray functionality is available on those objects. Perhaps explicitly testing/confirming which parts of the API do/do not currently work with sparse would support the improvements to the docs that Ryan mentioned, and reveal the work remaining to provide full(er) support.

Speaking a bit to things like cumprod, it's hard to support those natively with sparse data structures in many cases (at least as things stand in the current Numba framework).

While that doesn't apply in the case of cumprod, PyData/Sparse also has a policy that if the best algorithm available is a dense one, we simply raise an error, and the user should densify explicitly to avoid filling all available RAM or getting obscure MemoryErrors.

That's a totally valid scope limitation for the sparse package, and I understand the motivation.

I'm just saying that the principle of least astonishment is not being followed: the user cannot at the moment read either the xarray or sparse docs and know which portions of the xarray API will work when giving …, sparse=True, and which instead require a deliberate choice to densify, or see examples of how best to mix the two. It would be helpful to clarify—that's all.

I suspect (but don't know, as I'm just a user of xarray, not a developer) that it's also not thoroughly tested.

Existing sparse testing is here: https://github.com/pydata/xarray/blob/main/xarray/tests/test_sparse.py

We would welcome enhancements to this!

Thank you all so much for the feedback and resources! I agree (1) testing the limits of xArray's API compatibility with sparse and (2) developing some documentation for what is/isn't supported are great places to start, so I'll get on that while I think about the other I/O issues (serialization, etc.)

A use case example: https://ncar.github.io/esds/posts/2022/sparse-PFT-gridding/

Navigation Menu

Repository navigation

How should xarray use/support sparse arrays? #3213

26 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

How should xarray use/support sparse arrays? #3213

Description

Activity

khaeru commented on Aug 13, 2019

crusaderky commented on Aug 14, 2019

crusaderky commented on Aug 14, 2019

crusaderky commented on Aug 14, 2019

shoyer commented on Aug 14, 2019

ivirshup commented on Aug 15, 2019

shoyer commented on Aug 15, 2019

ivirshup commented on Aug 15, 2019

shoyer commented on Aug 15, 2019

normanb commented on Aug 22, 2019

darothen commented on Aug 22, 2019

fjanoos commented on Aug 30, 2019

shoyer commented on Aug 30, 2019

fjanoos commented on Aug 30, 2019

dcherian commented on Aug 30, 2019

26 remaining items

scottgigante-immunai commented on Oct 14, 2021

keewis commented on Oct 14, 2021

scottgigante-immunai commented on Oct 14, 2021

Material-Scientist commented on Jan 16, 2022

hameerabbasi commented on Jan 17, 2022

Material-Scientist commented on Jan 17, 2022

jbbutler commented on May 3, 2023

rabernat commented on May 4, 2023

khaeru commented on May 4, 2023

hameerabbasi commented on May 4, 2023

khaeru commented on May 4, 2023

rabernat commented on May 4, 2023

jbbutler commented on May 12, 2023

dcherian commented on Jun 7, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions