Repository navigation
Return a scalar instead of DataArray when the return value is a scalar #987
Description
Hi,
I'm not sure how devs will feel about this, but I wanted to ask because I'm getting into this issue frequently.
Currently many methods such as .min(), .max(), .mean() returns a DataArray even for the cases where the return value is a scaler. For example,
import numpy as np
import xarray as xr
test = xr.DataArray(data=np.ones((10, 10)))
In [6]: test.min()
Out[6]:
<xarray.DataArray ()>
array(1.0)which makes a lot of other things break down and I have to use test.min().values or float(test.min()).
I think it would be great that these methods return a scalar when the return value is a scaler. For example,
In [7]: np.ones((10, 10)).mean()
Out[7]: 1.0Thank you!
Activity
shoyer commentedon Aug 26, 2016
I agree that this can be annoying. The downside in making this switch is that we would lose xarray specific fields like
coordsandattrsthat are currently preserved, e.g.,Also, strictly from a simplicity point of view for xarray, it's nice for every function to return fixed types.
NumPy solved this problem by creating it's own scalar types (e.g.,
np.float64) that define fields likeshapeanddtypewhile also subclassing Python's builtin numeric types. We could do the same, but this could lead to a different set of subtle cross-compatibility issues.joonro commentedon Aug 26, 2016
I see - thanks a lot for the quick response. I knew there was a good reason for this.
I wonder if it is reasonable to return a scalar when there is neither
coordsnorattrsassociated with the return value, or it would be too much ad-hoc thing. For example, in the original example the return value was<xarray.DataArray ()>, which does not have any useful information.I think this might be reasonable because I only get into this issue when I'm doing an array-wide operation and I know I'm going to get an aggregate scalar and forget to use
.values.shoyer commentedon Aug 26, 2016
This is a bad path to go down :). Now your code might suddenly break when you add a metadata field!
In principle, we could pick some subset of operations for which to always do this and others for which to never do this (e.g., aggregating out all dimensions, but not indexing out all dimensions), but I think this inconsistency would be even more surprising. It's pretty easy to see how this could lead to bugs, too. At least now you know you always need to type
.valuesor.item()!darothen commentedon Aug 27, 2016
@joonro, I think there's a strong case to be made about returning a
DataArraywith some metadata appended. Referring to the latest draft of the CF Metadata Conventions, there is a clear way to indicate when operations such asmean,max, orminhave been applied to a variable by using the cell_methods attribute.It might be more prudent to add this attribute whenever we apply these operations to a
DataArray(or perhaps variable-wise when applied to aDataset). That way, there is a clear reason to not return a scalar - the documentation of what operations were applied to produce that final result.I can whip up a working example/pull request if people think this is a direction to go. I'd probably build a decorator which handles inspection of the operator name and arguments and uses that to add the cell_methods attribute, that way people can add the same functionality to homegrown methods/operators.
shoyer commentedon Aug 27, 2016
@darothen Let's discuss this over in #988.
joonro commentedon Aug 27, 2016
Thanks a lot for the discussions. I agree it is very important to be consistent and explicit. Another thing was that sometimes
.valuesmakes a line of code really long - especially when I want to index aDataArraywith anotherDataArraywith some conditions, as I often have to use.valuesfor each of them.Currently I do not have a good idea about how to improve this - I will report back if one occurs to me. Thanks again!
shoyer commentedon Aug 27, 2016
Can you give an example of how you need to use .values in xarray
operations? Within xarray, we should be able to remove the need to use that.
On Sat, Aug 27, 2016 at 1:06 PM Joon Ro notifications@github.com wrote:
joonro commentedon Aug 27, 2016
Sure. My actual usage is usually much more complicated, but basically, with
if I want to choose only values larger than 0 from X, it seems I cannot do
X[X > 0], I have to doX.values[X.values > 0]. You can see how this thing can quickly get long if I'm doing this for assignment with multidimensional xarrays - something likeMaybe I'm mistaken and there is a way to do this more nicely, but I haven't been able to figure it out.
Thank you!
shoyer commentedon Aug 28, 2016
@joonro Yes, this does get messy. We'll eventually support indexing like
X[X > 0]directly, which will help significantly.In the meantime, you can still break things up onto multiple lines by saving temporary variables:
Using abbreviations like
...for:, :, :(assuming'variable'is along the last axis) can also help.joonro commentedon Aug 28, 2016
@shoyer I think I saw
...a long time ago and must have forgotten about it. Thank you so much for reminding me - I was really hoping for something like...for a while.Btw, I must say not only that xarray is just so useful for many of my research, but also the devs' responses on the issues have been superb. Definitely one of the most pleasant experiences I have had with developers. Thank you.
shoyer commentedon Aug 28, 2016
Thanks @joonro, you are very kind!
I'm going to close this issue since I think we resolved the original question.