Repository navigation
Attribute style access is slow #4741
Description
I appreciate xarray's ability to use attribute style access ds.foo as an alternative to ds["foo"] as it requires less characters/keystrokes and has less 'visual clutter'.
A drawback is that it can be much slower as lookup time seems to display O(n) behaviour instead of O(1) with n being the number of variables in the dataset. For e.g. n=100 it is approximately 100 times slower than dictionary-style access:
# Dataset with many (100) variables
ds = xr.Dataset({f'var{v}': [] for v in range(100)})
%timeit ds['var0']
462 µs ± 1.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit ds.var0
47.1 ms ± 205 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
dir() and _ipython_key_completions_() which are used for e.g. tab completion in iPython are equally slow:
%timeit dir(ds)
47 ms ± 163 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit ds._ipython_key_completions_()
46.8 ms ± 210 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
I would like to see xarray having much better performance for attribute style access.
Activity
rhkleijn commentedon Dec 30, 2020
With some modest refactoring in https://github.com/rhkleijn/xarray/tree/faster-attr-access I managed to speed up attribute style access,
dir()and_ipython_key_completions_(in this case ~100 fold) by using a more lazy approach and especially avoiding the eager{d: self[d] for d in self.dims}which constructs many (mostly unneeded)DataArrayobjects.Shall I open a pull request for this?
dcherian commentedon Dec 30, 2020
Yes please! This looks great!