Repository navigation
Comprehensive benchmarking suite #4648
Description
I think a good "infrastructure" target for the NASA OSS call would be to expand our benchmarking suite (https://pandas.pydata.org/speed/xarray/#/)
AFAIK running these in a useful manner on CI is still unsolved (please correct me if I'm wrong). But we can always run it on an NCAR machine using a cron job.
Thoughts?
cc @scottyhq
A quick survey of work needed (please append):
- indexing & slicing Improve indexing performance benchmarks #3382 Performance: numpy indexes small amounts of data 1000 faster than xarray #2799 Slow performance of isel #2227
- DataArray construction Speed up Dataset._construct_dataarray #4744
- attribute access Attribute style access is slow #4741, speedup attribute style access and tab completion #4742
- property access Should we cache some small properties? #3514
- reindexing? slow performance with open_mfdataset #1385 (comment)
- alignment Performance problem when doing computation between two arrays with discontinuous indexes #3755, [skip-ci] Add alignment benchmarks #7738
- assignment Needs performance check / improvements in value assignment of DataArray #1771
- coarsen Optimize padding for coarsening. #10921
- groupby groupby very slow compared to pandas #659 [skip-ci] Add cftime groupby, resample benchmarks #7795 Speed up .dt accessor by preserving Index objects. #7796
- resample Resample is ~100x slower than Pandas resample; Speed is related to resample period (unlike Pandas) #4498 [skip-ci] Add cftime groupby, resample benchmarks #7795
- weighted Allow skipna in .dot() #4482 weighted operations: performance optimisations #3883
- concat Improve concat performance #7824, Minimize concat memory usage #10866
- merge
- open_dataset, open_mfdataset We need a fast path for open_mfdataset #1823
- stack / unstack
- apply_ufunc?
- interp Faster interp #4740 Improve interp performance #7843
- reprs Speed up Dataset._construct_dataarray #4744
- to_(dask)_dataframe Improve to_dask_dataframe performance #7844 Add benchmarks for to_dataframe and to_dask_dataframe #7474
Related: #3514
Reacted by Maximilian Roos
Activity
scottyhq commentedon Dec 3, 2020
thanks for the ping @dcherian, i really like the idea! One other thing that often gets neglected in test suites is operating on remote data. I understand the need to avoid long-running tests and tests prone to network failures for PRs, but running these sorts of examples as a cron job could be very helpful for benchmarking and detecting issues.
In intake-xarray we recently added tests against a local HTTP server and "S3" server:
https://github.com/intake/intake-xarray/blob/master/intake_xarray/tests/test_remote.py
Also added several simple tests requiring a network connection to public data (no auth required) that we run locally but not in CI currently:
https://github.com/intake/intake-xarray/blob/master/intake_xarray/tests/test_network.py
dcherian commentedon Dec 4, 2020
Thanks @scottyhq
This is lining up with the "pangeo integration tests" that came up in a Pangeo meeting (cc @rabernat).
Regardless whether it fits, I think adding benchmarks+tests for the xarray+zarr+fsspec (or xarray+mfdataset+netCDF) is an important and unmet need of the Pangeo community in general that we could address.
max-sixty commentedon Dec 30, 2020
This would be great.
Down a couple of levels — I think potentially we could run this as a cron job on GitHub Actions. NCAR would also be a good plan. I'm also happy to supply a VM if that's helpful.
dcherian commentedon Aug 18, 2021
Looks like Quansight thinks that GH actions is a good place to benchmark scikit-learn: https://labs.quansight.org/blog/2021/08/github-actions-benchmarks/ so may be we can set that up for our existing benchmarks.
Here's the workflow: https://github.com/jaimergp/scikit-image/blob/main/.github/workflows/benchmarks-cron.yml
dcherian commentedon Nov 8, 2021
@TomAugspurger are you still in charge of the pydata benchmarking machine? If so, could you add xarray to the list please (https://pandas.pydata.org/speed/)? @Illviljan has made major improvements so it should be a lot faster now
TomAugspurger commentedon Nov 9, 2021
"In charge of" is overstating it a bit. It's been segfaulting when building pandas and I haven't had a chance to debug it.
If / when I get around to fixing it I'll try adding xarray, but it might be a bit.