Skip to content

Resample is ~100x slower than Pandas resample; Speed is related to resample period (unlike Pandas) #4498

New issue
New issue
Closed
Closed
Resample is ~100x slower than Pandas resample; Speed is related to resample period (unlike Pandas)#4498
Labels
topic-performance

Description

@mankoff
mankoff
opened on Oct 9, 2020

What happened:

I have a 10 minute frequency time series. When I resample to hourly it is slow. When I resample to daily it is fast. If I drop to Pandas and resample the speeds are ~100x faster than xarray, and also the same time regardless of the resample period. I've posted this to SO: https://stackoverflow.com/questions/64282393/

What you expected to happen:

I expect xarray to be within an order of magnitude speed of Pandas, not > 2 orders of magnitude slower.

Minimal Complete Verifiable Example:

import numpy as np
import xarray as xr
import pandas as pd
import time

size = 10000
times = pd.date_range('2000-01-01', periods=size, freq="10Min")
da = xr.DataArray(data = np.random.random(size), dims = ['time'], coords = {'time': times}, name='foo')

start = time.time()
da_ = da.resample({'time':"1H"}).mean()
print("1H", 'xr', str(time.time() - start))

start = time.time()
da_ = da.to_dataframe().resample("1H").mean()
print("1H", 'pd', str(time.time() - start), "\n")


start = time.time()
da_ = da.resample({'time':"1D"}).mean()
print("1D", 'xr', str(time.time() - start))

start = time.time()
da_ = da.to_dataframe().resample("1D").mean()
print("1D", 'pd', str(time.time() - start))

Output/timings

: 1H xr 0.1761918067932129
: 1H pd 0.0021948814392089844 
: 
: 1D xr 0.00958395004272461
: 1D pd 0.001646280288696289

Anything else we need to know?:

Environment:

Output of xr.show_versions()

xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.8.5 | packaged by conda-forge | (default, Aug 21 2020, 18:21:27)
[GCC 7.5.0]
python-bits: 64
OS: Linux
OS-release: 5.4.0-48-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.6
libnetcdf: 4.7.4

xarray: 0.16.0
pandas: 1.1.1
numpy: 1.19.1
scipy: 1.5.2
netCDF4: 1.5.4
pydap: None
h5netcdf: 0.8.1
h5py: 2.10.0
Nio: None
zarr: None
cftime: 1.2.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.1.5
cfgrib: None
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: 3.3.1
cartopy: None
seaborn: None
numbagg: None
pint: 0.15
setuptools: 49.6.0.post20200814
pip: 20.2.2
conda: None
pytest: None
IPython: 7.17.0
sphinx: None

Activity

  1. dcherian commented on Oct 9, 2020

    @dcherian
    dcherian
    on Oct 9, 2020
    Contributor

    @mankoff (hi!) This is interesting. If I comment out .mean() I get

    1H xr 0.003030538558959961
1H pd 0.0014064311981201172 

1D xr 0.0026717185974121094
1D pd 0.0013244152069091797

    i.e. we are 2x slower just on factorizing.

  2. dcherian
    added
    topic-performance
    on Oct 9, 2020

  3. mankoff commented on Oct 10, 2020

    @mankoff
    mankoff
    on Oct 10, 2020
    ContributorAuthor

    "performance" is a good tag. My actual use case is a dataset with 500,000 timestamps and 15 variables (10 minute weather station for a decade).

    In this case, pandas takes 0.03 seconds, and xarray takes 200 seconds. 4 orders of magnitude. Should I change the title to reflect the larger difference in performance? Here is that MWE:

    import numpy as np
import xarray as xr
import pandas as pd
import time

size = 500000
times = pd.date_range('2000-01-01', periods=size, freq="10Min")
ds = xr.Dataset({
    'foo': xr.DataArray(
        data   = np.random.random(size),
        dims   = ['time'],
        coords = {'time': times}
    )})
for v in 'abcdefghijelm':
    ds[v] = (('time'), np.random.random(size))

start = time.time()
ds_r = ds.resample({'time':"1H"}).mean()
print('xr', str(time.time() - start))

start = time.time()
ds_r = ds.to_dataframe().resample("1H").mean()
print('pd', str(time.time() - start))

    Result:

    xr 202.2967929840088
pd 0.03381085395812988

    The strange thing here is if I drop the .mean()'s, most of the time I see what you see.

    : xr 0.03333306312561035
: pd 0.020237445831298828

    But every 4th or 5th time that I run this, I get this:

    : xr 0.8518760204315186
: pd 0.02686452865600586

    This is repeatable. I've Run this code 100s of times now, and every 4th or 5th run it takes 10x. Nothing else is going on on my computer.

  4. mankoff commented on Oct 10, 2020

    @mankoff
    mankoff
    on Oct 10, 2020
    ContributorAuthor

    The every 4th or 5th lag is not in the creation, it's in the resample:

    #+BEGIN_SRC jupyter-python :kernel ds :session bugreport
for i in np.arange(25):
    start = time.time()
    ds_r = ds.resample({'time':"1H"})
    print('xr', str(time.time() - start))

#+END_SRC

#+RESULTS:
#+begin_example
xr 0.04479050636291504
xr 0.047682762145996094
xr 0.8904871940612793
xr 0.05605506896972656
xr 0.0452876091003418
xr 0.0467374324798584
xr 0.8709239959716797
xr 0.05595755577087402
xr 0.046492576599121094
xr 0.04648017883300781
xr 0.045223236083984375
xr 0.8187246322631836
xr 0.05060911178588867
xr 0.04763054847717285
xr 0.8156075477600098
xr 0.055490970611572266
xr 0.047312259674072266
xr 0.04651069641113281
xr 0.8001837730407715
xr 0.05546212196350098
xr 0.04549074172973633
xr 0.04680013656616211
xr 0.04383039474487305
xr 0.7662224769592285
xr 0.04914355278015137
#+end_example

  5. shoyer commented on Oct 11, 2020

    @shoyer
    shoyer
    on Oct 11, 2020
    Member

    resample uses the same machinery in xarray as other grouped aggregations.

    Right now, grouped aggregations are very slow when there are many groups (like in resample) because we use a Python loop over groups.

    Probably the most obvious way to speed this up would be to wrap the "numpy-groupies" package in xarray: #4473

  6. shoyer commented on Oct 11, 2020

    @shoyer
    shoyer
    on Oct 11, 2020
    Member

    I might add that this is somewhat I've wanted to speed-up in xarray since the very early days. But until I noticed the numpy-groupies package, it seemed like a pretty challenging task.

  7. mankoff commented on Oct 11, 2020

    @mankoff
    mankoff
    on Oct 11, 2020
    ContributorAuthor

    Thanks for the clarification that this is a real issue not due to just my coding, and the suggestion to solve this elsewhere. For now I just use the fast Pandas version with this code:

    df_h = ds.to_dataframe().resample("1H").mean()  # what we want (quickly), but in Pandas form
vals = [xr.DataArray(data=df_h[c], dims=['time'], coords={'time':df_h.index}, attrs=ds[c].attrs) for c in df_h.columns]
ds_h = xr.Dataset(dict(zip(df_h.columns,vals)), attrs=ds.attrs)

  8. mankoff commented on Oct 11, 2020

    @mankoff
    mankoff
    on Oct 11, 2020
    ContributorAuthor

    The linked issues refer to groupby not resample so this could stay open or be closed as a duplicate - I leave it to you to decide. Thank you for the assistance.

  9. mankoff
    added a commit that references this issue on Oct 11, 2020

    Fast resample using Pandas

    a4f0da9
  10. dcherian
    added 3 commits that reference this issue on May 13, 2022

    Enable `flox` in `GroupBy` and `resample` (pydata#5734)

    9caa760

    Enable `flox` in `GroupBy` and `resample` (pydata#5734)

    7043bdb

    Enable flox in GroupBy and resample (pydata#5734)

    e02b1c3
  11. dcherian
    closed this as completedin e02b1c3on May 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    topic-performance

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions