weighted operations: performance optimisations #3883

Activity

1. mathause commented on Mar 25, 2020

   

   mathause

   on Mar 25, 2020

   CollaboratorAuthor

   More actions

   maybe relevant: #1995
2. mathause commented on May 18, 2020

   

   mathause

   on May 18, 2020

   Last edited by mathause

   CollaboratorAuthor

   More actions

   %load_ext line_profiler
   
   
   import numpy as np
   import xarray as xr
   
   from xarray.core.weighted import Weighted as w
   
   shape_weights = (1000, 1000)
   shape_data = (1000, 1000, 10)
   add_nans = False
   
   def lprun_weighted(shape_weights, shape_data, add_nans, skipna=None):
   
       weights = xr.DataArray(np.random.randn(*shape_weights))
   
       data = np.random.randn(*shape_data)
   
       # add approximately 25 % NaNs
       if add_nans:
           c = int(data.size * 0.25)
           data.ravel()[np.random.choice(data.size, c, replace=False)] = np.NaN
   
       data = xr.DataArray(data)
   
   
       return data.weighted(weights).mean(skipna=skipna)
   
   
   
   %lprun -f w._reduce -f w._weighted_mean -f w._sum_of_weights -f w._weighted_sum -f w.__init__ -f lprun_weighted -u 1e-03 lprun_weighted(shape_weights, shape_data, add_nans, skipna=None)
3. mathause commented on May 26, 2020

   

   mathause

   on May 26, 2020

   Last edited by mathause

   CollaboratorAuthor

   More actions

   weighted(weights).mean(skipna=True) calls

   xarray/xarray/core/weighted.py

   Line 143 in d1f7cb8

   mask = da.notnull()

   and

   xarray/xarray/core/weighted.py

   Line 130 in d1f7cb8

   da = da.fillna(0.0)

   da.fillna(0.0) in turn calls where(null(data), other). Thus null/ notnull is called twice. This could be optimized by doing self.null = obj.null() in __init__(...). This might also allow self.any_null = self.null.any() to skip NaN handling if there are none. However, this needs more thinking if a Dataset is passed. Probably overkill but leaving this here for reference.
4. dcherian

   added

   topic-performance

   on Jul 4, 2021