round-trip performance with save_mfdataset / open_mfdataset #1340

Activity

1. shoyer commented on Mar 30, 2017

   

   shoyer

   on Mar 30, 2017

   Member

   More actions

   My strong suspicion is that the bottleneck here is xarray checking all the coordinates for equality in concat, when deciding whether to add a "time" dimension or not.

   Try passing coords='minimal' and see if that speeds things up. See the concat documentation for details:
   http://xarray.pydata.org/en/stable/generated/xarray.concat.html#xarray.concat

   This was a convenient check for small/in-memory datasets but possibly it's not a good one going forward. It's generally slow to load all the coordinate data for comparisons, but it's even worse with the current implementation, which computes pair-wise comparisons of arrays with dask instead of doing them in parallel all at once.
2. shoyer

   added

   topic-performance

   on Mar 30, 2017
3. rabernat commented on Mar 30, 2017

   

   rabernat

   on Mar 30, 2017

   ContributorAuthor

   More actions

   coords is not a valid kwarg for open_mfdataset

   http://xarray.pydata.org/en/latest/generated/xarray.open_mfdataset.html#xarray-open-mfdataset
4. shoyer commented on Mar 30, 2017

   

   shoyer

   on Mar 30, 2017

   Member

   More actions

   Indeed, it's not. We should add some way to pipe this arguments through auto_combine on to concat.

   👍React with 👍3Reacted by Ryan Abernathey, Jonas and Noah D. Brenowitz
5. rabernat commented on Mar 30, 2017

   

   rabernat

   on Mar 30, 2017

   ContributorAuthor

   More actions

   This sounds like the kind of thing I could manage.
6. karenamckinnon commented on Apr 26, 2017

   

   karenamckinnon

   on Apr 26, 2017

   More actions

   I'm running into the same problem as Ryan, regarding the ValueError. However, when I try the same fix

   ds = xr.open_mfdataset(files[:4], concat_dim = 'time')

   I get the error

   TypeError: Must pass list-like as names.

   Apologies if this should be on a different chain, but any idea what might be going on?
7. shoyer commented on Apr 26, 2017

   

   shoyer

   on Apr 26, 2017

   Member

   More actions

   @karenamckinnon could you please share a traceback for the error?
8. karenamckinnon commented on Apr 26, 2017

   

   karenamckinnon

   on Apr 26, 2017

   More actions

   TypeErrorTraceback (most recent call last)
   <ipython-input-152-52564f498ac3> in <module>()
   ----> 1 ds = xr.open_mfdataset(files[:2], concat_dim='time')
   
   /glade/u/home/mckinnon/.local/lib/python2.7/site-packages/xarray/backends/api.pyc in open_mfdataset(paths, chunks, concat_dim, preprocess, engine, lock, **kwargs)
       304         datasets = [preprocess(ds) for ds in datasets]
       305 
   --> 306     combined = auto_combine(datasets, concat_dim=concat_dim)
       307     combined._file_obj = _MultiFileCloser(file_objs)
       308     return combined
   
   /glade/u/home/mckinnon/.local/lib/python2.7/site-packages/xarray/core/combine.pyc in auto_combine(datasets, concat_dim)
       376     grouped = itertoolz.groupby(lambda ds: tuple(sorted(ds.data_vars)),
       377                                 datasets).values()
   --> 378     concatenated = [_auto_concat(ds, dim=concat_dim) for ds in grouped]
       379     merged = merge(concatenated)
       380     return merged
   
   /glade/u/home/mckinnon/.local/lib/python2.7/site-packages/xarray/core/combine.pyc in _auto_concat(datasets, dim)
       327                                  'explicitly')
       328             dim, = concat_dims
   --> 329         return concat(datasets, dim=dim)
       330 
       331 
   
   /glade/u/home/mckinnon/.local/lib/python2.7/site-packages/xarray/core/combine.pyc in concat(objs, dim, data_vars, coords, compat, positions, indexers, mode, concat_over)
       115         raise TypeError('can only concatenate xarray Dataset and DataArray '
       116                         'objects, got %s' % type(first_obj))
   --> 117     return f(objs, dim, data_vars, coords, compat, positions)
       118 
       119 
   
   /glade/u/home/mckinnon/.local/lib/python2.7/site-packages/xarray/core/combine.pyc in _dataset_concat(datasets, dim, data_vars, coords, compat, positions)
       206     dim, coord = _calc_concat_dim_coord(dim)
       207     datasets = [as_dataset(ds) for ds in datasets]
   --> 208     datasets = align(*datasets, join='outer', copy=False, exclude=[dim])
       209 
       210     concat_over = _calc_concat_over(datasets, dim, data_vars, coords)
   
   /glade/u/home/mckinnon/.local/lib/python2.7/site-packages/xarray/core/alignment.pyc in align(*objects, **kwargs)
        78     all_indexes = defaultdict(list)
        79     for obj in objects:
   ---> 80         for dim, index in iteritems(obj.indexes):
        81             if dim not in exclude:
        82                 all_indexes[dim].append(index)
   
   /glade/apps/opt/python/2.7.7/gnu-westmere/4.8.2/lib/python2.7/_abcoll.pyc in iteritems(self)
       385         'D.iteritems() -> an iterator over the (key, value) items of D'
       386         for key in self:
   --> 387             yield (key, self[key])
       388 
       389     def keys(self):
   
   /glade/u/home/mckinnon/.local/lib/python2.7/site-packages/xarray/core/coordinates.pyc in __getitem__(self, key)
       245     def __getitem__(self, key):
       246         if key in self:
   --> 247             return self._variables[key].to_index()
       248         else:
       249             raise KeyError(key)
   
   /glade/u/home/mckinnon/.local/lib/python2.7/site-packages/xarray/core/variable.pyc in to_index(self)
      1173             index = index.set_names(valid_level_names)
      1174         else:
   -> 1175             index = index.set_names(self.name)
      1176         return index
      1177 
   
   /glade/apps/opt/pandas/0.14.0/gnu/4.8.2/lib/python2.7/site-packages/pandas-0.14.0-py2.7-linux-x86_64.egg/pandas/core/index.pyc in set_names(self, names, inplace)
       380         """
       381         if not com.is_list_like(names):
   --> 382             raise TypeError("Must pass list-like as `names`.")
       383         if inplace:
       384             idx = self
   
   TypeError: Must pass list-like as `names`.
   
9. shoyer commented on Apr 26, 2017

   

   shoyer

   on Apr 26, 2017

   Member

   More actions

   @karenamckinnon From your traceback, it looks like you're using pandas 0.14, but xarray requires at least pandas 0.15.
10. karenamckinnon commented on May 1, 2017

    

    karenamckinnon

    on May 1, 2017

    More actions

    Got it, thanks @shoyer ! In case this happens again, which component of the traceback provided that information to you?
11. shoyer commented on May 1, 2017

    

    shoyer

    on May 1, 2017

    Member

    More actions

    @karenamckinnon In this case, it was in the file paths, i.e., /glade/apps/opt/pandas/0.14.0/gnu/4.8.2/lib/python2.7/site-packages/pandas-0.14.0-py2.7-linux-x86_64.egg/pandas/core/index.pyc
12. stale commented on Apr 1, 2019

    

    staleBot

    on Apr 1, 2019

    More actions

    In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity

    If this issue remains relevant, please comment here or remove the stale label; otherwise it will be marked as closed automatically
13. stale

    added

    stale

    on Apr 1, 2019
14. stale

    closed this as completedon May 1, 2019