Opening dataset without loading any indexes? #6633

Activity

1. TomNicholas

   added

   topic-backends

   topic-indexing

   topic-lazy array

   enhancement

   on May 24, 2022
2. shoyer commented on May 25, 2022

   

   shoyer

   on May 25, 2022

   Member

   More actions

   Early versions of Xarray used to have lazy loading of data for indexes, but we removed this for the sake of simplicity. In principle we could restore lazy indexes, but another option (post explicit index refactor) might be an option for opening a dataset without creating indexes for 1D coordinates along dimensions.

   Another way to solve this sort of challenges might be to load index data in parallel when using Dask. Right now I believe the data corresponding to indexes is always loaded eagerly, without using Dask.

   All that said -- Do you have a specific example where this has been problematic? In my experience it has been pretty reasonable to use xarray.Dataset objects for schema-like templates, even with index data needing to be loaded eagerly. Possibly another Zarr chunking scheme for your index data could be more efficient?
3. benbovy commented on May 25, 2022

   

   benbovy

   on May 25, 2022

   Member

   More actions

   but another option (post explicit index refactor) might be an option for opening a dataset without creating indexes for 1D coordinates along dimensions.

   It might indeed be worth considering this case too in #6392. Maybe indexes=None (default) to create default indexes for 1D coordinates and indexes={} (empty dictionary) to explicitly skip creating indexes?
4. TomNicholas commented on May 25, 2022

   

   TomNicholas

   on May 25, 2022

   MemberAuthor

   More actions

   Thanks for replying both.

   All that said -- Do you have a specific example where this has been problematic?

   I'll have to defer to the others I tagged for the gory details. Perhaps one of them can cross-link to the specific issue they were having?

   indexes={} (empty dictionary) to explicitly skip creating indexes?

   I would probably do indexes=False just to avoid using a mutable default, but an option like this sounds good to me.
5. shoyer commented on May 25, 2022

   

   shoyer

   on May 25, 2022

   Member

   More actions

   but another option (post explicit index refactor) might be an option for opening a dataset without creating indexes for 1D coordinates along dimensions.

   It might indeed be worth considering this case too in #6392. Maybe indexes=None (default) to create default indexes for 1D coordinates and indexes={} (empty dictionary) to explicitly skip creating indexes?

   +1 this syntax makes sense to me!
6. rabernat commented on May 25, 2022

   

   rabernat

   on May 25, 2022

   Last edited by rabernat

   Contributor

   More actions

   Here is an example that really highlights the performance cost of always loading dimension coordinates:

   import zarr
   store = zarr.storage.FSStore("s3://mur-sst/zarr/", anon=True)
   %time list(zarr.open_consolidated(store)) # -> Wall time: 86.4 ms
   %time ds = xr.open_dataset(store, engine='zarr') # -> Wall time: 17.1 s

   %prun confirms that Xarray is spending most of its time just loading data for the time axis, which you can reproduce at the zarr level as:

   zgroup = zarr.open_consolidated(store)
   %time _ = zgroup['time'][:] # -> Wall time: 14.7 s

   Obviously this example is pretty extreme. There are things that could be done to optimize it, etc. But it really highlights the costs of eagerly loading dimension coordinates. If I don't care about label-based indexing for this dataset, I would rather have my 17s back!

   👍 to "indexes={} (empty dictionary) to explicitly skip creating indexes".
7. shoyer commented on May 25, 2022

   

   shoyer

   on May 25, 2022

   Member

   More actions

   Looking at this mur-sst dataset in particular, it stores time in chunks of size 5. That means fetching the 6443 time values requires 1288 separate HTTP requests -- no wonder it's so slow! If the time axis were instead stored in a single chunk of 51 KB, Xarray would only need 3 small size HTTP requests to load the lat, lon and time indexes, which would probably complete in a fraction of a second.

   That said, I agree that this would be nice to have in general.
8. rabernat commented on May 25, 2022

   

   rabernat

   on May 25, 2022

   Contributor

   More actions

   Yes it is definitely a pathological example. 💣 But the fact remains that there are many cases where we just want to discover dataset contents as quickly as possible and want to avoid the cost of loading coordinates and creating indexes.

   👍React with 👍4Reacted by Stephan Hoyer, Deepak Cherian, Ian Carroll and Tom Nicholas
9. dcherian commented on May 31, 2022

   

   dcherian

   on May 31, 2022

   Contributor

   More actions

   This would also fix #2233
10. dcherian commented on Jul 16, 2023

    

    dcherian

    on Jul 16, 2023

    Last edited by dcherian

    Contributor

    More actions

    Here's one from @lsetiawan that can't be opened because it has a 75GB time dimension coordinate.

    import zarr
    
    zarr.open_group(
        "s3://ooi-data/RS03ECAL-MJ03E-06-BOTPTA302-streamed-botpt_nano_sample",
        mode="r",
        storage_options=dict(anon=True),
    )

    👀React with 👀2Reacted by Joe Hamman and Tom Nicholas
11. itcarroll commented on Feb 23, 2024

    

    itcarroll

    on Feb 23, 2024

    Last edited by itcarroll

    Contributor

    More actions

    Stumbled into this issue while experimenting with page_buf_size on the h5netcdf backend and looking for ways to get XArray closer to the speed of h5py for loading variables when coordinates are too much baggage. As an alternative to #8051, I would like to submit for consideration an "open_variable" method as a fast path from a store to an xarray.Variable (or a Mapping if given either a list of variables in the store or None for all variables).
12. dcherian

    closed this as completedin #8051on Jul 8, 2025