CachingFileManager opens one file handle per open_dataset() call — could identical managers share a FILE_CACHE entry? #11581

Activity

1. djhoese

   added

   needs triageIssue that has not been reviewed by xarray team memberIssue that has not been reviewed by xarray team member

   on Sep 11, 2026
2. djhoese commented on Sep 11, 2026

   

   djhoese

   on Sep 11, 2026

   ContributorAuthor

   More actions

   Oh I should have noted that I've already started fixing our issue in Satpy by calling open_dataset once per group and holding on to the xarray Dataset for reuse.
3. headtr1ck commented on Oct 12, 2026

   

   headtr1ck

   on Oct 12, 2026

   Collaborator

   More actions

   Thanks for the detailed write-up, and sorry for the slow reply. The separate cache entry per open_dataset() call is intentional: #4879 added the manager id to the key so that opening a file again gives you its current state, e.g. after it was modified on disk (#4862), instead of a cached handle that may still show the old content.

   For your case of one file with many groups, xr.open_groups(path) or xr.open_datatree(path) should already do what you want: they open the file only once and share that handle for all groups. With a test file with 10 groups, open_dataset(..., group=...) per group opens the file 10 times, while open_groups/open_datatree open it once.

   The segfault you hit is most likely HDFGroup/hdf5#6721 (case 2): with several handles to the same file, closing one of them frees state that the others still use, and the next open of that file then crashes in H5Dget_create_plist. Opening each file only once avoids it.

   That said, I think your option 1 could make sense for HDF5-based files (engine="netcdf4" and "h5netcdf"), because there the separate handles don't actually give us the freshness that #4879 is about while another handle to the file is still open:

   • With HDF5 file locking (the default), the file can't be modified at all while it is open: writing to it from another process fails with an HDF error.
   • With HDF5_USE_FILE_LOCKING=FALSE, another process can modify it, but a second open_dataset while the first dataset is still open sees the old attributes together with the new data, because HDF5 shares its internal file object between handles to the same file. Only after all handles are closed does a new open_dataset see the new state consistently.

   So for these backends, managers with identical arguments in read mode could share one handle as long as any of them is alive (reference-counted, and closed with the last one), while a new open_dataset after all of them are closed still opens the file fresh. That would avoid the repeated opens and LRU thrashing you saw, and also the multiple-handle situation that triggers the HDF5 crash. For non-HDF5 formats like netCDF3, reopening does give fresh data, so they should keep the current behavior. Would that cover your use case?
4. headtr1ck

   added

   enhancement

   topic-backends

   needs discussion

   and removed

   needs triageIssue that has not been reviewed by xarray team memberIssue that has not been reviewed by xarray team member

   on Oct 12, 2026
5. djhoese commented on Oct 12, 2026

   

   djhoese

   on Oct 12, 2026

   ContributorAuthor

   More actions

   Thanks for the response.

   For your case of one file with many groups, xr.open_groups(path) or xr.open_datatree(path) should already do what you want

   Yes, however this requires decoding all of those groups when only a couple may be used.

   For the HDF5 but I think it is actually case 1 from that issue and I had filed it as a separate security issue with them.

   And yes I had only been thinking about read-only use cases here.

   That said, I think your option 1 could make sense for HDF5-based files (engine="netcdf4" and "h5netcdf"), because there the separate handles don't actually give us the freshness that #4879 is about while another handle to the file is still open:

   Ok I see. So technically the 4879 solution is "broken" for HDF5 files when a handle remains open. Good to know.

   For non-HDF5 formats like netCDF3, reopening does give fresh data, so they should keep the current behavior. Would that cover your use case?

   Yes. I'm only concerned about NetCDF4/HDF5 files in my current issue. I was trying to optimize (and avoid seg faults) the reading of these datasets with a lot of files with lots of groups with lots of variables.
6. headtr1ck commented on Oct 27, 2026

   

   headtr1ck

   on Oct 27, 2026

   Collaborator

   More actions

   Regarding decoding all groups with open_datatree: you can also open each file only once and still decode just the groups you need, by passing the open file to the backend store yourself:

   import h5netcdf
   import xarray as xr
   from xarray.backends import H5NetCDFStore
   
   f = h5netcdf.File(path, "r")
   ds_a = xr.open_dataset(H5NetCDFStore(f, group="group_a"))
   ds_b = xr.open_dataset(H5NetCDFStore(f, group="group_b"))
   ...
   f.close()

   or with netCDF4:

   import netCDF4
   from xarray.backends import NetCDF4DataStore
   
   nc = netCDF4.Dataset(path)
   ds_a = xr.open_dataset(NetCDF4DataStore(nc["group_a"]))

   All datasets then share one handle per file. Note that closing any of these datasets closes the shared file, so close the file yourself once you are done with all of them, instead of the individual datasets.
7. djhoese commented on Oct 27, 2026

   

   djhoese

   on Oct 27, 2026

   ContributorAuthor

   More actions

   Yep, thanks. This is exactly what I'm doing now actually. It's one of the things that sparked me filing #11814.