open_mfdataset -> to_netcdf() randomly leading to dead workers #4710

Activity

1. fmaussion

   added

   topic-dask

   on Dec 18, 2020
2. dcherian commented on Dec 18, 2020

   

   dcherian

   on Dec 18, 2020

   Contributor

   More actions

   I've run in to this and usually just call .load or write to zarr instead :)

   How are you setting up your dask cluster? Is distributed involved?
3. TimoRoth commented on Dec 18, 2020

   

   TimoRoth

   on Dec 18, 2020 via email

   Contributor

   More actions

   We are not setting up a dask cluster at all. OGGM is using python multiprocessing to distribute its work. And the distributed workers then are calling xarray functions.
4. markelg commented on Dec 22, 2020

   

   markelg

   on Dec 22, 2020

   Contributor

   More actions

   Perhaps this is related to #3961? Did you try to call open_mfdataset with lock=False?
5. fmaussion commented on Dec 22, 2020

   

   fmaussion

   on Dec 22, 2020

   MemberAuthor

   More actions

   Thanks for the tip @markelg.

   yes, it seems indeed very much related. Closing this in favor of #3961
6. fmaussion

   closed this as completedon Dec 22, 2020
7. millet5818 commented on May 27, 2024

   

   millet5818

   on May 27, 2024

   More actions

   This is:

   • xarray: 0.16.2
   • dask: 2.30.0

   I'm not sure a github issue is the right place to report this, but I'm not sure where else, so here it is.

   I just had two very long weeks of debugging stalled (i.e. "dead") OGGM jobs in a cluster environment. I finally nailed it down to ds.to_netcdf(path) in this situation:

   with xr.open_mfdataset(tmp_paths, combine='nested', concat_dim='rgi_id') as ds:
       ds.to_netcdf(path)

   tmp_paths are a few netcdf files (from 2 to about 60). The combined dataset is nothing close to big (a few hundred MB at most).

   Most of the time, this command works just fine. But in 30% of the cases, this would just... stop and stall. One or more of the workers would simply stop working without coming back or erroring.

   What I can give as additional information:

   • changing ds.to_netcdf(path) to ds.load().to_netcdf(path) solves the problem
   • the problem became worse (i.e. more often) when the files to concatenate increased in the number of variables (the final size of the concatenated file doesn't seem to matter at all, it occurs also with files < 1 MB)
   • I can't reproduce the problem locally. The files are here if someone's interested, but I don't think the files are the issue here.
   • the files use gzip compression
   • On cluster, we are dealing with 64 core nodes, which do a lot of work before arriving to these two lines. We use python multiprocessing ourselves before that, create our own pool and use it, etc. But at the moment the job hits these two lines, no other job is running.

   Is this is some kind of weird interaction between our own multiprocessing and dask? Is it more an IO problem that occurs only on cluster? I don't know.

   I know this is a crappy bug report, but the fact that I lost a lot of time on this recently has gone on my nerves 😉 (I'm mostly angry at myself for taking so long to find out that these two lines were the problem).

   In order to make a question out of this crappy report: how can I possibly debug this? I solved my problem now (with ds.load()), but this is not really satisfying. Any tip is appreciated!

   cc @TimoRoth our cluster IT whom I annoyed a lot before finding out that the problem was in xarray/dask

   Hi, @fmaussion
   I encountered the same problem as you, and I solved it in the same way(ds.load().to_netcdf()). However, when the file is very large, the "load" function does not solve the problem very well. Do you have a better solution?