Repository navigation
Stricter defaults for concat, combine, open_mfdataset, merge #8778
Description
Is your feature request related to a problem?
The defaults for concat are excessively permissive: data_vars="all", coords="different", compat="no_conflicts", join="outer". This comment illustrates why this can be hard to predict or understand: a seemingly unrelated option decode_cf controls whether a variable is in data_vars or coords, and can result in wildly different concatenation behaviour.
- This always concatenates data_vars along
concat_dimeven if they did not have that dimension to begin with. - If the same coordinate var exists in different datasets/files, they will be sequentially compared for equality to decide whether they get concatenated.
- The outer join (applied along all dimensions that are not
concat_dim) can result in very large datasets due to small floating points differences in the indexes, and also questionable behaviour with staggered grid datasets. - "no_conflicts" basically picks the first not-NaN value after aligning all datasets, but is quite slow (we should be using
duck_array_ops.nanfirsthere I think).
While "convenient" this really just makes the default experience quite bad with hard-to-understand slowdowns.
Describe the solution you'd like
I propose we migrate to data_vars="minimal", coords="minimal", join="exact", compat="override". This should
- only concatenate
data_varsandcoordsvariables when they already haveconcat_dim. - For any variables that do not have
concat_dim, it will blindly pick them from the first file. join="exact"will prevent ballooning of dimension sizes due to floating point inequalities.- These options will totally avoid any data reads unless explicitly requested by the user.
Unfortunately, this has a pretty big blast radius so we'd need a long deprecation cycle.
Describe alternatives you've considered
No response
Additional context
xref #4824
xref #1385
xref #8231
xref #5381
xref #2064
xref #2217
Reacted by Tom Nicholas, Ashley Barnes, Philippe Blain, Maximilian Roos, Stephan Hoyer, Benoit Bovy and Scott Henderson
Activity
TomNicholas commentedon Feb 22, 2024
Thanks for raising this @dcherian. I support this, along with some effort to improve the docstrings/documentation/faq to make the default behaviour and potential pitfalls clearer.
What would a deprecation cycle for this look like? We're talking about changing the default values of 4 different keyword arguments, that get used in
open_mfdataset,concat,combine_nestedandcombine_by_coords. Should we rip the bandaid off and change them all at once?dcherian commentedon Feb 23, 2024
+1. it would be massively confusing otherwise.
jsignell commentedon May 10, 2024
Regarding deprecation cycle. I think something like:
max-sixty commentedon May 10, 2024
Broadly agree with @jsignell, one dissent:
...iff the behavior from xarray would be different.
Otherwise this will be really noisy. Requires some work to assess whether the behavior will change in the
concatcall, but I think worthwhile / the cost of making the change...[-]Stricter defaults for concat, combine, open_mfdataset[/-][+]Stricter defaults for concat, combine, open_mfdataset, merge[/+]jsignell commentedon Feb 14, 2025
I just opened a draft PR (#10051) that tries to throw warnings iff the behavior from xarray would be different.
So far I am working with the existing tests and I have:
I'm still have more work to do on testing as well.
dcherian commentedon Feb 14, 2025
thank you thank you thank you so much for taking this one. It's the number one pain point for new Xarray users today IMO.
jsignell commentedon Feb 14, 2025
I'm trying 😅
@TomNicholas reminded me on the PR that this is not just about
open_mfdatasetsbut also about other combine functionality. I took a look and it seems like there is a some discrepancy in the defaults right now:open_mfdatasets,combine_nested,combine_by_coordsconcatmerge,xr.Dataset.mergedata_varscoordscompatjoinI think concat should get the updated to the same new values, but should merge?
dcherian commentedon Feb 14, 2025
Yes merge should be updated.
no_conflictsis a really bad and wasteful default (e.g. #5381). Its implementation could also be substantially improved IIRC.jsignell commentedon Mar 5, 2025
Ok! Just to close the loop. I finally finished updating tests on #10062 so that one is fully ready for review (it's just missing doc changes).
#10062 uses sentinels as the kwarg default values and the sentinels get pushed all the way down into the code where we can detect if the outcome would change with the new default value. This allows us to warn without doing additional computation and without too many false alarms. The sentinel also makes it easy to provide the user with both the old default value and the new default value in the warning message - even though the old default value differs based on the function it came from (see #8778 (comment))
#10062 also introduces a new top-level option that controls whether to use old or new default values for these 4 kwargs. The idea for that option came from this comment (#8778 (comment)). More on the new option and steps after #10062 lands: #10062 (comment)
jsignell commentedon Aug 1, 2025
@dcherian just merged #10062 which starts the deprecation cycle that will eventually close this issue and make the new defaults:
data_vars=None, coords="minimal", join="exact", compat="override"As of right now on main you can use
xr.set_options(use_new_combine_kwarg_defaults=True)to opt into the new behavior early. If you do not opt in, then you will get FutureWarnings when you do combine operations (merge, concat, open_mfdataset) that will get different results in the future.#10062 will be in the v2025.08.0 release. What release would be appropriate for flipping the switch on the option so that by default:
use_new_combine_kwarg_defaults=True?