Stricter defaults for concat, combine, open_mfdataset, merge
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Is your feature request related to a problem?
The defaults for concat are excessively permissive: data_vars="all", coords="different", compat="no_conflicts", join="outer". This comment illustrates why this can be hard to predict or understand: a seemingly unrelated option decode_cf controls whether a variable is in data_vars or coords, and can result in wildly different concatenation behaviour.
- This always concatenates data_vars along
concat_dimeven if they did not have that dimension to begin with. - If the same coordinate var exists in different datasets/files, they will be sequentially compared for equality to decide whether they get concatenated.
- The outer join (applied along all dimensions that are not
concat_dim) can result in very large datasets due to small floating points differences in the indexes, and also questionable behaviour with staggered grid datasets. - "no_conflicts" basically picks the first not-NaN value after aligning all datasets, but is quite slow (we should be using
duck_array_ops.nanfirsthere I think).
While "convenient" this really just makes the default experience quite bad with hard-to-understand slowdowns.
Describe the solution you'd like
I propose we migrate to data_vars="minimal", coords="minimal", join="exact", compat="override". This should
- only concatenate
data_varsandcoordsvariables when they already haveconcat_dim. - For any variables that do not have
concat_dim, it will blindly pick them from the first file. join="exact"will prevent ballooning of dimension sizes due to floating point inequalities.- These options will totally avoid any data reads unless explicitly requested by the user.
Unfortunately, this has a pretty big blast radius so we'd need a long deprecation cycle.
Describe alternatives you've considered
No response
Additional context
xref https://github.com/pydata/xarray/issues/4824
xref https://github.com/pydata/xarray/issues/1385
xref https://github.com/pydata/xarray/issues/8231
xref https://github.com/pydata/xarray/issues/5381
xref https://github.com/pydata/xarray/issues/2064
xref https://github.com/pydata/xarray/issues/2217
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the current defaults and behavior of concat, combine, open_mfdataset, and merge, then read the linked issues #4824, #1385, #8231, #5381, #2064, and #2217 for prior context. The proposed work requires a deprecation cycle; it is done when the stricter defaults and their migration path are agreed and applied consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100