pydata / pydata/xarray

Stricter defaults for concat, combine, open_mfdataset, merge

Open
#8,778 10 comments 7 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement topic-combine
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Is your feature request related to a problem?

The defaults for concat are excessively permissive: data_vars="all", coords="different", compat="no_conflicts", join="outer". This comment illustrates why this can be hard to predict or understand: a seemingly unrelated option decode_cf controls whether a variable is in data_vars or coords, and can result in wildly different concatenation behaviour.

  1. This always concatenates data_vars along concat_dim even if they did not have that dimension to begin with.
  2. If the same coordinate var exists in different datasets/files, they will be sequentially compared for equality to decide whether they get concatenated.
  3. The outer join (applied along all dimensions that are not concat_dim) can result in very large datasets due to small floating points differences in the indexes, and also questionable behaviour with staggered grid datasets.
  4. "no_conflicts" basically picks the first not-NaN value after aligning all datasets, but is quite slow (we should be using duck_array_ops.nanfirst here I think).

While "convenient" this really just makes the default experience quite bad with hard-to-understand slowdowns.

Describe the solution you'd like

I propose we migrate to data_vars="minimal", coords="minimal", join="exact", compat="override". This should

  1. only concatenate data_vars and coords variables when they already have concat_dim.
  2. For any variables that do not have concat_dim, it will blindly pick them from the first file.
  3. join="exact" will prevent ballooning of dimension sizes due to floating point inequalities.
  4. These options will totally avoid any data reads unless explicitly requested by the user.

Unfortunately, this has a pretty big blast radius so we'd need a long deprecation cycle.

Describe alternatives you've considered

No response

Additional context

xref https://github.com/pydata/xarray/issues/4824
xref https://github.com/pydata/xarray/issues/1385
xref https://github.com/pydata/xarray/issues/8231
xref https://github.com/pydata/xarray/issues/5381
xref https://github.com/pydata/xarray/issues/2064
xref https://github.com/pydata/xarray/issues/2217

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the current defaults and behavior of concat, combine, open_mfdataset, and merge, then read the linked issues #4824, #1385, #8231, #5381, #2064, and #2217 for prior context. The proposed work requires a deprecation cycle; it is done when the stricter defaults and their migration path are agreed and applied consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.