intake / intake/intake-esm

Concatenate datasets when some variables are present in one dataset and not present in other dataset

Open
#144 3 comments 2 reactions 0 assignees View on GitHub
needs triage
Dominant language
Python
Stars
164
Forks
54
PR merge metrics
No merged PRs in 30d

Description

Currently, `xarray.concat()` doesn't know how to deal with datasets for which one dataset contains some variables and these same variables are not present in the rest of the datasets to concatenate.

To address this issue, there are two options:

- Add NaNs by determining the set of variables that are not in all datasets but at least in one dataset
and use .full_like(..., fill_value=np.nan) to add the variables not present in the other dataset (
- Dropping the problematic variables
Either option would probably need to be done prior calling `xr.concat()` in `join_new()` and `join_existing()` introduced in #135

Contributor guide

Open the contributing guide

Research direction

Start in join_new() and join_existing(), introduced in #135, and trace how they call xarray.concat(). Reproduce the case where variables exist in only some datasets, then verify the chosen behavior for those variables—adding NaNs or dropping them—is consistent across both entry points.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.