Inconsistent/confusing behaviour when concatenating dimension coords
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
I noticed that with multiple conflicting dimension coords then concat can give pretty weird/counterintuitive results, at least compared to what the documentation suggests they should give:
# Create two datasets with conflicting coordinates
objs = [Dataset({'x': [0], 'y': [1]}), Dataset({'y': [0], 'x': [1]})]
[<xarray.Dataset>
Dimensions: (x: 1, y: 1)
Coordinates:
* x (x) int64 0
* y (y) int64 1
Data variables:
*empty*,
<xarray.Dataset>
Dimensions: (x: 1, y: 1)
Coordinates:
* y (y) int64 0
* x (x) int64 1
Data variables:
*empty*]
# Try to join along only 'x',
# coords='minimal' so concatenate "Only coordinates in which the dimension already appears"
concat(objs, dim='x', coords='minimal')
<xarray.Dataset>
Dimensions: (x: 2, y: 2)
Coordinates:
* y (y) int64 0 1
* x (x) int64 0 1
Data variables:
*empty*
# It's joined along x and y!
Based on my reading of the docstring for concat, I would have expected this to not attempt to concatenate y, because coords='minimal', and instead to throw an error because 'y' is a "non-concatenated variable" whose values are not the same across datasets.
Now let's try to get concat to broadcast 'y' across 'x':
# Try to join along only 'x' by setting coords='different'
concat(objs, dim='x', coords='different')
Now as "Data variables which are not equal (ignoring attributes) across all datasets are also concatenated" then I would have expected 'y' to be concatenated across 'x', i.e. to add the 'x' dimension to the 'y' coord, i.e:
<xarray.Dataset>
Dimensions: (x: 2, y: 1)
Coordinates:
* y (y, x) int64 1 0
* x (x) int64 0 1
Data variables:
*empty*
But that's not what we get!:
<xarray.Dataset>
Dimensions: (x: 2, y: 2)
Coordinates:
* y (y) int64 0 1
* x (x) int64 0 1
Data variables:
*empty*
Same again but without dimension coords
If we create the same sort of objects but the variables are data vars not coords, then everything behaves exactly as expected:
objs2 = [Dataset({'a': ('x', [0]), 'b': ('y', [1])}), Dataset({'a': ('x', [1]), 'b': ('y', [0])})]
[<xarray.Dataset>
Dimensions: (x: 1, y: 1)
Dimensions without coordinates: x, y
Data variables:
a (x) int64 0
b (y) int64 1,
<xarray.Dataset>
Dimensions: (x: 1, y: 1)
Dimensions without coordinates: x, y
Data variables:
a (x) int64 1
b (y) int64 0]
concat(objs2, dim='x', data_vars='minimal')
ValueError: variable b not equal across datasets
concat(objs2, dim='x', data_vars='different')
<xarray.Dataset>
Dimensions: (x: 2, y: 1)
Dimensions without coordinates: x, y
Data variables:
a (x) int64 0 1
b (x, y) int64 1 0
Also if you do the same again but with coordinates which are not dimension coords, i.e:
objs3 = [Dataset(coords={'a': ('x', [0]), 'b': ('y', [1])}), Dataset(coords={'a': ('x', [1]), 'b': ('y', [0])})]
[<xarray.Dataset>
Dimensions: (x: 1, y: 1)
Coordinates:
a (x) int64 0
b (y) int64 1
Dimensions without coordinates: x, y
Data variables:
*empty*,
<xarray.Dataset>
Dimensions: (x: 1, y: 1)
Coordinates:
a (x) int64 1
b (y) int64 0
Dimensions without coordinates: x, y
Data variables:
*empty*]
then this again gives the expected concatenation behaviour.
So this implies that the compatibility checks that are being done on the data vars are not being done on the coords, but only if they are dimension coordinates!
Either this is not the desired behaviour or the concat docstring needs to be a lot clearer. If we agree that this is not the desired behaviour then I will have a look inside concat to work out why it's happening.
EDIT: Presumably this has something to do with the ToDo in the code for concat: # TODO: support concatenating scalar coordinates even if the concatenated dimension already exists...
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the concat examples in the issue, then inspect the concat implementation and its docstring, including the noted TODO about scalar coordinates. Compare dimension-coordinate handling with data variables and non-dimension coordinates. Done means the behavior matches the documented coords options, or the documentation clearly explains the existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100