pydata / pydata/xarray

Inconsistent/confusing behaviour when concatenating dimension coords

Open
#2,975 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-combine
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

I noticed that with multiple conflicting dimension coords then concat can give pretty weird/counterintuitive results, at least compared to what the documentation suggests they should give:

# Create two datasets with conflicting coordinates
objs = [Dataset({'x': [0], 'y': [1]}), Dataset({'y': [0], 'x': [1]})]

[<xarray.Dataset>
 Dimensions:  (x: 1, y: 1)
 Coordinates:
   * x        (x) int64 0
   * y        (y) int64 1
 Data variables:
     *empty*, 
<xarray.Dataset>
 Dimensions:  (x: 1, y: 1)
 Coordinates:
   * y        (y) int64 0
   * x        (x) int64 1
 Data variables:
     *empty*]
# Try to join along only 'x',
# coords='minimal' so concatenate "Only coordinates in which the dimension already appears"
concat(objs, dim='x', coords='minimal') 

<xarray.Dataset>
Dimensions:  (x: 2, y: 2)
Coordinates:
  * y        (y) int64 0 1
  * x        (x) int64 0 1
Data variables:
    *empty*

# It's joined along x and y! 

Based on my reading of the docstring for concat, I would have expected this to not attempt to concatenate y, because coords='minimal', and instead to throw an error because 'y' is a "non-concatenated variable" whose values are not the same across datasets.

Now let's try to get concat to broadcast 'y' across 'x':

# Try to join along only 'x' by setting coords='different'
concat(objs, dim='x', coords='different') 

Now as "Data variables which are not equal (ignoring attributes) across all datasets are also concatenated" then I would have expected 'y' to be concatenated across 'x', i.e. to add the 'x' dimension to the 'y' coord, i.e:

<xarray.Dataset>
Dimensions:  (x: 2, y: 1)
Coordinates:
  * y        (y, x) int64 1 0
  * x        (x)    int64 0 1
Data variables:
    *empty*

But that's not what we get!:

<xarray.Dataset>
Dimensions:  (x: 2, y: 2)
Coordinates:
  * y        (y) int64 0 1
  * x        (x) int64 0 1
Data variables:
    *empty*
Same again but without dimension coords

If we create the same sort of objects but the variables are data vars not coords, then everything behaves exactly as expected:

objs2 = [Dataset({'a': ('x', [0]), 'b': ('y', [1])}), Dataset({'a': ('x', [1]), 'b': ('y', [0])})]

[<xarray.Dataset>
 Dimensions:  (x: 1, y: 1)
 Dimensions without coordinates: x, y
 Data variables:
     a        (x) int64 0
     b        (y) int64 1, 
<xarray.Dataset>
 Dimensions:  (x: 1, y: 1)
 Dimensions without coordinates: x, y
 Data variables:
     a        (x) int64 1
     b        (y) int64 0]

concat(objs2, dim='x', data_vars='minimal')

ValueError: variable b not equal across datasets

concat(objs2, dim='x', data_vars='different')

<xarray.Dataset>
Dimensions:  (x: 2, y: 1)
Dimensions without coordinates: x, y
Data variables:
    a        (x) int64 0 1
    b        (x, y) int64 1 0

Also if you do the same again but with coordinates which are not dimension coords, i.e:

objs3 = [Dataset(coords={'a': ('x', [0]), 'b': ('y', [1])}), Dataset(coords={'a': ('x', [1]), 'b': ('y', [0])})]

[<xarray.Dataset>
 Dimensions:  (x: 1, y: 1)
 Coordinates:
     a        (x) int64 0
     b        (y) int64 1
 Dimensions without coordinates: x, y
 Data variables:
     *empty*, 
<xarray.Dataset>
 Dimensions:  (x: 1, y: 1)
 Coordinates:
     a        (x) int64 1
     b        (y) int64 0
 Dimensions without coordinates: x, y
 Data variables:
     *empty*]

then this again gives the expected concatenation behaviour.

So this implies that the compatibility checks that are being done on the data vars are not being done on the coords, but only if they are dimension coordinates!

Either this is not the desired behaviour or the concat docstring needs to be a lot clearer. If we agree that this is not the desired behaviour then I will have a look inside concat to work out why it's happening.

EDIT: Presumably this has something to do with the ToDo in the code for concat: # TODO: support concatenating scalar coordinates even if the concatenated dimension already exists...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the concat examples in the issue, then inspect the concat implementation and its docstring, including the noted TODO about scalar coordinates. Compare dimension-coordinate handling with data variables and non-dimension coordinates. Done means the behavior matches the documented coords options, or the documentation clearly explains the existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.