pydata / pydata/xarray

Merge with join left or right does not what you'd expect as a Pandas user

Open
#4,766 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-documentation
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Hello everyone,

First of all, thank you for this great library and the work that all contributers put into it!

I try to do, what some call, a table join in xarray. But xr.merge with parameter join='left' or join='right' does not work as expected, at least not as Pandas' users would.

import numpy as np
import xarray as xr
y = xr.DataArray(np.arange(5, 10), [('x', np.arange(0,5))], name='y')  # create y(x)
z = xr.DataArray(np.arange(15,20), [('y', np.arange(5, 10))], name='z')  # create z(y)
expected_result = xr.DataArray(np.arange(15,20), [('x', np.arange(0,5))], name='z')  # This is what I want: z(x)

Try to do left join, to get z(x):

result = xr.merge([y, z], join='left')  # this fails with MergeError

MergeError: unable to determine if these variables should be coordinates or not in the merged result: {'y'}

Try workaroud according to error message with xr.Dataset and and set_coords:

y_ds = xr.Dataset({'y': y}).set_coords('y')
result = xr.merge([y_ds, z], join='left')  # does not error, but gives z(y)

The workaround does not error but gives z(y) with unreferenced additional x coordinate rather than z(x), thus not a true left join.

I suppose that this is a use case with quite heavy use, so a fix would be very appreciated.

Is there a true left join workaround (at best without falling back to Pandas) that I have overseen in the documentation?

Environment:

Output of xr.show_versions() INSTALLED VERSIONS ------------------ commit: None python: 3.8.6 | packaged by conda-forge | (default, Dec 26 2020, 05:05:16) [GCC 9.3.0] python-bits: 64 OS: Linux OS-release: 4.15.0-128-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: de_DE.UTF-8 LOCALE: de_DE.UTF-8 libhdf5: 1.10.6 libnetcdf: 4.7.4

xarray: 0.16.2
pandas: 1.0.5
numpy: 1.19.4
scipy: 1.6.0
netCDF4: 1.5.5.1
pydap: installed
h5netcdf: None
h5py: None
Nio: None
zarr: 2.6.1
cftime: 1.3.0
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.20.0
distributed: 2.20.0
matplotlib: 3.3.3
cartopy: 0.18.0
seaborn: None
numbagg: None
pint: None
setuptools: 49.6.0.post20201009
pip: 20.3.3
conda: None
pytest: None
IPython: 7.19.0
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported xr.merge examples with join='left' and join='right', comparing the result with the stated Pandas-style expectation. Clarify the intended behavior when a data variable is also a dimension name, then establish tests showing the expected aligned result and ensuring the current MergeError and incorrect z(y) workaround are addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.