Merge with join left or right does not what you'd expect as a Pandas user
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Hello everyone,
First of all, thank you for this great library and the work that all contributers put into it!
I try to do, what some call, a table join in xarray. But xr.merge with parameter join='left' or join='right' does not work as expected, at least not as Pandas' users would.
import numpy as np
import xarray as xr
y = xr.DataArray(np.arange(5, 10), [('x', np.arange(0,5))], name='y') # create y(x)
z = xr.DataArray(np.arange(15,20), [('y', np.arange(5, 10))], name='z') # create z(y)
expected_result = xr.DataArray(np.arange(15,20), [('x', np.arange(0,5))], name='z') # This is what I want: z(x)
Try to do left join, to get z(x):
result = xr.merge([y, z], join='left') # this fails with MergeError
MergeError: unable to determine if these variables should be coordinates or not in the merged result: {'y'}
Try workaroud according to error message with xr.Dataset and and set_coords:
y_ds = xr.Dataset({'y': y}).set_coords('y')
result = xr.merge([y_ds, z], join='left') # does not error, but gives z(y)
The workaround does not error but gives z(y) with unreferenced additional x coordinate rather than z(x), thus not a true left join.
I suppose that this is a use case with quite heavy use, so a fix would be very appreciated.
Is there a true left join workaround (at best without falling back to Pandas) that I have overseen in the documentation?
Environment:
Output of xr.show_versions()
INSTALLED VERSIONS ------------------ commit: None python: 3.8.6 | packaged by conda-forge | (default, Dec 26 2020, 05:05:16) [GCC 9.3.0] python-bits: 64 OS: Linux OS-release: 4.15.0-128-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: de_DE.UTF-8 LOCALE: de_DE.UTF-8 libhdf5: 1.10.6 libnetcdf: 4.7.4xarray: 0.16.2
pandas: 1.0.5
numpy: 1.19.4
scipy: 1.6.0
netCDF4: 1.5.5.1
pydap: installed
h5netcdf: None
h5py: None
Nio: None
zarr: 2.6.1
cftime: 1.3.0
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.20.0
distributed: 2.20.0
matplotlib: 3.3.3
cartopy: 0.18.0
seaborn: None
numbagg: None
pint: None
setuptools: 49.6.0.post20201009
pip: 20.3.3
conda: None
pytest: None
IPython: 7.19.0
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported xr.merge examples with join='left' and join='right', comparing the result with the stated Pandas-style expectation. Clarify the intended behavior when a data variable is also a dimension name, then establish tests showing the expected aligned result and ensuring the current MergeError and incorrect z(y) workaround are addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100