xarray change subset of values of Dataset variable obtained from pandas DataFrame with pandas v3
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What is your issue?
The following is from the xarray documentation on "Assigning values with indexing":
ds = xr.tutorial.open_dataset("air_temperature") # requires pooch
# add an empty 2D dataarray
ds["empty"] = xr.full_like(ds.air.mean("time"), fill_value=0)
# modify one grid point using loc()
ds["empty"].loc[dict(lon=260, lat=30)] = 100
Whereas the above produces the expected output for me with both pandas v3 and pandas v2, when I try to follow the same approach with a toy example (see below) with pandas v2, it works as expected but with pandas v3, I get a ValueError: assignment destination is read-only similar to (but not equivalent to) that described here:
import xarray as xr
import pandas as pd
import numpy as np
# obtain Dataset from DataFrame
xrds = pd.DataFrame({
'location': ['a', 'b', 'c', 'd'],
'lat': np.arange(-11, -12.5, step=-0.4),
'lon': np.array([15.43, np.nan, np.nan, 14.67]),
'rain': [1432.2, 1321.1, 345.5, 444.]
}
).set_index('location').to_xarray()
# replacement values stored in DataFrame
locdf = pd.DataFrame({
'location': ['b', 'c'],
'lon': [12, 14.5]
}
).set_index('location')
xrds['lon'].loc({'location': xrds.lon.isnull()}) = 14.3 # assignment destination read-only
# trying alternative approach discussed in documentation
xrds['lon'] = xrds.where(xrds.lon.notnull(), other=14.3) # works for case with fixed replacement
xrds['lon'].loc({'location': xrds.lon.isnull()}) = locdf.lon # assignment destination read-only
xrds['lon'] = xrds.where(xrds.lon.notnull(), other=locdf.lon) # fails because locdf.location is not the same length as xrds.location
Note that 'lon' in the above is not a coordinate variable.
I've tried an apparently equivalent approach not involving a DataFrame to check that the apparent inconsistency is coming form the way the Dataset is constructed:
xrds2 = xr.Dataset(
data_vars={
'lat': (["location"], np.arange(-11, -12.5, step=-0.4)),
'lon':(["location"], np.array([15.43, np.nan, np.nan, 14.67])),
'rain': (["location"], [1432.2, 1321.1, 345.5, 444.])
},
coords={"location": ['a', 'b', 'c', 'd']}
)
xrds2['lon'].loc[{'location': xrds2.lon.isnull()}] = locdf.lon # works!
This inconsistency appears undesirable, especially since it was not present with pandas version 2. Perhaps a note in the documentation would be useful?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the minimal DataFrame.to_xarray() reproducer with pandas v2 and v3, then compare it with the directly constructed xr.Dataset example. Trace the assignment path for xrds['lon'].loc and determine whether the pandas v3 behavior should be fixed or documented; done means the expected assignment behavior is consistent or the limitation is clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100