pydata / pydata/xarray

xarray change subset of values of Dataset variable obtained from pandas DataFrame with pandas v3

Open
#11,527 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs triage
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What is your issue?

The following is from the xarray documentation on "Assigning values with indexing":

ds = xr.tutorial.open_dataset("air_temperature") # requires pooch

# add an empty 2D dataarray
ds["empty"] = xr.full_like(ds.air.mean("time"), fill_value=0)

# modify one grid point using loc()
ds["empty"].loc[dict(lon=260, lat=30)] = 100

Whereas the above produces the expected output for me with both pandas v3 and pandas v2, when I try to follow the same approach with a toy example (see below) with pandas v2, it works as expected but with pandas v3, I get a ValueError: assignment destination is read-only similar to (but not equivalent to) that described here:

import xarray as xr
import pandas as pd
import numpy as np

# obtain Dataset from DataFrame
xrds = pd.DataFrame({
     'location': ['a', 'b', 'c', 'd'],
     'lat': np.arange(-11, -12.5, step=-0.4),
     'lon': np.array([15.43, np.nan, np.nan, 14.67]),
     'rain': [1432.2, 1321.1, 345.5, 444.]
}
).set_index('location').to_xarray()

# replacement values stored in DataFrame
locdf = pd.DataFrame({
    'location': ['b', 'c'],
    'lon': [12, 14.5]
}
).set_index('location')

xrds['lon'].loc({'location': xrds.lon.isnull()}) = 14.3 # assignment destination read-only

# trying alternative approach discussed in documentation
xrds['lon'] = xrds.where(xrds.lon.notnull(), other=14.3) # works for case with fixed replacement

xrds['lon'].loc({'location': xrds.lon.isnull()}) = locdf.lon # assignment destination read-only

xrds['lon'] = xrds.where(xrds.lon.notnull(), other=locdf.lon) # fails because locdf.location is not the same length as xrds.location

Note that 'lon' in the above is not a coordinate variable.

I've tried an apparently equivalent approach not involving a DataFrame to check that the apparent inconsistency is coming form the way the Dataset is constructed:

xrds2 = xr.Dataset(
data_vars={
    'lat': (["location"], np.arange(-11, -12.5, step=-0.4)),
    'lon':(["location"], np.array([15.43, np.nan, np.nan, 14.67])),
    'rain': (["location"], [1432.2, 1321.1, 345.5, 444.])
},
coords={"location": ['a', 'b', 'c', 'd']}
)

xrds2['lon'].loc[{'location': xrds2.lon.isnull()}] = locdf.lon # works!

This inconsistency appears undesirable, especially since it was not present with pandas version 2. Perhaps a note in the documentation would be useful?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the minimal DataFrame.to_xarray() reproducer with pandas v2 and v3, then compare it with the directly constructed xr.Dataset example. Trace the assignment path for xrds['lon'].loc and determine whether the pandas v3 behavior should be fixed or documented; done means the expected assignment behavior is consistent or the limitation is clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.