pydata / pydata/xarray

Reduction operations fail with Weighted DataArrayWeighted

Open
#9,841 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What happened?

I have a DataSet with some weighted DataArrays. This set-up is extremely useful to me as I can filter and perform operations over the whole dataset and all shared dimensions. One of the DataArrays is weighted, and I was hoping this would be automatically handled in groupbys and general reduction operations, but the error thrown is below. If I call mean on the dataset.

I'm happy to raise a PR to fix if I can work out how to do it, but I just want to make sure that it's agreed that this isn't correct behaviour.

What did you expect to happen?

I would like DataArrays that are unweighted to return the usual mean, and for DataArrayWeighted to return a mean reflecting their weights, as if I'd just called da_weighted.mean(). This would allow me to calculate means in groupbys on the DataSet.

Minimal Complete Verifiable Example
import numpy as np
import xarray as xr

da = xr.DataArray(
    data=[[4.0, 5.0, 6.0], [1.0, 2.0, np.nan], [np.nan, np.nan, np.nan]],
    dims=["t", "x"],
    coords={"t": [0, 1, 2], "x": ["a", "b", "c"]}
)

dw = xr.DataArray(
    data=[0.1, 0.2, 0.3],
    dims=["t"],
    coords={"t": [0, 1, 2]}
)
db = da.copy(deep=True).weighted(dw)
ds = xr.Dataset({"a": da, "b": db, "w": dw})

# This works
print(db.mean())

# Errors
print(ds["b"].mean())

# Errors
print(ds.mean(dim="t"))

# Errors
print(ds.groupby_bins("t", bins=[0, 2.5]).mean())
MVCE confirmation
  • Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • Complete example — the example is self-contained, including all data and the text of any traceback.
  • Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • New issue — a search of GitHub Issues suggests this is not a duplicate.
  • Recent environment — the issue occurs with the latest version of xarray and its dependencies.
Relevant log output
python weighted_demo.py
<xarray.DataArray ()> Size: 8B
array(3.)
Traceback (most recent call last):
  File <redact>, line 22, in <module>
    print(ds["b"].mean())
          ^^^^^^^^^^^^^^
  File "<redact>/.venv/lib/python3.12/site-packages/xarray/core/_aggregations.py", line 2982, in mean
    return self.reduce(
           ^^^^^^^^^^^^
  File "<redact>/.venv/lib/python3.12/site-packages/xarray/core/dataarray.py", line 3839, in reduce
    var = self.variable.reduce(func, dim, axis, keep_attrs, keepdims, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redact>/.venv/lib/python3.12/site-packages/xarray/core/variable.py", line 1677, in reduce
    result = super().reduce(
             ^^^^^^^^^^^^^^^
  File "<redact>/.venv/lib/python3.12/site-packages/xarray/namedarray/core.py", line 918, in reduce
    data = func(self.data, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redact>/.venv/lib/python3.12/site-packages/xarray/core/duck_array_ops.py", line 680, in mean
    return _mean(array, axis=axis, skipna=skipna, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redact>/.venv/lib/python3.12/site-packages/xarray/core/duck_array_ops.py", line 447, in f
    return func(values, axis=axis, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redact>/.venv/lib/python3.12/site-packages/xarray/core/nanops.py", line 124, in nanmean
    return _nanmean_ddof_object(0, a, axis=axis, dtype=dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redact>/.venv/lib/python3.12/site-packages/xarray/core/nanops.py", line 117, in _nanmean_ddof_object
    data = np.sum(value, axis=axis, dtype=dtype, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redact>/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py", line 2389, in sum
    return _wrapreduction(
           ^^^^^^^^^^^^^^^
  File "<redact>/.venv/lib/python3.12/site-packages/numpy/_core/fromnumeric.py", line 86, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: float() argument must be a string or a real number, not 'DataArrayWeighted'
Anything else we need to know?

There are several functions that might fall into this category like std, but I think they could all be handled similarly.

Environment

INSTALLED VERSIONS

commit: None
python: 3.12.3 (main, Jul 31 2024, 17:43:48) [GCC 13.2.0]
python-bits: 64
OS: Linux
OS-release: 5.15.153.1-microsoft-standard-WSL2
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF-8
LOCALE: ('C', 'UTF-8')
libhdf5: 1.14.4
libnetcdf: None

xarray: 2024.10.0
pandas: 2.2.3
numpy: 2.0.2
scipy: 1.14.1
netCDF4: None
pydap: None
h5netcdf: 1.4.0
h5py: 3.12.1
zarr: None
cftime: None
nc_time_axis: None
iris: None
bottleneck: 1.4.2
dask: None
distributed: None
matplotlib: 3.9.2
cartopy: None
seaborn: 0.13.2
numbagg: None
fsspec: None
cupy: None
pint: None
sparse: 0.15.4
flox: None
numpy_groupies: None
setuptools: 75.3.0
pip: 24.0
conda: None
pytest: 8.3.3
mypy: 1.13.0
IPython: 8.29.0
sphinx: None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the weighted_demo.py reproduction and trace the reduction path through xarray/core/_aggregations.py, dataarray.py, variable.py, duck_array_ops.py, and nanops.py. Compare Dataset and groupby_bins reductions with DataArrayWeighted.mean; done means unweighted variables retain normal reductions while weighted variables produce weight-aware results, with coverage for mean and related operations.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.