pydata / pydata/xarray

DataArrayCoarsen.reduce passes kwargs to coordinate function

Open
#8,059 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

API design topic-documentation
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What is your issue?

When using custom reducing function with the .reduce(func, keep_attrs=None, **kwargs) method of xarray.core.rolling.DataArrayCoarsen object, keyword arguments of the reduce method are passed to the given reducing function (func) as documented. However, if a different function is given to reduce the coordinates when the coarsen or rolling object is created, e.g. coord_func="mean", the same kwargs get also passed to the coordinate reducing function. This is quite often undesired, as the function to reduce the coordinates might not have the same kwargs as the function used to reduce the actual data.

Here is an example:

import numpy as np
import xarray as xr

def reduce_func_with_kwargs(x, axis, dummy=None):
    dummy = 0
    return np.sum(x, axis=axis)

np.random.seed(0)
temperature = 15 + 8 * np.random.randn(2, 2)
lon = [[-99.83, -99.32], [-99.79, -99.23]]
lat = [[42.25, 42.21], [42.63, 42.59]]

da = xr.DataArray(
    data=temperature,
    dims=["x", "y"],
    coords=dict(
        lon=(["x", "y"], lon),
        lat=(["x", "y"], lat),
    ),
    attrs=dict(
        description="Ambient temperature.",
        units="degC",
    ),
)

# This works fine
coarsen_mean = da.coarsen({"x": 2, "y": 2}, coord_func="mean").reduce(
    reduce_func_with_kwargs
) 
# This raises TypeError: nanmean() got an unexpected keyword argument 'dummy'
coarsen_reduce = da.coarsen({"x": 2, "y": 2}, coord_func="mean").reduce(
    reduce_func_with_kwargs, dummy=1
)

The first reduction works as expected, as no kwargs are passed to the reduce method. However, the latter reduction raises a TypeError

Traceback (most recent call last):
  File "/home/saskartt/Code/misc/xarray_bug_example.py", line 28, in <module>
    coarsen_reduce = da.coarsen({"x": 2, "y": 2}, coord_func="mean").reduce(reduce_func_with_kwargs, dummy=1) 
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/saskartt/miniconda3/envs/xarray-bug/lib/python3.11/site-packages/xarray/core/rolling.py", line 1064, in reduce
    return wrapped_func(self, keep_attrs=keep_attrs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/saskartt/miniconda3/envs/xarray-bug/lib/python3.11/site-packages/xarray/core/rolling.py", line 1012, in wrapped_func
    coords[c] = v.variable.coarsen(
                ^^^^^^^^^^^^^^^^^^^
  File "/home/saskartt/miniconda3/envs/xarray-bug/lib/python3.11/site-packages/xarray/core/variable.py", line 2522, in coarsen
    return self._replace(data=func(reshaped, axis=axes, **kwargs), attrs=_attrs)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/saskartt/miniconda3/envs/xarray-bug/lib/python3.11/site-packages/xarray/core/duck_array_ops.py", line 615, in mean
    return _mean(array, axis=axis, skipna=skipna, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/saskartt/miniconda3/envs/xarray-bug/lib/python3.11/site-packages/xarray/core/duck_array_ops.py", line 382, in f
    return func(values, axis=axis, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: nanmean() got an unexpected keyword argument 'dummy'

This is because kwarg dummy=1 gets passed to coordinate reduction function "mean" (nanmean) in addition to reduce_func_with_kwargs in xarray/core/rolling.py:1012.

Currently, it is undocumented that the kwargs are also passed to the coordinate reduction function, so this is probably technically not a bug. However, I think in many cases you might want to use different kwargs for the functions and some solution for this case would be nice.

INSTALLED VERSIONS
------------------
commit: None
python: 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:08:17) [GCC 12.2.0]

xarray: 2023.7.0
pandas: 2.0.3
numpy: 1.25.2

Cheers,
Sasu Karttunen

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in xarray/core/rolling.py around line 1012 and reproduce the DataArrayCoarsen.reduce example with coord_func="mean" and dummy=1. Trace how kwargs reach the data and coordinate reduction functions; done when the custom reducer still receives its kwargs without causing the coordinate reducer to raise the reported TypeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.