assigning values with incompatible dtype
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
The behavior of xarray when assigning values with incompatible dtypes is a bit arbitrary. This is partly due to the behavior of numpy.... numpy 1.20 got a bit cleverer but still seems inconsistent at times... I am not sure what to do about this (and if we should actually be clever here).
- Direct assignment (dupe of #4612)
import xarray as xr
import numpy as np
arr = np.array([2])
arr[0] = np.nan
# ValueError (since numpy 1.20)
arr[0:1] = np.array([np.nan])
# -> array([-9223372036854775808])
da = xr.DataArray([5], dims="x")
da[0] = np.nan
# <xarray.DataArray (x: 1)>
# array([-9223372036854775808])
# Dimensions without coordinates: x
(because this gets converted to da.variable._data[0:1, 0:1] = np.array([np.nan]) (approximately).
da[0] = 1.2345
# casts constant_values to int
- Via a numpy function (
pad,shift,rolling)
pad
da.pad(x=1, constant_values=np.nan)
# ValueError: cannot convert float NaN to integer
da.pad(x=1, constant_values=None)
# casts da to float
da.pad(x=1, constant_values=1.5)
# casts constant_values to int
shift
da.shift(x=1, fill_value=np.nan)
# ValueError: cannot convert float NaN to integer
# da.shift(x=1, fill_value=None)
# None not allowed by shift
da.shift(x=1, fill_value=1.5)
# casts fill_value to int
rolling
da.rolling(x=1).construct("new_axis", stride=3, fill_value=np.nan)
# ValueError: cannot convert float NaN to integer
# da.rolling(x=1).construct("new_axis", stride=3, fill_value=None)
# None not allowed by rolling
da.rolling(x=3).construct("new_axis", stride=3, fill_value=1.5)
# casts fill_value to int
To check:
- What does dask do in these cases?
- What does pandas do?
- What about
strdtypes?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the listed DataArray assignment, pad, shift, and rolling cases, then compare their behavior with NumPy, pandas, and dask as requested. A complete contribution needs an agreed policy for incompatible and string dtypes, followed by tests covering the chosen behavior; the issue does not name files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100