Consistent Handling of Type Casting Hierarchy
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
As brought up in #3643, there appears to be some inconsistencies in how xarray handles other numeric/duck array types with regards to a well-defined type casting hierarchy across operations. For example, in the following:
Construction/Wrapping
- Allows
xarray.core.indexing.ExplicitlyIndexedpandas.Index- Dask array
__array_function__implementers
- Automatically converts
- Anything with a
valuesattribute to its values - Datetime-like array types
- Masked arrays
- Anything else for which
np.asarray(data)is valid
- Anything with a
- Doesn't reject any type when trying to wrap (for an upcast type such as a HoloViews Dataset, this may be needed?)
Binary Ops
- Defers based on xarray's internal hierarchy (Dataset, DataArray, Variable), otherwise relies upon methods of underlying data, and then wraps result.
(would be one less category to worry about if refactored to use __array_ufunc__, see https://github.com/pydata/xarray/pull/3936#issuecomment-610516784)
__array_ufunc__
- Allows a list of supported types
https://github.com/pydata/xarray/blob/9b5140e0711247c373987b56726282140b406d7f/xarray/core/arithmetic.py#L24-L30
along withSupportsArithmetic - Defers to all other types
__array_function__
- To be implemented (https://github.com/pydata/xarray/issues/3917)
One concrete example of where this has been problematic is with xarray DataArrays and Pint Quantities (#3643). xarray DataArray is above Pint Quantity in the (generally agreed upon) type casting hierarchy, and wrapping and binary ops work properly since Pint Quantities defer and xarray DataArrays handle the operation. However, ufuncs fail because they both attempt to defer to the other. Having a consistent way of handling type compatibility across all relevant areas in xarray should be able to remove these kinds of issues.
However, it would be good to keep in mind that an agreed upon way of how to do this in the broader ecosystem doesn't seem to be there yet, so this would still be treading in uncertain waters for the moment. I've been operating under these assumptions when working with Pint, but I definitely think there is a need for more authoritative guidance.
Also, if I'm mistaken in any of the things mentioned above, please do let me know!
cc @keewis, @shoyer
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with xarray/core/arithmetic.py, especially the supported types used by array_ufunc, then trace construction/wrapping and binary-operation dispatch. Compare those paths with the linked NumPy and Dask guidance and the Pint example; the work is done when the project has an agreed, consistent compatibility hierarchy across these operations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100