Typing of internal datatypes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Is your feature request related to a problem?
Currently there is no static typing of the underlying data structures used in DataArrays.
Simply running
reveal_type(da.data) returns Any.
Adding static typing support to that is unfortunately non-trivial since xarray supports a wide variety of duck-types.
This also comes with internal typing difficulties.
Describe the solution you'd like
I think the way to go is making the DataArray class generic in it's underlying data type.
Something like DataArray[np.ndarray] or DataArray[dask.array].
The implementation would require a TypeVar that is bound to some minimal required Protocol for internal consistency (I think at least it needs dtype and shape attributes).
Datasets would have to be typed the same way, this means only one datatype for all variables is possible, when you mix it it will fall back to the common ancestor which will be the before mentioned protocol. This is basically the same restriction that a dict has.
Now to the main issue that I see with this approach:
I don't know how to type coordinates. They have the same problems than mentioned above for Datasets.
I think it is very common to have dask arrays in the variables but simple numpy arrays in the coordinates, so either one excludes them from the typing or in such cases the common generic typing falls back to the protocol again.
Not sure what is the best approach here.
Describe alternatives you've considered
Since the most common workflow for beginners and intermediate-advanced users is to stick with the DataArrays themself and never touch the underlying data, I am not sure if this change is as beneficial as I want it to be. Maybe it just complicates things and leaving it as Any is easier to solve for advanced users that then have to cast or ignore this.
Additional context
It came up in this discussion:
https://github.com/pydata/xarray/pull/7020#discussion_r972617770_
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the linked discussion on pull request #7020 and inspect how DataArray and Dataset currently expose their underlying data. Use reveal_type(da.data) as the baseline, then establish a consistent typing approach for data, variables, and coordinates. Done means the design is resolved and static type checking no longer reports the underlying data as Any.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data, developer-experience
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100