Making xarray math lazy
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
At SciPy, I had the realization that it would be relatively straightforward to make element-wise math between xarray objects lazy. This would let us support lazy coordinate arrays, a feature that has quite a few use-cases, e.g., for both geoscience and astronomy.
The trick would be to write a lazy array class that holds an element-wise vectorized function and passes indexers on to its arguments. I haven't thought too hard about this yet for vectorized indexing, but it could be quite efficient for outer indexing. I have some prototype code but no tests yet.
The question is how to hook this into xarray operations. In particular, supposing that the inputs to a function do no hold dask arrays:
- Should we try to make every element-wise operation with vectorized functions (ufuncs) lazy by default? This might have negative performance implications and would be a little tricky to implement with xarray's current code, since we still implement binary operations like
+with separate logic fromapply_ufunc. - Should we make every element-wise operation that explicitly uses
apply_ufunc()lazy by default? - Or should we only make element-wise operations lazy with
apply_ufunc()if you use some special flag, e.g.,apply_ufunc(..., lazy=True)?
I am leaning towards the last option for now but would welcome other opinions.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review xarray's apply_ufunc and separate binary-operation paths, along with the proposed lazy array class and its indexing behavior. Done requires resolving the issue's API and default-behavior questions, then implementing and testing the selected scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100