perf improvement for interp: set `assume_sorted` automatically
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What is your issue?
assume_sorted is False, so for vectorized interpolation across multiple dimensions, we end up lexsorting the coordinates all the time. For some reason, this can be quite slow with dask.
Instead we should be able to do
obj = self
# sort by slicing if we can
for coord in set(indexers) and set(self._indexes):
# TODO: better check for PandasIndex
if self.indexes[coord].is_monotonic_decreasing:
obj = obj.isel(coord: slice(None, None, -1))
# TODO: make None the new default
if assume_sorted is None:
# TODO: dims without coordinates are fine too
assume_sorted = all(self.indexes[coord].is_monotonic_increasing for coord in indexers)
I'll add a reproducible example later, but the problem I've been playing gets much faster for graph construction:
xref #6799
cc @mpiannucci @Illviljan
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at xarray/core/dataset.py around line 4081 and inspect interp's indexer handling and PandasIndex monotonicity checks. Reproduce the reported dask graph-construction slowdown when possible; done means sorted coordinates avoid repeated lexsorting while unsorted coordinates retain current behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100