pydata / pydata/xarray

perf improvement for interp: set `assume_sorted` automatically

Open
#9,758 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-interpolation topic-performance
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What is your issue?

assume_sorted is False, so for vectorized interpolation across multiple dimensions, we end up lexsorting the coordinates all the time. For some reason, this can be quite slow with dask.

https://github.com/pydata/xarray/blob/6df8bd606a8a9a3378c7672c087e08ced00b2e15/xarray/core/dataset.py#L4081

Instead we should be able to do

obj = self
# sort by slicing if we can
for coord in set(indexers) and set(self._indexes):
    # TODO: better check for PandasIndex
    if self.indexes[coord].is_monotonic_decreasing:
        obj = obj.isel(coord: slice(None, None, -1))

# TODO: make None the new default
if assume_sorted is None:
    # TODO: dims without coordinates are fine too
    assume_sorted = all(self.indexes[coord].is_monotonic_increasing for coord in indexers)

I'll add a reproducible example later, but the problem I've been playing gets much faster for graph construction:
image

xref #6799

cc @mpiannucci @Illviljan

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at xarray/core/dataset.py around line 4081 and inspect interp's indexer handling and PandasIndex monotonicity checks. Reproduce the reported dask graph-construction slowdown when possible; done means sorted coordinates avoid repeated lexsorting while unsorted coordinates retain current behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.