pydata / pydata/xarray

indexing by a list of slices

Open
#10,479 11 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

API design design question enhancement topic-indexing topic-lazy array
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Is your feature request related to a problem?

In some situations, it is important to be able to select multiple (disconnected) contiguous regions along a dimension, even if the dimension itself is very large.

If there's enough client memory, it is possible to emulate this by materializing the slices into an array of integers, but this becomes infeasible if that array is too large (and while supposedly we should be able to index by a dask integer array, I'm not sure how efficient that would be).

Examples of where this would be useful include:

  • The healpix MOC index at xarray-contrib/xdggs#151, where cell ids are represented as a set of disconnected ranges at the smallest possible refinement level. To be able to support Index.sel, I'd need to return a IndexSelResult with either a list of slices, or materialize these into an integer array and error out if that wouldn't fit into memory (or try to use dask as an indexer).
  • @tomwhite's use-case of selecting disconnected regions in a genome (see https://github.com/sgkit-dev/sgkit/pull/1330#discussion_r2177566488). I'll let him provide further details.

cc @benbovy, @shoyer, @TomNicholas, @dcherian

Describe the solution you'd like

I'd love to be able to specify this as another kind of indexer:

indexer = SliceSet([slice(20, 5000), slice(12078432, 1850372894)])
ds.isel(cells=indexer)

but that will obviously further increase the complexity of the indexing machinery

Describe alternatives you've considered

Manually iterating of the slices, then concatenating the result is possible, but will have an additional overhead if done using the xarray API. However, I don't see a way that can work as part of IndexSelResult.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the indexing machinery around IndexSelResult and ds.isel, then compare the existing slice and integer-array indexer paths. Define how disconnected slices should be represented without materializing a large integer array, and validate the design against the healpix MOC and genome-selection use cases described.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.