indexing by a list of slices
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Is your feature request related to a problem?
In some situations, it is important to be able to select multiple (disconnected) contiguous regions along a dimension, even if the dimension itself is very large.
If there's enough client memory, it is possible to emulate this by materializing the slices into an array of integers, but this becomes infeasible if that array is too large (and while supposedly we should be able to index by a dask integer array, I'm not sure how efficient that would be).
Examples of where this would be useful include:
- The healpix MOC index at xarray-contrib/xdggs#151, where cell ids are represented as a set of disconnected ranges at the smallest possible refinement level. To be able to support
Index.sel, I'd need to return aIndexSelResultwith either a list of slices, or materialize these into an integer array and error out if that wouldn't fit into memory (or try to usedaskas an indexer). - @tomwhite's use-case of selecting disconnected regions in a genome (see https://github.com/sgkit-dev/sgkit/pull/1330#discussion_r2177566488). I'll let him provide further details.
cc @benbovy, @shoyer, @TomNicholas, @dcherian
Describe the solution you'd like
I'd love to be able to specify this as another kind of indexer:
indexer = SliceSet([slice(20, 5000), slice(12078432, 1850372894)])
ds.isel(cells=indexer)
but that will obviously further increase the complexity of the indexing machinery
Describe alternatives you've considered
Manually iterating of the slices, then concatenating the result is possible, but will have an additional overhead if done using the xarray API. However, I don't see a way that can work as part of IndexSelResult.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the indexing machinery around IndexSelResult and ds.isel, then compare the existing slice and integer-array indexer paths. Define how disconnected slices should be represented without materializing a large integer array, and validate the design against the healpix MOC and genome-selection use cases described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100