xarray-contrib / xarray-contrib/xvec
Spatial partitioning, sorting and shuffling
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 139
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Description
When dealing with large sets of geometries it would be nice if we could partition (chunk) the geometry coordinate and the GeometryIndex based on spatial locality (thus requiring spatial sorting or shuffling), like explained for geo-dataframes in dask-geopandas' spatial partitioning user guide.
This would require a good amount of work both here and upstream, though:
- refactor
xarray.IndexVariableto allow dask arrays or other lazy arrays https://github.com/pydata/xarray/pull/8124 (although we can work around this by using regularxarray.Variableobjects for now) - add some sort of generic, out-of-core index in Xarray https://github.com/pydata/xarray/issues/1650, or add our own ad-hoc version here
- compute space-filling curve values from geometries (copied or factored out from dask-geopandas?)
- implement spatial shuffling, probably reusing what is being added in Xarray https://github.com/pydata/xarray/pull/9320
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the dask-geopandas spatial partitioning guide, then review the linked xarray.IndexVariable PR and out-of-core index issue. Check the linked Xarray spatial-shuffling PR and determine how space-filling curve values and chunked geometry indexes should fit in xvec. Done means spatially partitioned geometry coordinates and GeometryIndex support the intended out-of-core sorting and shuffling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100