Can I speed up the spatialdata.polygon_query when dealing with really big datasets?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 394
- Forks
- 95
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 7
Description
Hello,
thank you for providing the community with such a great tool!
I was wondering if you could help or advise on the following.
I have a very large xenium dataset (~2 million cells). I have read it and saved it as a zarr file and I have read it back in on my notebook. At the same time I have used QuPath to create annotations for my images, I have saved them as geojson files and read them back in as a geodataframes and subsequently converted them into polygons.
I would like to add this type of metadata information in my sdata object, so that I can annotate and filter my sdata.table based on that annotation. The way that I am currently trying to do this is by using the following piece of code, really in an effort to create subsets of my dataset based on polygons which I can then assign the annotation.
from spatialdata import polygon_query # type: ignore
cropped_sdata2 = polygon_query(
sdata,
polygon=polygon,
target_coordinate_system="global"
)
This piece of code is extremely slow because of the size of the dataset, so I was wondering if there is a way to further speed up this function (I am already running this analysis on a server), or if there is a simpler way incorporate my annotation to the sdata object.
Thanks a lot for your help!
Cheers,
Anastasia
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the public spatialdata.polygon_query entry point and reproduce the reported slowdown with a large Xenium dataset, a Zarr-backed sdata object, and the GeoJSON-derived polygon. Measure the current query behavior and identify a concrete, testable performance target or simpler annotation workflow; the issue does not name implementation files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100