Improve the shuffling/preprocessing code
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 68
- Forks
- 6
- Avg merge
- 7h 24m
- Merged PRs (30d)
- 6
Description
The current shuffling code is very slow, reportedly. The first step should be profiling the code especially in the wake of https://github.com/scverse/anndata/pull/2005. Other things should be considered in light of profiling
- Try out https://zarrs-python.readthedocs.io/en/stable/ for the writing if that appears to be the bottleneck
- Trying out a non-dask implementation instead of dask for loading X so we don't have to deal with whatever overhead is there
- Understand if tiny shards obviate the need for a complete reshuffle, in which case https://github.com/laminlabs/arrayloaders/blob/main/arrayloaders/io/store_creation.py#L16-L51 will likely suffice and we can simply re-write the data with tiny shards
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by profiling the shuffling and preprocessing code, especially in light of scverse/anndata pull request 2005. Compare the listed options, including zarrs-python for writing, a non-dask X loader, and the tiny-shard approach in arrayloaders/io/store_creation.py. Done means the bottleneck is identified and a measured direction is established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100