Column-wise `dask` `map_blocks`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 779
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 27
Description
What kind of feature would you like to request?
Additional function parameters / changed functionality / changed defaults?
Please describe your wishes
Almost all of our dask functionality relies implicitly or explicitly on row-wise chunking but likely many of the algorithms can be adapted to column-wise chunking.
For example (not a comprehensive list but should highlight the general procedure):
- https://github.com/scverse/scanpy/pull/3700 -
sc.get.aggregatecan usecscmatrices and break up the computation across features (since the computations are independent over features), and then concatenate. - PCA can likely be done as a multi-pass algorithm over CSC matrices - not efficient but doable.
- HVG in general operates as a feature-space algorithm and, aside from seurat v3, really only relies on a mean-var calculation which is already CSC compatible (even seurat v3 does but in this case, also has this additional
loessstep). The mark here would be seurat v3/batched HVG selection where row-wise chunking is actually bad for the computation since it requires (likely) random subsets. In this case, proceeding in a chunked manner i.e., chunk-of-genes by chunk-of-genes probably would not be too bad. -
top_segment_proportions(i.e., from thepercent_topargument incalculate_qc_metrics) could be done in feature-wise chunks as well, and then concatenated at the end
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing column-wise work in sc.get.aggregate and compare it with the listed candidates: PCA, HVG, and top_segment_proportions in calculate_qc_metrics. Determine which algorithm has a concrete scope and establish how feature-wise chunking, concatenation, and CSC compatibility should be validated before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- bioinformatics, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100