scverse / scverse/scanpy

Column-wise `dask` `map_blocks`

Open
#3,723 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Meta
Dominant language
Python
Stars
2.6k
Forks
779
Avg merge
1d 4h
Merged PRs (30d)
27

Description

What kind of feature would you like to request?

Additional function parameters / changed functionality / changed defaults?

Please describe your wishes

Almost all of our dask functionality relies implicitly or explicitly on row-wise chunking but likely many of the algorithms can be adapted to column-wise chunking.

For example (not a comprehensive list but should highlight the general procedure):

  • https://github.com/scverse/scanpy/pull/3700 - sc.get.aggregate can use csc matrices and break up the computation across features (since the computations are independent over features), and then concatenate.
  • PCA can likely be done as a multi-pass algorithm over CSC matrices - not efficient but doable.
  • HVG in general operates as a feature-space algorithm and, aside from seurat v3, really only relies on a mean-var calculation which is already CSC compatible (even seurat v3 does but in this case, also has this additional loess step). The mark here would be seurat v3/batched HVG selection where row-wise chunking is actually bad for the computation since it requires (likely) random subsets. In this case, proceeding in a chunked manner i.e., chunk-of-genes by chunk-of-genes probably would not be too bad.
  • top_segment_proportions (i.e., from the percent_top argument in calculate_qc_metrics) could be done in feature-wise chunks as well, and then concatenated at the end

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing column-wise work in sc.get.aggregate and compare it with the listed candidates: PCA, HVG, and top_segment_proportions in calculate_qc_metrics. Determine which algorithm has a concrete scope and establish how feature-wise chunking, concatenation, and CSC compatibility should be validated before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
bioinformatics, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.