scverse / scverse/squidpy

In place shuffle dedup

Open
#1,289 0 comments 0 reactions 1 assignee View on GitHub

@selmanozleyen is already working on this.

Since Sep 10, 2026.

Dominant language
Python
Stars
598
Forks
121
Avg merge
3d 11h
Merged PRs (30d)
3

Description

Here is an AI generated text. Numba's Generator is very slow it turns out and we can write our own. But I will first wait for #1023 to be merged. It is bit-reproducable so it's not a breaking change. Should probably also open an issue about this in numba repo

Numba's Generator.shuffle overload is about twice as slow as the primitive it is built on: it copies through a 0-d array buffer for every swap (numba/np/random/generator_methods.py:188-196, buf[...] = z[j, ...] plus two more slice assignments) where a scalar swap would do, which costs roughly as much as the random draws themselves. Replacing it with an inline Fisher–Yates loop over the same random_interval draws, in the same descending order, is bit-identical — same permutation, same number of draws consumed, so a shared generator stays in step — and measures 2.0x faster at n=10,000 (0.080 ms vs 0.161 ms) and 2.2x at n=100,000 (0.727 ms vs 1.566 ms); it also beats NumPy's own Generator.shuffle, so this is worth an upstream numba issue independently. Squidpy has exactly two in-numba shuffle sites, _ligrec._score_permutations and _nhood._shuffled_labels, both shuffling a label array inside a prange loop from a typed list of generators, so both get the win; in nhood_enrichment the shuffle is roughly 70% of per-permutation work, and the indexed variant additionally removes the gather/scatter through group_indices for the library_key path (2.3-2.7x there) and makes the single-group special case redundant, so it can be deleted along with its test. The one thing to declare in the PR: it imports random_interval from numba.np.random.random_methods, a numba internal — if numba moves it the import fails loudly at module load, and the bit-identity tests would catch any change in behaviour.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.