scverse / scverse/fast-array-utils
More precise mean_var for non-Dask dense arrays?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15
- Forks
- 5
- Avg merge
- 10h 33m
- Merged PRs (30d)
- 9
Description
In #127, I changed how mean_var works just for Dask:
We already have a custom numba implementation for sparse-in-core which operates on float64s, but doesn’t convert everything at once.
We could make the change also affect dense-in-core. Should we
- leave things as they are and keep
powerfor float32 dense arrays as float32? It takes less space and is faster, but less precise. - do exponentiation in 64 bit everywhere? this would allocate an exponentiated array of the input size in 64 bit, but I guess if people’s machines can handle a temporary 2n (prod(shape)×32×2), they can handle a temporary 3n (prod(shape)×32 + prod(shape)×64)?
- implement a custom numba implementation for dense-in-core as well?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading src/fast_array_utils/stats/_mean_var.py around line 40 and the change described in #127. Compare the dense-in-core, Dask, and sparse-in-core paths, focusing on float32 versus float64 exponentiation and temporary allocations. Done means selecting and documenting one precision strategy for dense arrays, with its memory and performance trade-offs clear.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100