Memory-efficient alternative of np.count_nonzero(mask_func(x))
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 32.8k
- Forks
- 12.8k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 197
Description
Feature
I've encountered quite a few cases in my work when it was necessary to count number of results of some ufunc that evaluates to True on the input array without knowing where.
E.g. if mask_func = np.isnan I will get the number of NaNs in array, if np.isinf - number of infinities etc.
Currently the most runtime efficient way is np.count_nonzero(mask_func(x)) but it creates bool array as a middle step that doesn't seems necessary in this usecase - not only it spends runtime constructing the array only to discard it right away, it also consumes memory which can be a problematic when you deal with huge arrays in memory.
Is it possible to implement this counter in a more memory efficient way - instead of all the hassle of creating intermittent array it will just increment int var for each calculated True and return the result.
When implemented outside the numpy in Python iteration over each element with np.nditer is much slower, so currently the most memory efficient way to do it without sacrificing performance for me is chunking.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining the existing np.count_nonzero entry point and compare it with np.isnan and np.isinf on large arrays. Determine how a counting operation could avoid retaining the intermediate boolean array while preserving the current behavior and performance goals. Done means the requested count is available without the memory cost of np.count_nonzero(mask_func(x)), with coverage for the described use cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100