Optimize ndrolling nanreduce
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
In #4219 we added ndrolling.
However, nanreduce, such as ds.rolling(x=3, y=2).mean() calls np.nanmean which copies the strided-array into a full-array.
This is memory-inefficient.
We can implement inhouse-nanreduce methods for the strided array.
For example, our .nansum currently does
make a strided array -> copy the array -> replace nan by 0 -> sum
but we can do instead
replace nan by 0 -> make a strided array -> sum
This is much more memory efficient.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the ndrolling implementation and the nanreduce path used by ds.rolling(x=3, y=2).mean(), then compare it with the existing nansum behavior described in the issue. Done means the relevant reductions avoid copying the strided array into a full array while preserving their nan-handling results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100