Implement rolling_exp for dask arrays
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Is your feature request related to a problem? Please describe.
I use dask-based chunking on my arrays regularly and would like to leverage the efficient numbagg implementation of move_exp_nanmean() with rolling_exp().
Describe the solution you'd like
It's possible to compute a rolling exp mean as a function of rolling exp means of contiguous, non-overlapping subsets (chunks). You just need to first "un-normalize" the rolling_exps of each chunk in order to split them into their corresponding numerators and denominators (see the ewm definition here under adjust=True). The normalization factor (denominator) to multiply back in to the chunk's move_exp_nanmean() in order to un-normalize it (numerator) is just the move_exp_nanmean() of 1's, replaced with NA's wherever the underlying data was also NA.
Then, scale each chunk's numerator and denominator series (derived from their move_exp_nanmean() series via above) down according to how many "lags-ago" they were, sum the rescaled numerators and denominators across chunks, and finally divide the total summed numerators and denominators.
Describe alternatives you've considered
I implemented my own inefficient weighted rolling mean using xarray's rolling(). This requires a bunch of duplicate computation as the window gets shifted.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating rolling_exp() and the dask-array rolling implementation, then review the referenced pandas ewm definition and numbagg's move_exp_nanmean(). The issue describes chunk-wise numerator and denominator handling; done means rolling_exp() supports dask arrays with results consistent with the adjusted exponential mean.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100