pydata / pydata/xarray

Implement rolling_exp for dask arrays

Open
#4,474 7 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Is your feature request related to a problem? Please describe.
I use dask-based chunking on my arrays regularly and would like to leverage the efficient numbagg implementation of move_exp_nanmean() with rolling_exp().

Describe the solution you'd like
It's possible to compute a rolling exp mean as a function of rolling exp means of contiguous, non-overlapping subsets (chunks). You just need to first "un-normalize" the rolling_exps of each chunk in order to split them into their corresponding numerators and denominators (see the ewm definition here under adjust=True). The normalization factor (denominator) to multiply back in to the chunk's move_exp_nanmean() in order to un-normalize it (numerator) is just the move_exp_nanmean() of 1's, replaced with NA's wherever the underlying data was also NA.

Then, scale each chunk's numerator and denominator series (derived from their move_exp_nanmean() series via above) down according to how many "lags-ago" they were, sum the rescaled numerators and denominators across chunks, and finally divide the total summed numerators and denominators.

Describe alternatives you've considered
I implemented my own inefficient weighted rolling mean using xarray's rolling(). This requires a bunch of duplicate computation as the window gets shifted.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating rolling_exp() and the dask-array rolling implementation, then review the referenced pandas ewm definition and numbagg's move_exp_nanmean(). The issue describes chunk-wise numerator and denominator handling; done means rolling_exp() supports dask arrays with results consistent with the adjusted exponential mean.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.