Support segmented scans
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Is your feature request related to a problem?
It is pretty common to want to run cumsum and have the sum reset when a boolean flag array is 1. This is so common it has its own Wikipedia page and is discussed in Blelloch (1993) (Section 1.5)
Here's a real example of someone trying to implement it in a fairly roundabout way.
time_cumsum = cube.cumsum(dim = 'time')
cumsum = time_cumsum - time_cumsum.where(cube== 0).ffill(dim = 'time').fillna(0)
We have a few options to implement it:
-
We could introduce a new method
DataArray.segmented_scan(flags, op="sum")or a new classDataArray.segment.cumsum()? A dask/cubed friendly version that does all of this in a single scan should be fairly straightforward to write (and similar to ourffill,bfillwrappers). -
In a way this generalizes
resampleand it just struck me that the example above could be written as the following, which should be OK once flox adds scansgroup_idx = (cube == 0).cumsum('time') cubed.groupby(group_idx).cumsum()- We could use our new
Grouperfunctionality to expose a "flag" grouper that hides thegroup_idx = (cube == 0).cumsum('time')line.
- We could use our new
My concern with (2) and (2.i) is that they are not at all obvious for most of our userbase.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue does not name implementation files or tests. Start by comparing the proposed DataArray.segmented_scan API with the flag Grouper approach, and review the existing cumsum and ffill wrappers mentioned in the issue. Done requires a decided API and a dask/cubed-friendly segmented scan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100