Extending Window capabilities
- Dominant language
- Scala
- Stars
- 1.2k
- Forks
- 199
- PR merge metrics
- No merged PRs in 30d
Description
I would like to extend the window capabilities and like to discuss how to best implement these. Considering existing functionality we can do:
```
val result = priceTSRdd.addWindows(Window.pastAbsoluteTime("1000ns"))
// time price window_past_1000ns
// ------------------------------------------------------
// 1000L 1.0 [[1000L, 1.0]]
// 1500L 2.0 [[1000L, 1.0], [1500L, 2.0]]
// 2000L 3.0 [[1000L, 1.0], [1500L, 2.0], [2000L, 3.0]]
// 2500L 4.0 [[1500L, 2.0], [2000L, 3.0], [2500L, 4.0]]
```
1) Window at predefined time stamps only
This creates a window at each row backward. For very "dense" time series with samples at nano scale we might not do the window at each observation but run some statistics or other discovery method to find those points at which we want to create a window.
2) Windows of varying length
In the trading world we can imagine windows of varying time length, e.g. determined by a "volume clock"
3) Windows of fixed number of observations. I saw a count window but not clear to me how to use it.
4) Two segment windows.
A first segment (section) of a window could be used to calculate some online statistics which are then consumed by a summary function which is applied over the second part of the window (adaptive summary stats, e.g. consider thresholds based on an online volatility estimator).
How would these more general features best implemented? Any good advise how to add these extensions? Happy to contribute as well.
Contributor guide
Research direction
Start by reviewing the existing Window APIs and the count-window functionality referenced in the issue, then trace how addWindows processes time-series observations. Compare the proposed predefined timestamps, variable-length windows, fixed observation counts, and two-segment windows with current capabilities. Done would require an agreed design and implementation scope, but the issue does not name files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100