twosigma / twosigma/flint

Extending Window capabilities

Open
#36 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.2k
Forks
199
PR merge metrics
No merged PRs in 30d

Description

I would like to extend the window capabilities and like to discuss how to best implement these. Considering existing functionality we can do:

```
val result = priceTSRdd.addWindows(Window.pastAbsoluteTime("1000ns"))
// time price window_past_1000ns
// ------------------------------------------------------
// 1000L 1.0 [[1000L, 1.0]]
// 1500L 2.0 [[1000L, 1.0], [1500L, 2.0]]
// 2000L 3.0 [[1000L, 1.0], [1500L, 2.0], [2000L, 3.0]]
// 2500L 4.0 [[1500L, 2.0], [2000L, 3.0], [2500L, 4.0]]
```

1) Window at predefined time stamps only
This creates a window at each row backward. For very "dense" time series with samples at nano scale we might not do the window at each observation but run some statistics or other discovery method to find those points at which we want to create a window.

2) Windows of varying length
In the trading world we can imagine windows of varying time length, e.g. determined by a "volume clock"

3) Windows of fixed number of observations. I saw a count window but not clear to me how to use it.

4) Two segment windows.
A first segment (section) of a window could be used to calculate some online statistics which are then consumed by a summary function which is applied over the second part of the window (adaptive summary stats, e.g. consider thresholds based on an online volatility estimator).

How would these more general features best implemented? Any good advise how to add these extensions? Happy to contribute as well.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing Window APIs and the count-window functionality referenced in the issue, then trace how addWindows processes time-series observations. Compare the proposed predefined timestamps, variable-length windows, fixed observation counts, and two-segment windows with current capabilities. Done would require an agreed design and implementation scope, but the issue does not name files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.