vectordotdev / vectordotdev/vector
New `rolling` transform
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
It would be nice to have a native transform which can perform rolling window calculations on events fields.
Use cases
One simple and common potential use case is combining it with fixed source (https://github.com/timberio/vector/issues/1317) to generate a sequence of events with increasing counter.
A more complex use case is doing simple smoothing or counting in Vector before sending the data to sinks, for example, computing moving averages with subsequent sampling.
Configuration
It would take a field and then calculate one of the following functions on the values of this field:
count- count number of events in which the source field was present;sum- sum values of the source field;average- calculate average of the values of the source field;rms- calculate residual mean square (sqrt(average(x^2) - average(x)^2)).
The rolling window could take one of the following types
expanding- the window contains all timeseries points from the beginning;exponential- the window weights timeseries points with exponential decay;rectangular- the window uses either fixed number of points or fixed time interval.
The size of the window could be specified using one of the following configuration options:
window_points- specifies size of the window as number of events;window_duration- specifies size of the window as time interval.
It should be possible to specify additional parameter window_field, which would:
- for
window_pointsuse number of times the fieldwindow_fieldoccured instead of total number of messages; - for
window_durationuse the fieldwindow_fieldas timestamp using which the duration is determined.
Additionally, there should be a Boolean parameter persistent, which would specify should the state of the window function stored on disk or not.
Example (common)
[transforms.rolling]
type = "rolling"
function = "count"
target_field = "count"
Example (advanced)
[transforms.rolling]
type = "rolling"
function = "average" # other values are `sum`, `average`, `rms`
window_type = "expanding" # other values are `exponential` and `rectangular`
window_time = 3
persistent = true # false by default
field = "message" # source field, optional for `count` function
target_field = "message_count" # optional if `field` is specified
drop_field = false # optional
References
Inspired by the rolling window API in Pandas.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No implementation files, tests, or entry points are named. Start by reading the existing transform architecture and the fixed source issue linked under Use cases, then map the requested functions, window types, configuration options, and persistence behavior to project conventions. Done means the rolling transform supports the specified examples and has coverage for its configuration and calculations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100