vectordotdev / vectordotdev/vector

New `rolling` transform

Open
#1,329 8 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

domain: data model domain: logs domain: metrics domain: processing domain: transforms needs: investigation needs: requirements needs: rfc transform: new
Dominant language
Rust
Stars
22.6k
Forks
2.3k
Avg merge
1d 7h
Merged PRs (30d)
146

Description

It would be nice to have a native transform which can perform rolling window calculations on events fields.

Use cases

One simple and common potential use case is combining it with fixed source (https://github.com/timberio/vector/issues/1317) to generate a sequence of events with increasing counter.

A more complex use case is doing simple smoothing or counting in Vector before sending the data to sinks, for example, computing moving averages with subsequent sampling.

Configuration

It would take a field and then calculate one of the following functions on the values of this field:

  • count - count number of events in which the source field was present;
  • sum - sum values of the source field;
  • average - calculate average of the values of the source field;
  • rms - calculate residual mean square (sqrt(average(x^2) - average(x)^2)).

The rolling window could take one of the following types

  • expanding - the window contains all timeseries points from the beginning;
  • exponential - the window weights timeseries points with exponential decay;
  • rectangular - the window uses either fixed number of points or fixed time interval.

The size of the window could be specified using one of the following configuration options:

  • window_points - specifies size of the window as number of events;
  • window_duration - specifies size of the window as time interval.

It should be possible to specify additional parameter window_field, which would:

  • for window_points use number of times the field window_field occured instead of total number of messages;
  • for window_duration use the field window_field as timestamp using which the duration is determined.

Additionally, there should be a Boolean parameter persistent, which would specify should the state of the window function stored on disk or not.

Example (common)

[transforms.rolling]
type = "rolling"
function = "count"
target_field = "count"

Example (advanced)

[transforms.rolling]
type = "rolling"
function = "average" # other values are `sum`, `average`, `rms`
window_type = "expanding" # other values are `exponential` and `rectangular`
window_time = 3
persistent = true # false by default

field = "message" # source field, optional for `count` function
target_field = "message_count" # optional if `field` is specified
drop_field = false # optional

References

Inspired by the rolling window API in Pandas.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No implementation files, tests, or entry points are named. Start by reading the existing transform architecture and the fixed source issue linked under Use cases, then map the requested functions, window types, configuration options, and persistence behavior to project conventions. Done means the rolling transform supports the specified examples and has coverage for its configuration and calculations.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.