influxdata / influxdata/influxdb
movingMedian function
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Proposal:__
Similar to movingAverage, there should be a function for movingMedian to filter out anomalous data points. There have been requests for such a function in multiple issues, but I still see no way to accomplish this.
EDIT: Or, even more flexible, a function like aggregateWindow, but that you specify a number of points rather than a time range. Such that `movingWindow(fn: median, n: 3)` would be the same as my proposed `movingMedian(n: 3)`
__Current behavior:__
You must window the data by time, rather than by number of data points to get a moving median.
__Desired behavior:__
A way to window data by number of points, or a movingMedian function to do the desired filtering directly.
__Alternatives considered:__
Use an `aggregateWindow(every: 15m, period: 45m, fn: median)` or something of that nature. This works depending on how regular your data points are, but it also leaves several windows at the beginning and end of the window which only had less than 3 points when the median function was applied.
__Use case:__
We have some sensors that occasionally emit a wrong data point which isn't in line with the rest of the values emitted by the sensor. Being able to grab the moving median of 3 points ensures we never do any calculations which use the spurious points.
Contributor guide
Research direction
Start by reviewing the existing movingAverage and aggregateWindow functions mentioned in the issue, including how they handle point windows and incomplete windows. Compare the proposed movingMedian(n: 3) and movingWindow(fn: median, n: 3) behaviors, then define the supported API and verify that sensor data can be filtered by a fixed number of points.
Written by the indexing model from the issue text.
Assessment
- Domain
- data, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100