influxdata / influxdata/influxdb
timeWeightedAvg should not be greater than maximal value / integral should not extrapolate based on incline
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
Consider the following query,
```flux
import "date"
import "array"
t0 = today()
t1 = date.add(to: t0, d: 11h)
t2 = date.add(to: t0, d: 12h)
t3 = date.add(to: t0, d: 13h)
end = date.add(to: t0, d: 1d)
input = array.from(rows: [{_time: t1, _value: 1},{_time: t2, _value: 0},{_time: t3, _value: 1}])
input
|> range(start: t0, stop: end)
|> yield(name: "input")
|> timeWeightedAvg(unit: 1s)
|> duplicate(column: "_start", as: "_time")
|> yield(name: "timeWeightedAvg")
input
|> range(start: t0, stop: end)
|> window(every: 1d)
|> integral(unit: 1d, interpolate:"linear")
|> duplicate(column: "_start", as: "_time")
|> yield(name: "integral")
```
__Expected behavior:__
The timeWeightedAvg should not be greater than the maximal input value and either be computed only within the existing data points or extrapolate more reasonably and not as sensitive on fluctuations on the boundary values.
__Actual behavior:__
The timeWeightedAvg/integral produces nonsensical values, **6** in the above example even though all values are between 0 and 1. It seems integral extrapolates based on the incline of the boundary values making it extremely sensitive to fluctuations.

This gets worse if the fluctuations at bounding data points happen in a shorter timespan. If we make the window during which the data points are 0 bigger the integral explodes:
```flux
t0 = today()
t1 = date.add(to: t0, d: 11h)
t2 = date.add(to: t0, d: 11h1s)
t3 = date.add(to: t0, d: 13h)
t4 = date.add(to: t0, d: 13h1s)
end = date.add(to: t0, d: 1d)
input = array.from(rows: [{_time: t1, _value: 1},{_time: t2, _value: 0},{_time: t3, _value: 0},{_time: t4, _value: 1}])
```
This **yields a timeWeightedAvg of 18150** even though all data points are still between 0 and 1 and the function is 0 for most of the time while there is data.

This makes the timeWeightedAvg and integral mostly useless in scenarios with unevenly distributed data points (like IoT data) where one usually would especially want a time weighted average.
__Environment info:__
* InfluxDB version: 2.5.1
Contributor guide
Research direction
Start by running the supplied Flux reproducer against the timeWeightedAvg and integral functions and compare the results with the stated bounds. Trace the implementation and existing tests for these functions; done means unevenly spaced values no longer produce results such as 6 or 18150 when all inputs are between 0 and 1.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100