influxdata / influxdata/influxdb

timeWeightedAvg should not be greater than maximal value / integral should not extrapolate based on incline

Open
#23,929 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
Consider the following query,

```flux
import "date"
import "array"

t0 = today()
t1 = date.add(to: t0, d: 11h)
t2 = date.add(to: t0, d: 12h)
t3 = date.add(to: t0, d: 13h)
end = date.add(to: t0, d: 1d)

input = array.from(rows: [{_time: t1, _value: 1},{_time: t2, _value: 0},{_time: t3, _value: 1}])

input
|> range(start: t0, stop: end)
|> yield(name: "input")
|> timeWeightedAvg(unit: 1s)
|> duplicate(column: "_start", as: "_time")
|> yield(name: "timeWeightedAvg")

input
|> range(start: t0, stop: end)
|> window(every: 1d)
|> integral(unit: 1d, interpolate:"linear")
|> duplicate(column: "_start", as: "_time")
|> yield(name: "integral")
```

__Expected behavior:__
The timeWeightedAvg should not be greater than the maximal input value and either be computed only within the existing data points or extrapolate more reasonably and not as sensitive on fluctuations on the boundary values.

__Actual behavior:__
The timeWeightedAvg/integral produces nonsensical values, **6** in the above example even though all values are between 0 and 1. It seems integral extrapolates based on the incline of the boundary values making it extremely sensitive to fluctuations.

![grafik](https://user-images.githubusercontent.com/1318553/202898559-8f9c8e29-f01a-441c-a981-79ba005f3287.png)

This gets worse if the fluctuations at bounding data points happen in a shorter timespan. If we make the window during which the data points are 0 bigger the integral explodes:
```flux
t0 = today()
t1 = date.add(to: t0, d: 11h)
t2 = date.add(to: t0, d: 11h1s)
t3 = date.add(to: t0, d: 13h)
t4 = date.add(to: t0, d: 13h1s)
end = date.add(to: t0, d: 1d)

input = array.from(rows: [{_time: t1, _value: 1},{_time: t2, _value: 0},{_time: t3, _value: 0},{_time: t4, _value: 1}])
```
This **yields a timeWeightedAvg of 18150** even though all data points are still between 0 and 1 and the function is 0 for most of the time while there is data.

![grafik](https://user-images.githubusercontent.com/1318553/202898623-9af068cd-dc69-4ad8-9eda-a3f9cf017489.png)

This makes the timeWeightedAvg and integral mostly useless in scenarios with unevenly distributed data points (like IoT data) where one usually would especially want a time weighted average.

__Environment info:__

* InfluxDB version: 2.5.1

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied Flux reproducer against the timeWeightedAvg and integral functions and compare the results with the stated bounds. Trace the implementation and existing tests for these functions; done means unevenly spaced values no longer produce results such as 6 or 18150 when all inputs are between 0 and 1.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.