influxdata / influxdata/influxdb

Fill data using last Value

Open
#23,773 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Proposal:__

I want to have a fill function such aggregation is computed using last value seen

2022-09-10T15:00:00.100Z -> 1
2022-09-10T15:00:00.700Z -> 2
---- No entries as value is still the same ----
2022-09-10T15:00:05.000Z -> 3
2022-09-10T15:00:05.500Z -> 2

If we build 1 second time series with this Raw data

Minimum aggregation with fill(usePrevious: true)

2022-09-10T15:00:00.000Z -> 1
2022-09-10T15:00:01.000Z -> 1
2022-09-10T15:00:02.000Z -> 1
2022-09-10T15:00:03.000Z -> 1
2022-09-10T15:00:04.000Z -> 1
2022-09-10T15:00:05.000Z -> 2

Minimun aggregation with fill and use "lastValue"

2022-09-10T15:00:00.000Z -> 1
2022-09-10T15:00:01.000Z -> 2 (As last value of the series is 2022-09-10T15:00:00.700Z -> 2)
2022-09-10T15:00:02.000Z -> 2 (As last value of the series is 2022-09-10T15:00:00.700Z -> 2)
2022-09-10T15:00:03.000Z -> 2 (As last value of the series is 2022-09-10T15:00:00.700Z -> 2)
2022-09-10T15:00:04.000Z -> 2 (As last value of the series is 2022-09-10T15:00:00.700Z -> 2)
2022-09-10T15:00:05.000Z -> 2

So if no entry, then last value should be used for computing the aggregation

__Current behavior:__
When using fill(usePrevious = true) with an aggregate window, fill function is returning last aggregated data for interval.
If there is not data within interval, the aggregate function is using previous value computed.

In this case, we are working with subscriptions to our device, this means we are only storing data if the value changes.
So I can have null values if there is no value change within time frame.

As you can see in the image, max, average and minimum are getting last computed value at the end of the chart, as there is notdata.

![InfluxDBIssue](https://user-images.githubusercontent.com/35955278/193588797-d5c8218a-e5fb-4d73-a93e-ccc6ad6fb9eb.PNG)

__Desired behavior:__
I would like to have some function that does not pick the last computed value, but it takes the real last value for building the new aggregation window with it, so functions like mean, min, max are not taking last mean, min, max value, but are equal to last value stored in Influxdb for the series.

![InfluxDBIssue_expected](https://user-images.githubusercontent.com/35955278/193588812-dd7305e6-ee97-4293-a007-d7449d4364d6.PNG)

So mean, max, min are computed taking as reference 1 single record with the last Value stored for it. In case of **mean, max and min** will be the same, as value has not changed and we only consider one record with las value seen

In case of functions like **stddev**, the standard deviation considering only 1 record with lastValue seen will be '0'

__Alternatives considered:__

__Use case:__

When working with IoT and subscription, we use to subscribe to variables and store new values only on change.

If we want to create a complete series from this "Raw Data", when empty values appears it means that no change on data has occurred, and therefore, instead using last computed data for aggregation, aggregation should be computed using las real "raw value" for the measurement.

Contributor guide

Open the contributing guide

Research direction

Start by tracing fill(usePrevious: true) through the aggregate-window query path. Compare the current use of the last aggregated value with the requested last raw value semantics, including mean, min, max, and stddev. Done means empty intervals use the latest stored raw value when computing their aggregation, as shown in the examples.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.