influxdata / influxdata/influxdb

Tick scripts generates values from older dates for percentile calculation

Open
#24,461 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

We are using Kapacitor scripts to perform transformation on our measurement,
Sample script:

stream
|from()
.measurement('tracking_timeTaken')
|delete()
.field('timeTaken41')
.field('timeTaken42')
.field('id')
.field('document')
.tag('checkpoint')
|window()
.period(5m)
.every(5m)
|groupBy('itemType')
|percentile('processingTime', 95.0)
|log()
|influxDBOut()
.database(db_name)
.retentionPolicy(retention_policy)
.measurement('tracking_95p')

The measurement 'tracking_timeTaken' looks like
time itemType document id checkpoint timeTaken41 timeTaken42 processingTime
2023-11-14T14:51:30.442+05:30 docType1 docType1 testdata12345 inprocess 0 0 6285
2023-11-14T14:51:31.470+05:30 docType1 docType1 testdata12346 open 0 0 7309
2023-11-14T15:08:26.118+05:30 docType1 docType1 testdata12347 inprocess 0 0 4428
2023-11-14T15:08:27.128+05:30 docType1 docType1 testdata12348 completed 0 0 5404
2023-11-14T14:55:11.012+05:30 docType2 docType2 testdata12349 inprocess 0 0 3400
2023-11-15T01:54:13.956+05:30 docType2 docType2 testdata12350 inprocess 0 0 11443
2023-11-15T02:00:56.706+05:30 docType2 docType2 testdata12351 error 0 0 4779

Expected behaviour: For itemType docType1 the response measurement tracking_95p considering 5mins window should contain values
time itemType percentile
2023-11-14T15:08:27.128+05:30 docType1 5404
2023-11-14T14:51:31.470+05:30 docType1 7309

Actual behaviour: However, in response we are getting older timestamps which are not present in source measurement
time itemType percentile
2023-11-13T15:14:44.015+05:30 docType1 6285
2023-11-14T02:05:56.706+05:30 docType1 7309
2023-11-14T15:14:45.025+05:30 docType1 7309

Here the time, '2023-11-13T15:14:44.015+05:30' is no where present in the source table and also 7309 is repeated.
Please share feedback and resolution tips

Contributor guide

Open the contributing guide

Research direction

Start with the supplied Kapacitor stream and compare the source timestamps with the window and percentile output. Reproduce the repeated values and unexpected timestamps using the example data, then trace how the window and percentile stages assign output times. Done means identifying the cause and documenting a verified correction or limitation.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.