influxdata / influxdata/kapacitor
Query + Join Issue
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
I am seeing some weird behavior/issue that i am unable to explain fully.
* I have a TickScript that runs 2 queries against influx table and joins the results that later get inserted into another table in influx. The queries have a “where clause”, run every(1m), period(30m), groupby(5m)
* When i insert the data into the parent table in influx, my kapacitor script inserts the joined data correctly into the new table in influx. As i insert new data into my parent table, kapacitor does the calculations correctly for the first 30 minutes or so. After that as my script runs every minute, slowly the results start changing for the previously correct data.
* Please see my tick script below.
* To reproduce: insert the below data at random intervals every few minutes or so.
meters,meterid=m1 value=1
meters,meterid=m2 value=1
* As you run a “select * on formula1” and you will notice that the data is correct for sometime and then slowly the data in the previously correct slots changes even though we have not inserted any more records into those slots.
* I think the issue is being caused by the “period” when the query is issued to influx. Every time a query is issued the time range changes with the clock(ex: time between 4:02 and 4:32, next time its time between 4:03 and 4:33) . This might be causing influx to return different results depending upon when the record was inserted even if it was in the correct 5m bucket. So, when the job runs 4:31, query is going to return rows inserted between 4:01 and 4:05 but when it runs again at 4:32 the query returns data between 4:02 and 4:05. So i think any records inserted between 4:01 and 4:02 will not be returned by the query.
* I think the query start time should be adjusted to match the groupby bucket.
* Would appreciate some pointers.
== script ==
dbrp “multi”.“autogen”
var series1 = batch
|query(’’‘SELECT mean(value) FROM “multi”.“autogen”."meters"
where meterid = ‘M1’ ‘’’)
.period(30m)
.every(1m)
.groupBy(time(5m) )
.align()
.fill(0)
var series2 = batch
|query(’’‘SELECT mean(value) FROM “multi”.“autogen”."meters"
where meterid = ‘M2’ ‘’’)
.period(30m)
.every(1m)
.groupBy(time(5m) )
.align()
.fill(0)
series1
|join(series2)
.as(‘M1’,‘M2’)
.fill(0)
|eval(lambda: float(“M1.mean”) * 5.0 + float(“M2.mean”) )
.as(‘value’)
.keep()
|influxDBOut()
.database(‘multi’)
.retentionPolicy(‘autogen’)
.measurement(‘formula1’)
.tag(‘seriesid’,‘Snew’)
Contributor guide
Research direction
Start by reproducing the TickScript with the supplied meters data and inspect the batch query behavior for period(30m), every(1m), groupBy(time(5m)), and align(). Compare successive formula1 results as the query window moves. Done means establishing whether bucket results change without new points and identifying the relevant Kapacitor behavior or correction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100