influxdata / influxdata/kapacitor

Query + Join Issue

Open
#1,816 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I am seeing some weird behavior/issue that i am unable to explain fully.

* I have a TickScript that runs 2 queries against influx table and joins the results that later get inserted into another table in influx. The queries have a “where clause”, run every(1m), period(30m), groupby(5m)

* When i insert the data into the parent table in influx, my kapacitor script inserts the joined data correctly into the new table in influx. As i insert new data into my parent table, kapacitor does the calculations correctly for the first 30 minutes or so. After that as my script runs every minute, slowly the results start changing for the previously correct data.

* Please see my tick script below.

* To reproduce: insert the below data at random intervals every few minutes or so.
meters,meterid=m1 value=1
meters,meterid=m2 value=1

* As you run a “select * on formula1” and you will notice that the data is correct for sometime and then slowly the data in the previously correct slots changes even though we have not inserted any more records into those slots.

* I think the issue is being caused by the “period” when the query is issued to influx. Every time a query is issued the time range changes with the clock(ex: time between 4:02 and 4:32, next time its time between 4:03 and 4:33) . This might be causing influx to return different results depending upon when the record was inserted even if it was in the correct 5m bucket. So, when the job runs 4:31, query is going to return rows inserted between 4:01 and 4:05 but when it runs again at 4:32 the query returns data between 4:02 and 4:05. So i think any records inserted between 4:01 and 4:02 will not be returned by the query.

* I think the query start time should be adjusted to match the groupby bucket.

* Would appreciate some pointers.

== script ==

dbrp “multi”.“autogen”

var series1 = batch
|query(’’‘SELECT mean(value) FROM “multi”.“autogen”."meters"
where meterid = ‘M1’ ‘’’)
.period(30m)
.every(1m)
.groupBy(time(5m) )
.align()
.fill(0)

var series2 = batch
|query(’’‘SELECT mean(value) FROM “multi”.“autogen”."meters"
where meterid = ‘M2’ ‘’’)
.period(30m)
.every(1m)
.groupBy(time(5m) )
.align()
.fill(0)

series1
|join(series2)
.as(‘M1’,‘M2’)
.fill(0)
|eval(lambda: float(“M1.mean”) * 5.0 + float(“M2.mean”) )
.as(‘value’)
.keep()
|influxDBOut()
.database(‘multi’)
.retentionPolicy(‘autogen’)
.measurement(‘formula1’)
.tag(‘seriesid’,‘Snew’)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the TickScript with the supplied meters data and inspect the batch query behavior for period(30m), every(1m), groupBy(time(5m)), and align(). Compare successive formula1 results as the query window moves. Done means establishing whether bucket results change without new points and identifying the relevant Kapacitor behavior or correction.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.