influxdata / influxdata/kapacitor

BATCH - Eval broken with join

Open
#1,302 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

kapacitor version : `Kapacitor 1.2.0 (git: master 5408057e5a3493d3b5bd38d5d535ea45b587f8ff)`
OS: `Ubuntu 16.04.1 LTS`
Kernel: `4.4.0-53-generic x86_64`

Hi Team,

When i'm using several batch with a join including eval, eval dont works.
Values
- newvaluepending
- newvalueok

are not present because eval not ran.

Maybe i'm doing something wrong but look like a bug around join using eval.

Thanks for your feedback. All informations are bellow.

See bellow my tick:

```
var data_pending_5min = batch
|query('''select last(value) as value from "metrics_business"."autogen"."reg" where metric='last5min' and result='Pending' ''')
.groupBy('channel','result')
.every(1m)
.period(5m)
.align()
|default()
.field('value', 0)
|log()
.prefix('[DEBUG]-1')
.level('DEBUG')

var data_ok_5min = batch
|query('''select last(value) as value from "metrics_business"."autogen"."reg" where metric='last5min' and result='OK' ''')
.groupBy('channel','result')
.every(1m)
.period(5m)
.align()
|default()
.field('value', 0)
|log()
.prefix('[DEBUG]-2')
.level('DEBUG')

data_ok_5min
|join(data_pending_5min)
.as('data_pending_5min', 'data_ok_5min')
|eval(lambda: "data_pending_5min.value",
lambda: "data_ok_5min.value")
.as('newvaluepending',
'newvalueok')
.keep()
|log()
.prefix('[DEBUG]-3')
.level('DEBUG')
```

Find bellow the kapacitor show:

```
var data_pending_5min = batch
|query('''select last(value) as value from "metrics_business"."autogen"."reg" where metric='last5min' and result='Pending' ''')
.groupBy('channel', 'result')
.every(1m)
.period(5m)
.align()
|default()
.field('value', 0)
|log()
.prefix('[DEBUG]-1')
.level('DEBUG')

var data_ok_5min = batch
|query('''select last(value) as value from "metrics_business"."autogen"."reg" where metric='last5min' and result='OK' ''')
.groupBy('channel', 'result')
.every(1m)
.period(5m)
.align()
|default()
.field('value', 0)
|log()
.prefix('[DEBUG]-2')
.level('DEBUG')

data_ok_5min
|join(data_pending_5min)
.as('data_pending_5min', 'data_ok_5min')
|eval(
lambda: "data_pending_5min.value",
lambda: "data_ok_5min.value"
)
.as(
'newvaluepending',
'newvalueok'
)
.keep()
|log()
.prefix('[DEBUG]-3')
.level('DEBUG')

DOT:
digraph alerting_reg_batch {
graph [throughput="0.00 batches/s"];

query4 [avg_exec_time_ns="0s" batches_queried="2" points_queried="2" query_errors="0" ];
query4 -> default5 [processed="2"];

default5 [avg_exec_time_ns="0s" fields_defaulted="0" tags_defaulted="0" ];
default5 -> log6 [processed="2"];

log6 [avg_exec_time_ns="0s" ];
log6 -> join8 [processed="2"];

query1 [avg_exec_time_ns="0s" batches_queried="2" points_queried="2" query_errors="0" ];
query1 -> default2 [processed="2"];

default2 [avg_exec_time_ns="0s" fields_defaulted="0" tags_defaulted="0" ];
default2 -> log3 [processed="2"];

log3 [avg_exec_time_ns="0s" ];
log3 -> join8 [processed="2"];

join8 [avg_exec_time_ns="0s" ];
join8 -> eval9 [processed="0"];

eval9 [avg_exec_time_ns="0s" eval_errors="0" ];
eval9 -> log10 [processed="0"];

log10 [avg_exec_time_ns="0s" ];
}
```

Log from kapacitor :

```
[alerting_reg_batch:query4] 2017/04/06 16:15:00 D! starting next batch query: SELECT last(value) AS value FROM metrics_business.autogen.reg WHERE metric = 'last5min' AND result = 'OK' AND time >= '2017-04-06T16:10:00Z' AND time < '2017-04-06T16:15:00Z' GROUP BY channel, result
[alerting_reg_batch:query1] 2017/04/06 16:15:00 D! starting next batch query: SELECT last(value) AS value FROM metrics_business.autogen.reg WHERE metric = 'last5min' AND result = 'Pending' AND time >= '2017-04-06T16:10:00Z' AND time < '2017-04-06T16:15:00Z' GROUP BY channel, result

[alerting_reg_batch:log3] 2017/04/06 16:15:00 D! [DEBUG]-1 {"name":"reg","tmax":"2017-04-06T16:15:00Z","group":"channel=channel1,result=Pending","tags":{"channel":"channel1","result":"Pending"},"points":[{"time":"2017-04-06T16:14:01.526306187Z","fields":{"value":16},"tags":{"channel":"channel1","result":"Pending"}}]}

[alerting_reg_batch:log6] 2017/04/06 16:15:00 D! [DEBUG]-2 {"name":"reg","tmax":"2017-04-06T16:15:00Z","group":"channel=channel1,result=OK","tags":{"channel":"channel1","result":"OK"},"points":[{"time":"2017-04-06T16:12:01.732661132Z","fields":{"value":2},"tags":{"channel":"channel1","result":"OK"}}]}

[alerting_reg_batch:log3] 2017/04/06 16:15:00 D! [DEBUG]-1 {"name":"reg","tmax":"2017-04-06T16:15:00Z","group":"channel=channel2,result=Pending","tags":{"channel":"channel2","result":"Pending"},"points":[{"time":"2017-04-06T16:14:01.526306187Z","fields":{"value":8},"tags":{"channel":"channel2","result":"Pending"}}]}

[alerting_reg_batch:log6] 2017/04/06 16:15:00 D! [DEBUG]-2 {"name":"reg","tmax":"2017-04-06T16:15:00Z","group":"channel=channel2,result=OK","tags":{"channel":"channel2","result":"OK"},"points":[{"time":"2017-04-06T16:14:01.526306187Z","fields":{"value":5},"tags":{"channel":"channel2","result":"OK"}}]}

```

Contributor guide

Open the contributing guide

Research direction

Start with the supplied batch tick, focusing on the join and eval nodes, then inspect the DOT output showing join8 processed zero batches. Reproduce the pipeline with the provided queries and logs; done means the joined batches reach eval and the newvaluepending and newvalueok fields appear in the final log output.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.