influxdata / influxdata/kapacitor

changeDetect node problem with RegEx and/or Retention Policies in batch

Open
#1,976 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Hi there, this is my first bug reporting, so sorry if I'm not clear enough (especially with the bug title).

I'm using:
CentOS 7 x86_64
Kapacitor OSS v. 1.5.0
InfluxDB OSS v. 1.5.2

I have different measurements (10k) all of them with the same schema: Four fields and one tag named "level" with 2 values "ALL_SAMPLES" and "ONLY_CHANGES"
I store all data recived from Logstash with the ALL_SAMPLES level, and once a day I want to generate the ONLY_CHANGES level with changeDetect node in all my measurements at the same time.

My first TICKscript doesn't work:

```
dbrp "db_name"."rp_1_name"
batch
|query(''' SELECT value, valid, "limit", type FROM "db_name"."rp_1_name"./.*/ WHERE "level"='ALL_SAMPLES' ''')
.offset(1h)
.period(1d)
.every(1d)
.groupByMeasurement()
|changeDetect('value')
|influxDBOut()
.database('db_name')
.retentionPolicy('rp_1_name')
.tag('level', 'ONLY_CHANGES')
```

I thought it could be a problem with the regular expresion/backreferencing in the measurement (the docs doesn't talk about it in Kapacitor, but I found this topic in InfluxData Community: https://community.influxdata.com/t/backreferencing-regex-in-kapacitor-tick-scripts-for-continuous-queries/4750 ) with the changeDetect node. But I'd been trying different things, and if I create a new retention policy named "rp_2_name" and I store the ONLY_CHANGES level on it, the TICKscript works fine:

```
dbrp "db_name"."rp_1_name"
batch
|query(''' SELECT value, valid, "limit", type FROM "db_name"."rp_1_name"./.*/ WHERE "level"='ALL_SAMPLES' ''')
.offset(1h)
.period(1d)
.every(1d)
.groupByMeasurement()
|changeDetect('value')
|influxDBOut()
.database('db_name')
.retentionPolicy('rp_2_name')
.tag('level', 'ONLY_CHANGES')
```

I've create a equivalent to this TICKscript with the stream node and I don't need a second retention policy:

```
dbrp "db_name"."rp_name"
stream
|from()
.database('db_name')
.groupByMeasurement()
.where(lambda: "level" == 'ALL_SAMPLES')
|changeDetect('value')
|influxDBOut()
.database('db_name')
.retentionPolicy('rp_name')
.tag('level', 'ONLY_CHANGES')
```

I'm not sure if this is a bug or an intentional behavior (or perhaps I have something wrong).

Have a nice day! :)

Ilyasbel

Contributor guide

Open the contributing guide

Research direction

Reproduce both batch TICKscripts, comparing the regular-expression query, groupByMeasurement, changeDetect, and influxDBOut behavior when writing to the same or a different retention policy. Start by tracing the batch and changeDetect entry points and contrast them with the working stream pipeline. Done means the retention-policy behavior is explained and the reported batch case is corrected or documented with a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.