influxdata / influxdata/kapacitor
changeDetect node problem with RegEx and/or Retention Policies in batch
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Hi there, this is my first bug reporting, so sorry if I'm not clear enough (especially with the bug title).
I'm using:
CentOS 7 x86_64
Kapacitor OSS v. 1.5.0
InfluxDB OSS v. 1.5.2
I have different measurements (10k) all of them with the same schema: Four fields and one tag named "level" with 2 values "ALL_SAMPLES" and "ONLY_CHANGES"
I store all data recived from Logstash with the ALL_SAMPLES level, and once a day I want to generate the ONLY_CHANGES level with changeDetect node in all my measurements at the same time.
My first TICKscript doesn't work:
```
dbrp "db_name"."rp_1_name"
batch
|query(''' SELECT value, valid, "limit", type FROM "db_name"."rp_1_name"./.*/ WHERE "level"='ALL_SAMPLES' ''')
.offset(1h)
.period(1d)
.every(1d)
.groupByMeasurement()
|changeDetect('value')
|influxDBOut()
.database('db_name')
.retentionPolicy('rp_1_name')
.tag('level', 'ONLY_CHANGES')
```
I thought it could be a problem with the regular expresion/backreferencing in the measurement (the docs doesn't talk about it in Kapacitor, but I found this topic in InfluxData Community: https://community.influxdata.com/t/backreferencing-regex-in-kapacitor-tick-scripts-for-continuous-queries/4750 ) with the changeDetect node. But I'd been trying different things, and if I create a new retention policy named "rp_2_name" and I store the ONLY_CHANGES level on it, the TICKscript works fine:
```
dbrp "db_name"."rp_1_name"
batch
|query(''' SELECT value, valid, "limit", type FROM "db_name"."rp_1_name"./.*/ WHERE "level"='ALL_SAMPLES' ''')
.offset(1h)
.period(1d)
.every(1d)
.groupByMeasurement()
|changeDetect('value')
|influxDBOut()
.database('db_name')
.retentionPolicy('rp_2_name')
.tag('level', 'ONLY_CHANGES')
```
I've create a equivalent to this TICKscript with the stream node and I don't need a second retention policy:
```
dbrp "db_name"."rp_name"
stream
|from()
.database('db_name')
.groupByMeasurement()
.where(lambda: "level" == 'ALL_SAMPLES')
|changeDetect('value')
|influxDBOut()
.database('db_name')
.retentionPolicy('rp_name')
.tag('level', 'ONLY_CHANGES')
```
I'm not sure if this is a bug or an intentional behavior (or perhaps I have something wrong).
Have a nice day! :)
Ilyasbel
Contributor guide
Research direction
Reproduce both batch TICKscripts, comparing the regular-expression query, groupByMeasurement, changeDetect, and influxDBOut behavior when writing to the same or a different retention policy. Start by tracing the batch and changeDetect entry points and contrast them with the working stream pipeline. Done means the retention-policy behavior is explained and the reported batch case is corrected or documented with a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100