influxdata / influxdata/kapacitor

ServerID is not accounted when consuming subscription

Open
#2,038 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

We run Kapacitor as a Docker container (`FROM kapacitor:1.5.1`). And as we see no need to persist any "runtime" data, we don't use any dedicated volume for it.

However, what we sometimes do is change Kapacitor configuration (`conf`, handlers, alert definitions), which is then followed by image re-build and container re-run, while Kapacitor URL/port remains the same.

We also have some subscriptions configured in our `kapacitor.conf`:
```
[[influxdb]]
...
[influxdb.subscriptions]
telegraf = ["autogen"]
...
```
Which causes Kapacitor to create a new subscription every time the server is initialized, if subscription is not there. And it uses the current Kapacitor instance' `ServerID` to name a subscription. So on our InfluxDB (`1.4.2`) server it then looks like:
```
> show subscriptions
name: telegraf
retention_policy name mode destinations
---------------- ---- ---- ------------
autogen kapacitor-c11618fc-5af5-4a1c-bb81-a66d19f0fa23 ANY [http://my.kapacitor.hostname:9092]
```

The problem is that the mentioned `ServerID` attribute changes every time the server is initialized (container is re-run), which makes it to create a new subscription as this time the name for a subscription would not have a match within the existing subscriptions. So we end up with a new subscription added, something like:
```
> show subscriptions
name: telegraf
retention_policy name mode destinations
---------------- ---- ---- ------------
autogen kapacitor-c11618fc-5af5-4a1c-bb81-a66d19f0fa23 ANY [http://my.kapacitor.hostname:9092]
autogen kapacitor-4399ee14-a4aa-4c11-9d55-1f3c99ef6223 ANY [http://my.kapacitor.hostname:9092]
```

That wouldn't be a problem, considering that Kapacitor would accept a subscription data only if URL matches and name matches (something I derived from https://github.com/influxdata/kapacitor/issues/349#issuecomment-197908046). But the problem is that with 2 existing subscriptions as above, I then receive a duplicated data when using a `stream()` node - each coming from the existing subscription. And if I drop one subscription, I stop getting the duplicated data. If I instead re-run the Kapacitor again it'll generate a new subscription and I'll be receiving the data thrice, and so on. So it looks like a subscription name (which includes `ServerID`) is not accounted when Kapacitor decides whether to accept a data or not.

What would be a suggested solution to run Kapacitor in such environment and scenario?

I can create a container pre-run script which wipes out not used subscriptions, but it looks more like a workaround. Alternatively, I can remove subscription section in `kapacitor.conf` and pre-create a subscription manually - but this is a workaround too. Even if Kapacitor would account the name (comprised of `ServerID`) of a subscription, I would still end up with tens of not used subscriptions accumulated over the time (considering we re-deploy the container often) - what would be a solution for Kapacitor then?

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or entry points are named in the report. Start by locating Kapacitor's subscription creation and subscription-data matching paths, then reproduce repeated container initialization against InfluxDB; done would require an agreed solution that prevents duplicate delivery and handles obsolete subscriptions.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.