influxdata / influxdata/kapacitor

Kapacitor fails to start when subscription already exists

Open
#1,479 1 comment 5 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Relates to issue #679.
I have following configuration: all services run in AWS, 2 influxdb servers, kapacitor runs in a container ECS and had following env variables: `KAPACITOR_INFLUXDB_0_URLS_0=http://influx-server1:8086`, `KAPACITOR_INFLUXDB_0_URLS_1=http://influx-server2:8086`.
When one of the servers dies kapacitor stops sending alerts. Here's the error:
```
Jul 17 06:40:39 ip-172-16-71-104 docker/kapacitor[499]: [system.nodata:query1] 2017/07/17 04:40:39 E! Post http://influx-server1:8086/query?db=&q=SELECT+mean%28usage_user%29+AS+stat+FROM+telegraf.autogen.cpu+WHERE+host+%21~+%2F%28-jenkins-%7Cecs-apps%29%2F+AND+time+%3E%3D+%272017-07-17T04%3A38%3A39.572177535Z%27+AND+time+%3C+%272017-07-17T04%3A40%3A39.572177535Z%27+GROUP+BY+time%281m%2C+0s%29%2C+host: dial tcp 172.16.71.160:8086: getsockopt: connection refused
```
Probably it connects to the first server only and ignores the second. Not sure how container reads configuration cause i didn't find any documentation on environment variables interpretation.

So i switched influxdb to be behind ALB and configured `KAPACITOR_INFLUXDB_0_URLS_0` to ALB dns name, but kapacitor refused to start.

```
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: 2017/07/17 09:13:45 Using configuration at: /etc/kapacitor/kapacitor.conf
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [run] 2017/07/17 09:13:45 I! Kapacitor starting, version 1.3.1, branch master, commit 3b5512f7276483326577907803167e4bb213c613
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [run] 2017/07/17 09:13:45 I! Go version go1.7.5
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [srv] 2017/07/17 09:13:45 I! Kapacitor hostname: kapacitor.example.com
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [srv] 2017/07/17 09:13:45 I! ClusterID: 77f9dd12-a1e4-482e-90ea-7e57619e7091 ServerID: 06b5e43b-2d14-4752-ac7e-cde685de57e5
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [task_master:main] 2017/07/17 09:13:45 I! opened
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [httpd] 2017/07/17 09:13:45 I! Closed HTTP service
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [httpd] 2017/07/17 09:13:45 I! Closed HTTP service
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [task_master:main] 2017/07/17 09:13:45 I! closed
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [run] 2017/07/17 09:13:45 E! open server: open service *influxdb.Service: failed to link subscription on startup: creating sub kapacitor-77f9dd12-a1e4-482e-90ea-7e57619e7091 for db "telegraf" and rp "autogen": subscription already exists
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: run: open server: open service *influxdb.Service: failed to link subscription on startup: creating sub kapacitor-77f9dd12-a1e4-482e-90ea-7e57619e7091 for db "telegraf" and rp "autogen": subscription already exists
```

Please advise if that's an issue with my configuration or kapacitor can't work with load balanced influxdb or that's kapacitor's bug.

Contributor guide

Open the contributing guide

Research direction

Start with /etc/kapacitor/kapacitor.conf and the KAPACITOR_INFLUXDB_0_URLS_* environment variables, then reproduce the subscription-startup failure against the two InfluxDB endpoints and the ALB. Check whether failover and restart with an existing subscription behave as reported; done means Kapacitor starts without the duplicate-subscription error and continues sending alerts when one endpoint is unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker
Domain
backend, databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.