influxdata / influxdata/kapacitor
Kapacitor fails to start when subscription already exists
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Relates to issue #679.
I have following configuration: all services run in AWS, 2 influxdb servers, kapacitor runs in a container ECS and had following env variables: `KAPACITOR_INFLUXDB_0_URLS_0=http://influx-server1:8086`, `KAPACITOR_INFLUXDB_0_URLS_1=http://influx-server2:8086`.
When one of the servers dies kapacitor stops sending alerts. Here's the error:
```
Jul 17 06:40:39 ip-172-16-71-104 docker/kapacitor[499]: [system.nodata:query1] 2017/07/17 04:40:39 E! Post http://influx-server1:8086/query?db=&q=SELECT+mean%28usage_user%29+AS+stat+FROM+telegraf.autogen.cpu+WHERE+host+%21~+%2F%28-jenkins-%7Cecs-apps%29%2F+AND+time+%3E%3D+%272017-07-17T04%3A38%3A39.572177535Z%27+AND+time+%3C+%272017-07-17T04%3A40%3A39.572177535Z%27+GROUP+BY+time%281m%2C+0s%29%2C+host: dial tcp 172.16.71.160:8086: getsockopt: connection refused
```
Probably it connects to the first server only and ignores the second. Not sure how container reads configuration cause i didn't find any documentation on environment variables interpretation.
So i switched influxdb to be behind ALB and configured `KAPACITOR_INFLUXDB_0_URLS_0` to ALB dns name, but kapacitor refused to start.
```
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: 2017/07/17 09:13:45 Using configuration at: /etc/kapacitor/kapacitor.conf
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [run] 2017/07/17 09:13:45 I! Kapacitor starting, version 1.3.1, branch master, commit 3b5512f7276483326577907803167e4bb213c613
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [run] 2017/07/17 09:13:45 I! Go version go1.7.5
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [srv] 2017/07/17 09:13:45 I! Kapacitor hostname: kapacitor.example.com
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [srv] 2017/07/17 09:13:45 I! ClusterID: 77f9dd12-a1e4-482e-90ea-7e57619e7091 ServerID: 06b5e43b-2d14-4752-ac7e-cde685de57e5
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [task_master:main] 2017/07/17 09:13:45 I! opened
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [httpd] 2017/07/17 09:13:45 I! Closed HTTP service
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [httpd] 2017/07/17 09:13:45 I! Closed HTTP service
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [task_master:main] 2017/07/17 09:13:45 I! closed
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: [run] 2017/07/17 09:13:45 E! open server: open service *influxdb.Service: failed to link subscription on startup: creating sub kapacitor-77f9dd12-a1e4-482e-90ea-7e57619e7091 for db "telegraf" and rp "autogen": subscription already exists
Jul 17 11:13:45 ip-172-16-71-104 docker/kapacitor[499]: run: open server: open service *influxdb.Service: failed to link subscription on startup: creating sub kapacitor-77f9dd12-a1e4-482e-90ea-7e57619e7091 for db "telegraf" and rp "autogen": subscription already exists
```
Please advise if that's an issue with my configuration or kapacitor can't work with load balanced influxdb or that's kapacitor's bug.
Contributor guide
Research direction
Start with /etc/kapacitor/kapacitor.conf and the KAPACITOR_INFLUXDB_0_URLS_* environment variables, then reproduce the subscription-startup failure against the two InfluxDB endpoints and the ALB. Check whether failover and restart with an existing subscription behave as reported; done means Kapacitor starts without the duplicate-subscription error and continues sending alerts when one endpoint is unavailable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker
- Domain
- backend, databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100