influxdata / influxdata/kapacitor
panic: invalid timer state
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Hello, I've installed kapacitor from official repo and setup telegraf/influxdb/kapacitor stack. The kapacitor fails everyday :(
OS: CentOS release 6.9 (Final)
kapacitor.x86_64 1.3.3-1 @influxdb
# cat /var/log/kapacitor/kapacitord.err
2017/12/01 08:33:07 Using configuration at: /etc/kapacitor/kapacitor.conf
panic: invalid timer state
goroutine 98 [running]:
panic(0x1bb4de0, 0xc4230d3780)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/influxdata/kapacitor/timer.(*timer).Start(0xc42006eea0)
/root/go/src/github.com/influxdata/kapacitor/timer/timer.go:68 +0xfd
github.com/influxdata/kapacitor.(*InfluxQLNode).runBatchInfluxQL(0xc42046cc00, 0x1c72b20, 0xc42030a2b0)
/root/go/src/github.com/influxdata/kapacitor/influxql.go:176 +0x170
github.com/influxdata/kapacitor.(*InfluxQLNode).runInfluxQLs(0xc42046cc00, 0x0, 0x0, 0x0, 0xc42049cf58, 0xc42049cf68)
/root/go/src/github.com/influxdata/kapacitor/influxql.go:45 +0x4c
github.com/influxdata/kapacitor.(*InfluxQLNode).(github.com/influxdata/kapacitor.runInfluxQLs)-fm(0x0, 0x0, 0x0, 0xc42049cf80, 0x1)
/root/go/src/github.com/influxdata/kapacitor/influxql.go:36 +0x48
github.com/influxdata/kapacitor.(*node).start.func1(0xc42046cc00, 0x0, 0x0, 0x0)
/root/go/src/github.com/influxdata/kapacitor/node.go:140 +0x8e
created by github.com/influxdata/kapacitor.(*node).start
/root/go/src/github.com/influxdata/kapacitor/node.go:141 +0x5d
# tail -n 10 /var/log/kapacitor/kapacitor.log
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:00 +0000] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" f9677af1-d690-11e7-a93b-000000000000 984
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:01 +0000] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" fa1e4784-d690-11e7-a93c-000000000000 707
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:01 +0000] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" fa2e1e7c-d690-11e7-a93d-000000000000 872
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:02 +0000] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" fa527ca6-d690-11e7-a93e-000000000000 660
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:02 +0000] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" fa52b5ff-d690-11e7-a93f-000000000000 546
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:03 +0000] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" fafce855-d690-11e7-a940-000000000000 1122
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:10 +0000] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" ff01d44d-d690-11e7-a941-000000000000 890
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:10 +0000] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" ff2fb8f1-d690-11e7-a942-000000000000 580
[cpu_load:percentile5] 2017/12/01 12:13:10 E! failed to emit batch: edged aborted
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:10 +0000] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" ff5d729d-d690-11e7-a943-000000000000 969
Contributor guide
Research direction
Start with timer/timer.go at timer.Start and trace the call from influxql.go, especially runBatchInfluxQL, using the supplied stack trace as the entry point. Reproduce the recurring failure if possible and determine the timer state that triggers the panic; done means the failure no longer occurs and the behavior is covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100