influxdata / influxdata/kapacitor

panic: invalid timer state

Open
#1,705 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Hello, I've installed kapacitor from official repo and setup telegraf/influxdb/kapacitor stack. The kapacitor fails everyday :(

OS: CentOS release 6.9 (Final)
kapacitor.x86_64 1.3.3-1 @influxdb

# cat /var/log/kapacitor/kapacitord.err
2017/12/01 08:33:07 Using configuration at: /etc/kapacitor/kapacitor.conf
panic: invalid timer state

goroutine 98 [running]:
panic(0x1bb4de0, 0xc4230d3780)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/influxdata/kapacitor/timer.(*timer).Start(0xc42006eea0)
/root/go/src/github.com/influxdata/kapacitor/timer/timer.go:68 +0xfd
github.com/influxdata/kapacitor.(*InfluxQLNode).runBatchInfluxQL(0xc42046cc00, 0x1c72b20, 0xc42030a2b0)
/root/go/src/github.com/influxdata/kapacitor/influxql.go:176 +0x170
github.com/influxdata/kapacitor.(*InfluxQLNode).runInfluxQLs(0xc42046cc00, 0x0, 0x0, 0x0, 0xc42049cf58, 0xc42049cf68)
/root/go/src/github.com/influxdata/kapacitor/influxql.go:45 +0x4c
github.com/influxdata/kapacitor.(*InfluxQLNode).(github.com/influxdata/kapacitor.runInfluxQLs)-fm(0x0, 0x0, 0x0, 0xc42049cf80, 0x1)
/root/go/src/github.com/influxdata/kapacitor/influxql.go:36 +0x48
github.com/influxdata/kapacitor.(*node).start.func1(0xc42046cc00, 0x0, 0x0, 0x0)
/root/go/src/github.com/influxdata/kapacitor/node.go:140 +0x8e
created by github.com/influxdata/kapacitor.(*node).start
/root/go/src/github.com/influxdata/kapacitor/node.go:141 +0x5d

# tail -n 10 /var/log/kapacitor/kapacitor.log
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:00 +0000] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" f9677af1-d690-11e7-a93b-000000000000 984
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:01 +0000] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" fa1e4784-d690-11e7-a93c-000000000000 707
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:01 +0000] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" fa2e1e7c-d690-11e7-a93d-000000000000 872
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:02 +0000] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" fa527ca6-d690-11e7-a93e-000000000000 660
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:02 +0000] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" fa52b5ff-d690-11e7-a93f-000000000000 546
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:03 +0000] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" fafce855-d690-11e7-a940-000000000000 1122
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:10 +0000] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" ff01d44d-d690-11e7-a941-000000000000 890
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:10 +0000] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" ff2fb8f1-d690-11e7-a942-000000000000 580
[cpu_load:percentile5] 2017/12/01 12:13:10 E! failed to emit batch: edged aborted
[httpd] 127.0.0.1 - - [01/Dec/2017:12:13:10 +0000] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" ff5d729d-d690-11e7-a943-000000000000 969

Contributor guide

Open the contributing guide

Research direction

Start with timer/timer.go at timer.Start and trace the call from influxql.go, especially runBatchInfluxQL, using the supplied stack trace as the entry point. Reproduce the recurring failure if possible and determine the timer state that triggers the panic; done means the failure no longer occurs and the behavior is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.