influxdata / influxdata/kapacitor
Very confusing behavior when flushInterval is set to 0
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
I think I misunderstood what flushInterval was supposed to do in a TICK script for a UDF in Kapacitor. My goal was to get processed results of raw events to propagate as quickly as possible to the output database. I naively thought I could accomplish this by setting the `flushInterval` to 0, which would basically be an unbuffered dump from the UDF.
I did that, and some time later (after Kapacitor was restarted -- I'm not sure of the exact sequence of events that led to this error hitting), Kapacitor started dying. It wasn't until I ran `kapacitord` directly from the shell where I found this output:
```
panic(0x1c834e0, 0xc420772010)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
time.NewTicker(0x0, 0x20ea170)
/usr/local/go/src/time/tick.go:23 +0x18d
github.com/influxdata/kapacitor.(*writeBuffer).run(0xc4201eab60)
/root/go/src/github.com/influxdata/kapacitor/influxdb_out.go:227 +0x9c
created by github.com/influxdata/kapacitor.(*writeBuffer).start
/root/go/src/github.com/influxdata/kapacitor/influxdb_out.go:212 +0x5c
```
In version 1.3.3 of Kapacitor, that corresponds to these calls:
```go
func (w *writeBuffer) start() {
w.wg.Add(1)
go w.run()
}
```
calling
```go
func (w *writeBuffer) run() {
defer w.wg.Done()
flushTick := time.NewTicker(w.flushInterval)
defer flushTick.Stop()
var err error
```
Obviously if `flushInterval` is 0, `time.NewTicker` panics, which isn't protected here. If I changed my tick script to set the flushInterval to 1s (or any duration greater than 0), `kapacitord` failed with the same error. I tried running `kapacitor define ` to refresh the task to no avail.
Ultimately, what I had to do was delete the task and redefine it (without changing the tick script). But even that posed challenges, since I couldn't delete the existing task without starting kapacitor, but I couldn't start kapacitor without that task deleted. The only way around that was to disable the influxdb binding in kapacitor.conf, start kapacitor, disable/delete the task, re-enable influxdb in kapacitor.conf, and restart kapacitor.
Then I was able to redefine the stream UDF with a non-zero flushInterval and it started working again.
Whew! Is there anything that stands out that I'm just doing very wrong? Is there a way to disable buffering of stream UDFs?
Thanks!
Contributor guide
Research direction
Start with influxdb_out.go, especially writeBuffer.start and writeBuffer.run, and reproduce the kapacitord restart failure using a TICK script with flushInterval set to 0, then changed to a positive value. Trace how the task configuration is loaded and persisted; done should define the supported buffering behavior and prevent this configuration from leaving Kapacitor unable to start.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100