influxdata / influxdata/kapacitor

Very confusing behavior when flushInterval is set to 0

Open
#1,781 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I think I misunderstood what flushInterval was supposed to do in a TICK script for a UDF in Kapacitor. My goal was to get processed results of raw events to propagate as quickly as possible to the output database. I naively thought I could accomplish this by setting the `flushInterval` to 0, which would basically be an unbuffered dump from the UDF.

I did that, and some time later (after Kapacitor was restarted -- I'm not sure of the exact sequence of events that led to this error hitting), Kapacitor started dying. It wasn't until I ran `kapacitord` directly from the shell where I found this output:

```
panic(0x1c834e0, 0xc420772010)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
time.NewTicker(0x0, 0x20ea170)
/usr/local/go/src/time/tick.go:23 +0x18d
github.com/influxdata/kapacitor.(*writeBuffer).run(0xc4201eab60)
/root/go/src/github.com/influxdata/kapacitor/influxdb_out.go:227 +0x9c
created by github.com/influxdata/kapacitor.(*writeBuffer).start
/root/go/src/github.com/influxdata/kapacitor/influxdb_out.go:212 +0x5c
```

In version 1.3.3 of Kapacitor, that corresponds to these calls:

```go
func (w *writeBuffer) start() {
w.wg.Add(1)
go w.run()
}
```

calling

```go
func (w *writeBuffer) run() {
defer w.wg.Done()
flushTick := time.NewTicker(w.flushInterval)
defer flushTick.Stop()
var err error
```

Obviously if `flushInterval` is 0, `time.NewTicker` panics, which isn't protected here. If I changed my tick script to set the flushInterval to 1s (or any duration greater than 0), `kapacitord` failed with the same error. I tried running `kapacitor define ` to refresh the task to no avail.

Ultimately, what I had to do was delete the task and redefine it (without changing the tick script). But even that posed challenges, since I couldn't delete the existing task without starting kapacitor, but I couldn't start kapacitor without that task deleted. The only way around that was to disable the influxdb binding in kapacitor.conf, start kapacitor, disable/delete the task, re-enable influxdb in kapacitor.conf, and restart kapacitor.

Then I was able to redefine the stream UDF with a non-zero flushInterval and it started working again.

Whew! Is there anything that stands out that I'm just doing very wrong? Is there a way to disable buffering of stream UDFs?

Thanks!

Contributor guide

Open the contributing guide

Research direction

Start with influxdb_out.go, especially writeBuffer.start and writeBuffer.run, and reproduce the kapacitord restart failure using a TICK script with flushInterval set to 0, then changed to a positive value. Trace how the task configuration is loaded and persisted; done should define the supported buffering behavior and prevent this configuration from leaving Kapacitor unable to start.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.