influxdata / influxdata/kapacitor

Recording hangs and records nothing if data to task stops before recording end time

Open
#1,122 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

My use case is to use recording and replay for automated testing. I can't be the only one to think of this. We are using kapacitor to raise alerts given certain `TICK` logic and a given stream of input data events. Ideally we would like to:

- start recording
- send in a fixed deteministic set of event data at a kapacitor script
- stop recording
- validate alerts were raised

However, if I follow these steps, which involve bracketing the sending of the fixed set of event data with the start and stop of recording, the recording always hangs and records a 0-byte file. (This is reported both from the kap CLI and in data file in `/var/lib/kapacitor/replay`.) What I discovered is that if I force the event data to flow to a time past the end time of the recording, the recording always succeeds.

Some more detail and a small example ...

- I am sending event data to a local kapacitor instance via UDP to 9100
- I have confirmed events are arriving with `tcpdump` on the port, that they are being ingested into `kapacitor` with `stats ingress`, and that they are being processed by the task with `show task`
- Further, I can verify that alerts are being generated (the last step in the `TICK` script), because they are being sent by `post()` to a designated port and I'm watching that traffic as well.
- In other words, yes, I have verified the round trip of the data flow; that is not the issue.

What I have observed, every time, is that running a `recording` from either the CLI directly in a separate shell session or from within a Python application using `subprocess`, is that recording will hang if the event stream stops before the recording end time. When I eventually keyboard interrupt to kill the process, there is a 0-byte recording created.

Examples:

**Failure**
- Events are being sent to a task that collects data on a 1-second period
- The task needs to see 5 events over 5 periods matching some criteria to generate an alert
- The expected behavior is verified, per above
- One set of 7 events over 7 seconds is sent to kapacitor, once per second. (I always send 2 extra to avoid window boundary issues affecting the test)
- Recording is started immediately before generating events and set to run for 9 seconds
- Recording hangs and records nothing

**Success**
- Events are being sent to a task that collects data on a 1-second period
- The task needs to see 5 events over 5 periods matching some criteria to generate an alert
- The expected behavior is verified, per above
- **Two** sets of 7 events over 7 seconds are sent to kapacitor, once per second
- Recording is started immediately before generating events and set to run for 9 (or anything less than 14) seconds
- Recording succeeds

Thanks for looking into this.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the recording command, a 9-second window, and one 7-event stream, then compare it with the successful two-stream case. Start by tracing the recording and replay flow associated with /var/lib/kapacitor/replay and the kap CLI; use stats ingress, show task, and the UDP 9100 traffic checks to confirm input and task processing. Done means the short stream produces a non-empty recording without hanging.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.