influxdata / influxdata/kapacitor
Recording hangs and records nothing if data to task stops before recording end time
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
My use case is to use recording and replay for automated testing. I can't be the only one to think of this. We are using kapacitor to raise alerts given certain `TICK` logic and a given stream of input data events. Ideally we would like to:
- start recording
- send in a fixed deteministic set of event data at a kapacitor script
- stop recording
- validate alerts were raised
However, if I follow these steps, which involve bracketing the sending of the fixed set of event data with the start and stop of recording, the recording always hangs and records a 0-byte file. (This is reported both from the kap CLI and in data file in `/var/lib/kapacitor/replay`.) What I discovered is that if I force the event data to flow to a time past the end time of the recording, the recording always succeeds.
Some more detail and a small example ...
- I am sending event data to a local kapacitor instance via UDP to 9100
- I have confirmed events are arriving with `tcpdump` on the port, that they are being ingested into `kapacitor` with `stats ingress`, and that they are being processed by the task with `show task`
- Further, I can verify that alerts are being generated (the last step in the `TICK` script), because they are being sent by `post()` to a designated port and I'm watching that traffic as well.
- In other words, yes, I have verified the round trip of the data flow; that is not the issue.
What I have observed, every time, is that running a `recording` from either the CLI directly in a separate shell session or from within a Python application using `subprocess`, is that recording will hang if the event stream stops before the recording end time. When I eventually keyboard interrupt to kill the process, there is a 0-byte recording created.
Examples:
**Failure**
- Events are being sent to a task that collects data on a 1-second period
- The task needs to see 5 events over 5 periods matching some criteria to generate an alert
- The expected behavior is verified, per above
- One set of 7 events over 7 seconds is sent to kapacitor, once per second. (I always send 2 extra to avoid window boundary issues affecting the test)
- Recording is started immediately before generating events and set to run for 9 seconds
- Recording hangs and records nothing
**Success**
- Events are being sent to a task that collects data on a 1-second period
- The task needs to see 5 events over 5 periods matching some criteria to generate an alert
- The expected behavior is verified, per above
- **Two** sets of 7 events over 7 seconds are sent to kapacitor, once per second
- Recording is started immediately before generating events and set to run for 9 (or anything less than 14) seconds
- Recording succeeds
Thanks for looking into this.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the recording command, a 9-second window, and one 7-event stream, then compare it with the successful two-stream case. Start by tracing the recording and replay flow associated with /var/lib/kapacitor/replay and the kap CLI; use stats ingress, show task, and the UDP 9100 traffic checks to confirm input and task processing. Done means the short stream produces a non-empty recording without hanging.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100