influxdata / influxdata/kapacitor
kapacitor UDF getting stuck on batch tasks
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Hi, we create several UDFs on several kapacitor servers and we encountered some issue repeating for different UDFs on different kapacitors.
For example we created some UDF with kapacitor task (they both described in the topic we created in influxdata community: https://community.influxdata.com/t/unexpected-eof-error-when-running-a-simple-udf-in-kapacitor/5456)
when the UDF sends the generated points back to the kapacitor ( self._agent.write_response(point_response) )
it seems it gets stuck.
After some debugging I see that the UDF is stuck in the self._out.write(data) in write_response.
It also seems that for some reason at some stage the kapacitor close the PIPE and stop reading the incoming points and that's why the self._out.write(data) is blocked
We still did not figured why it happends but when I open the PIPE again for reading (using cat /proc/kapacitor_pid/fd/<pytyon_stdout_file_descriptor>
I see the remaining points are printed and the UDF is again unblocked.
The question is why is it getting blocked and why the points are not read anymore..
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the UDF entry points self._agent.write_response(point_response) and self._out.write(data), then trace the Kapacitor pipe reader for batch tasks. Use the linked community discussion and reproduce the reported blocking behavior. Done means generated points are consumed and the UDF and task do not remain blocked.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, python
- Domain
- backend, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100