influxdata / influxdata/kapacitor

kapacitor UDF getting stuck on batch tasks

Open
#1,964 13 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Hi, we create several UDFs on several kapacitor servers and we encountered some issue repeating for different UDFs on different kapacitors.

For example we created some UDF with kapacitor task (they both described in the topic we created in influxdata community: https://community.influxdata.com/t/unexpected-eof-error-when-running-a-simple-udf-in-kapacitor/5456)

when the UDF sends the generated points back to the kapacitor ( self._agent.write_response(point_response) )
it seems it gets stuck.
After some debugging I see that the UDF is stuck in the self._out.write(data) in write_response.

It also seems that for some reason at some stage the kapacitor close the PIPE and stop reading the incoming points and that's why the self._out.write(data) is blocked
We still did not figured why it happends but when I open the PIPE again for reading (using cat /proc/kapacitor_pid/fd/<pytyon_stdout_file_descriptor>
I see the remaining points are printed and the UDF is again unblocked.

The question is why is it getting blocked and why the points are not read anymore..

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the UDF entry points self._agent.write_response(point_response) and self._out.write(data), then trace the Kapacitor pipe reader for batch tasks. Use the linked community discussion and reproduce the reported blocking behavior. Done means generated points are consumed and the UDF and task do not remain blocked.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, python
Domain
backend, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.