influxdata / influxdata/kapacitor

UDF is shutdown immediately after measurements are imported

Open
#1,411 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Hi,
I am testing the UDF [mirror.py](https://github.com/influxdata/kapacitor/blob/master/udf/agent/examples/mirror/mirror.py) function. When I import a file with about 2000 measurements in influxdb (influx -import -path=$file), the UDF is stopped after processing about 1000 measurements:
`[task_store] 2017/05/30 16:17:36 D! task normal_days finished
[task_master:main] 2017/05/30 16:17:36 E! Stopped task: normal_days mirror5: stopping UDF server: node aborted
[task_store] 2017/05/30 16:17:36 E! task normal_days finished with error: mirror5: stopping UDF server: node aborted
`

The error from mirror.py UDF:
`2017-05-30 16:17:35,550 INFO:root: Starting Agent for connection 3
Traceback (most recent call last):
File "/Users/.../code/udf/kapacitor/udf/agent.py", line 122, in _read_loop
self.handler.point(request.point)
File "/Users/.../code/udf/mirror.py", line 46, in point
self._agent.write_response(response, True)
File "/Users/.../code/udf/kapacitor/udf/agent.py", line 86, in write_response
self._out.flush()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 307, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
2017-05-30 16:17:36,216 ERROR:root: error processing request of type point: [Errno 32] Broken pipe
Exception in thread Thread-5:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/Users/.../code/udf/mirror.py", line 61, in accept
a.wait()
File "/Users/.../code/udf/kapacitor/udf/agent.py", line 68, in wait
self._in.close()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 283, in close
self.flush()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 307, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
`

For me it seems like Kapacitor is stopping the UDF immediately after all measurements are inserted into InfluxDB but not yet processed by the UDF (but not fully sure about this, just an assumption).
The "node aborted" error message is located [here](https://github.com/influxdata/kapacitor/blob/master/udf.go#L55)

macOS 10.12.5
both installed with homebrew:
Kapacitor v1.3.0 (git: unknown 7e27fd070e6a629a9b760c6300fade41364f4c7f)
InfluxDB vv1.2.4 (git: master 77909d7c7826afe597b12d957996d6e16cd1afaa)

Thanks in advance,
Simon

Contributor guide

Open the contributing guide

Research direction

Start with udf.go around the "node aborted" error, then trace the UDF lifecycle through udf/agent.py, especially write_response and wait. Reproduce the import using udf/agent/examples/mirror/mirror.py and determine why processing stops after roughly half the measurements; done means the UDF remains available until the imported measurements are processed.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.