Azure / Azure/azure-functions-host
Python worker channel restarted for host while using tensorflow
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
I am attempting to expose Tensorflow Object Detection API using a python Azure Function running on Azure Kubernetes Service (See [gist](https://gist.github.com/SpicySyntax/7244117ba3f1eb19e32bdc13ca3da584)) The python functions receive a queue message and then perform inference on an image. This is successful about half the time. The other half log the error message in the logs of the attached image: 'Restarting worker channel for runtime:python'

#### Investigative information
Please provide the following:
- Timestamp: 05/21/2019 1:08 PM Central time
- Function App name: ObjectDetectionSegmentation
- Function name(s) (as appropriate): segment
- Core Tools version: 2.4.498
#### Repro steps
Provide the steps required to reproduce the problem:
Queue a few inference requests and wait for one to restart the python worker channel
#### Expected behavior
Provide a description of the expected behavior.
The function runs and returns the results from tensorflow
#### Actual behavior
Provide a description of the actual behavior observed.
The python worker channel intermittently restarts
#### Known workarounds
Provide a description of any known workarounds.
NA
#### Related information
Provide any related information
* (See [gist](https://gist.github.com/SpicySyntax/7244117ba3f1eb19e32bdc13ca3da584))
* Azure Storage Queue Bindings used
Contributor guide
Research direction
Start with the linked gist and the reported 'Restarting worker channel for runtime:python' log while reviewing the queue-triggered inference flow. Reproduce with several inference requests on the stated Azure Kubernetes Service setup and inspect the worker and host logs. Done means the intermittent restart cause is identified and the function reliably returns TensorFlow results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, kubernetes, python, tensorflow
- Domain
- backend, cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100