Azure / Azure/azure-functions-host

Python worker channel restarted for host while using tensorflow

Open
#4,479 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 10h
Merged PRs (30d)
36

Description

I am attempting to expose Tensorflow Object Detection API using a python Azure Function running on Azure Kubernetes Service (See [gist](https://gist.github.com/SpicySyntax/7244117ba3f1eb19e32bdc13ca3da584)) The python functions receive a queue message and then perform inference on an image. This is successful about half the time. The other half log the error message in the logs of the attached image: 'Restarting worker channel for runtime:python'

![image](https://user-images.githubusercontent.com/19228996/58120562-2290c000-7bcb-11e9-929b-631ebb4910dc.png)

#### Investigative information

Please provide the following:

- Timestamp: 05/21/2019 1:08 PM Central time
- Function App name: ObjectDetectionSegmentation
- Function name(s) (as appropriate): segment
- Core Tools version: 2.4.498

#### Repro steps

Provide the steps required to reproduce the problem:
Queue a few inference requests and wait for one to restart the python worker channel
#### Expected behavior

Provide a description of the expected behavior.
The function runs and returns the results from tensorflow

#### Actual behavior

Provide a description of the actual behavior observed.
The python worker channel intermittently restarts

#### Known workarounds

Provide a description of any known workarounds.
NA

#### Related information

Provide any related information

* (See [gist](https://gist.github.com/SpicySyntax/7244117ba3f1eb19e32bdc13ca3da584))
* Azure Storage Queue Bindings used

Contributor guide

Open the contributing guide

Research direction

Start with the linked gist and the reported 'Restarting worker channel for runtime:python' log while reviewing the queue-triggered inference flow. Reproduce with several inference requests on the stated Azure Kubernetes Service setup and inspect the worker and host logs. Done means the intermittent restart cause is identified and the function reliably returns TensorFlow results.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, kubernetes, python, tensorflow
Domain
backend, cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.