exadel-inc / exadel-inc/CompreFace

DAMN ! worker 1 (pid: 38) died, killed by signal 9 :( trying respawn ...

Open
#854 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
8.3k
Forks
1.2k
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**

Trying to run Mobilenet-gpu - I upload test image and then it fails in logs with "DAMN ! worker 1 (pid: 38) died, killed by signal 9 :( trying respawn ..." Compreface freezes and reboots, endless loop when i try again.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
GPU: NVIDIA GeForce RTX 3050
GPU Memory: 4.0
CPU: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GhZ
Ram: 8.0GB

**NVIDIA-SMI 510.73.05 Driver Version: 516.93 CUDA Version: 11.7 |**

**Additional context**
Add any other context about the problem here.

`compreface-ui | 172.18.0.1 - - [30/Jul/2022:03:56:43 +0000] "GET /admin/user/demo/model HTTP/1.1" 200 52 "http://localhost:8000/login?redirect=%2F" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
compreface-core | [03:56:49] ../src/nnvm/legacy_json_util.cc:208: Loading symbol saved by previous version v1.2.0. Attempting to upgrade...
compreface-core | [03:56:49] ../src/nnvm/legacy_json_util.cc:216: Symbol successfully upgraded!
compreface-core | [03:56:49] ../src/engine/engine.cc:54: MXNet start using engine: ThreadedEnginePerDevice
compreface-core | [03:56:49] ../src/base.cc:79: cuDNN lib mismatch: linked-against version 8101 != compiled-against version 8100.  Set MXNET_CUDNN_LIB_CHECKING=0 to quiet this warning.
compreface-core | DAMN ! worker 1 (pid: 38) died, killed by signal 9 :( trying respawn ...
compreface-core | Respawned uWSGI worker 1 (new pid: 59)
compreface-core | [03:57:06] ../src/nnvm/legacy_json_util.cc:208: Loading symbol saved by previous version v1.2.0. Attempting to upgrade...
compreface-core | [03:57:06] ../src/nnvm/legacy_json_util.cc:216: Symbol successfully upgraded!
compreface-core | [03:57:06] ../src/engine/engine.cc:54: MXNet start using engine: ThreadedEnginePerDevice
compreface-core | [03:57:06] ../src/base.cc:79: cuDNN lib mismatch: linked-against version 8101 != compiled-against version 8100.  Set MXNET_CUDNN_LIB_CHECKING=0 to quiet this warning.
`
``
[Logs.txt](https://github.com/exadel-inc/CompreFace/files/9224767/Logs.txt)

Contributor guide

Open the contributing guide

Research direction

Start with Logs.txt and the compreface-core output from a Mobilenet-gpu image upload. Investigate the signal 9 worker death alongside the reported cuDNN version mismatch and determine whether the GPU configuration triggers the respawn loop. Done means the cause is identified and the upload no longer freezes or repeatedly respawns the worker.

Written by the indexing model from the issue text.

Assessment

Domain
backend, computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.