exadel-inc / exadel-inc/CompreFace
DAMN ! worker 1 (pid: 38) died, killed by signal 9 :( trying respawn ...
- Dominant language
- Java
- Stars
- 8.3k
- Forks
- 1.2k
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
Trying to run Mobilenet-gpu - I upload test image and then it fails in logs with "DAMN ! worker 1 (pid: 38) died, killed by signal 9 :( trying respawn ..." Compreface freezes and reboots, endless loop when i try again.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
GPU: NVIDIA GeForce RTX 3050
GPU Memory: 4.0
CPU: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GhZ
Ram: 8.0GB
**NVIDIA-SMI 510.73.05 Driver Version: 516.93 CUDA Version: 11.7 |**
**Additional context**
Add any other context about the problem here.
`compreface-ui | 172.18.0.1 - - [30/Jul/2022:03:56:43 +0000] "GET /admin/user/demo/model HTTP/1.1" 200 52 "http://localhost:8000/login?redirect=%2F" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
compreface-core | [03:56:49] ../src/nnvm/legacy_json_util.cc:208: Loading symbol saved by previous version v1.2.0. Attempting to upgrade...
compreface-core | [03:56:49] ../src/nnvm/legacy_json_util.cc:216: Symbol successfully upgraded!
compreface-core | [03:56:49] ../src/engine/engine.cc:54: MXNet start using engine: ThreadedEnginePerDevice
compreface-core | [03:56:49] ../src/base.cc:79: cuDNN lib mismatch: linked-against version 8101 != compiled-against version 8100. Set MXNET_CUDNN_LIB_CHECKING=0 to quiet this warning.
compreface-core | DAMN ! worker 1 (pid: 38) died, killed by signal 9 :( trying respawn ...
compreface-core | Respawned uWSGI worker 1 (new pid: 59)
compreface-core | [03:57:06] ../src/nnvm/legacy_json_util.cc:208: Loading symbol saved by previous version v1.2.0. Attempting to upgrade...
compreface-core | [03:57:06] ../src/nnvm/legacy_json_util.cc:216: Symbol successfully upgraded!
compreface-core | [03:57:06] ../src/engine/engine.cc:54: MXNet start using engine: ThreadedEnginePerDevice
compreface-core | [03:57:06] ../src/base.cc:79: cuDNN lib mismatch: linked-against version 8101 != compiled-against version 8100. Set MXNET_CUDNN_LIB_CHECKING=0 to quiet this warning.
`
``
[Logs.txt](https://github.com/exadel-inc/CompreFace/files/9224767/Logs.txt)
Contributor guide
Research direction
Start with Logs.txt and the compreface-core output from a Mobilenet-gpu image upload. Investigate the signal 9 worker death alongside the reported cuDNN version mismatch and determine whether the GPU configuration triggers the respawn loop. Done means the cause is identified and the upload no longer freezes or repeatedly respawns the worker.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend, computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100