livepeer / livepeer/go-livepeer
CUDA device mapping TensorFlow error when loading DNN model
Open
Nobody has claimed this yet.
type: bug
- Dominant language
- Go
- Stars
- 586
- Forks
- 226
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 19
Description
E tensorflow/core/common_runtime/session.cc:91] Failed to create session: Already exists: TensorFlow device (GPU:0) is being mapped to multiple CUDA devices (1 now, and 0 previously), which is not supported. This may be the result of providing different GPU configurations (ConfigProto.gpu_options, for example different visible_device_list) when creating multiple Sessions in the same process. This is not currently supported, see https://github.com/tensorflow/tensorflow/issues/19083
E tensorflow/c/c_api.cc:2184] Already exists: TensorFlow device (GPU:0) is being mapped to multiple CUDA devices (1 now, and 0 previously), which is not supported. This may be the result of providing different GPU configurations (ConfigProto.gpu_options, for example different visible_device_list) when creating multiple Sessions in the same process. This is not currently supported, see https://github.com/tensorflow/tensorflow/issues/19083
[dnn_tensorflow @ 0x7f57a4052500] Failed to create new session with model graph
[dnn_tensorflow @ 0x7f57a4052500] Failed to load native model
[livepeer_dnn @ 0x7f57a4058400] could not load DNN model
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator fatal error: unexpected signal during runtime execution
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x476085]
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator runtime stack:
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator runtime.throw(0x1f1d68a, 0x2a)
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator /usr/local/go/src/runtime/panic.go:1116 +0x72
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator runtime.sigpanic()
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator /usr/local/go/src/runtime/signal_unix.go:726 +0x4ac
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator goroutine 19245 [syscall]:
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator runtime.cgocall(0x11b4810, 0xc00091d810, 0x1d69e20)
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator /usr/local/go/src/runtime/cgocall.go:133 +0x5b fp=0xc00091d7e0 sp=0xc00091d7a8 pc=0x4a64bb
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator github.com/livepeer/lpms/ffmpeg._Cfunc_lpms_transcode_new_with_dnn(0xc000a4f240, 0x0)
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator _cgo_gotypes.go:370 +0x4a fp=0xc00091d810 sp=0xc00091d7e0 pc=0xd14b6a
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator github.com/livepeer/lpms/ffmpeg.NewTranscoderWithDetector.func5(0xc000a4f240, 0xc0007ad480)
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator /go/pkg/mod/github.com/livepeer/lpms@v0.0.0-20210806125031-9fdbf80c8575/ffmpeg/ffmpeg.go:541 +0x4d fp=0xc00091d840 sp=0xc00091d810 pc=0xd1a2ed
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator github.com/livepeer/lpms/ffmpeg.NewTranscoderWithDetector(0x211dfc0, 0xc00012d720, 0x7ffda59a3db6, 0x1, 0x0, 0x0, 0x0)
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator /go/pkg/mod/github.com/livepeer/lpms@v0.0.0-20210806125031-9fdbf80c8575/ffmpeg/ffmpeg.go:541 +0x245 fp=0xc00091d8e0 sp=0xc00091d840 pc=0xd17d25
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator github.com/livepeer/go-livepeer/core.NewNvidiaTranscoderWithDetector(0x211dfc0, 0xc00012d720, 0x7ffda59a3db6, 0x1, 0x1, 0x7ffda59a3db6, 0x1, 0xc000aae7b0)
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator /build/core/transcoder.go:149 +0x4d fp=0xc00091d940 sp=0xc00091d8e0 pc=0x1103d6d
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator github.com/livepeer/go-livepeer/core.(*LoadBalancingTranscoder).createSession(0xc0002ca000, 0xc00060d860, 0x0, 0x0, 0x0)
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator /build/core/lb.go:95 +0x389 fp=0xc00091da78 sp=0xc00091d940 pc=0x10f5789
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator github.com/livepeer/go-livepeer/core.(*LoadBalancingTranscoder).Transcode(0xc0002ca000, 0xc00060d860, 0x2e89680, 0x16ee98, 0x1ffe00)
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator /build/core/lb.go:67 +0xd4 fp=0xc00091dae8 sp=0xc00091da78 pc=0x10f5254
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator github.com/livepeer/go-livepeer/core.(*LivepeerNode).transcodeSeg(0xc0005cc420, 0x214a480, 0xc000e704c0, 0x214a480, 0xc000e704c0, 0xc000e70480, 0xc00060d860, 0x781ba5)
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator /build/core/orchestrator.go:550 +0x3dc fp=0xc00091de80 sp=0xc00091dae8 pc=0x10fb8bc
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator github.com/livepeer/go-livepeer/core.(*LivepeerNode).transcodeSegmentLoop.func1(0x214a480, 0xc000e704c0, 0x214a480, 0xc000e704c0, 0xc0005cc420, 0xc00060d860, 0xc013210600, 0x214a480, 0xc000e704c0,
This occurred when using the branch for https://github.com/livepeer/go-livepeer/pull/1979.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the branch for pull request #1979 with the failure path through core/transcoder.go:149 and core/lb.go:95. Trace ffmpeg.NewTranscoderWithDetector and the TensorFlow session creation error, then reproduce the model-loading failure and confirm that transcoding no longer crashes when the device mapping is encountered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, tensorflow
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100