livepeer / livepeer/go-livepeer

CUDA device mapping TensorFlow error when loading DNN model

Open
#1,980 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type: bug
Dominant language
Go
Stars
586
Forks
226
Avg merge
1d 17h
Merged PRs (30d)
19

Description

E tensorflow/core/common_runtime/session.cc:91] Failed to create session: Already exists: TensorFlow device (GPU:0) is being mapped to multiple CUDA devices (1 now, and 0 previously), which is not supported. This may be the result of providing different GPU configurations (ConfigProto.gpu_options, for example different visible_device_list) when creating multiple Sessions in the same process. This is not  currently supported, see https://github.com/tensorflow/tensorflow/issues/19083 
E tensorflow/c/c_api.cc:2184] Already exists: TensorFlow device (GPU:0) is being mapped to multiple CUDA devices (1 now, and 0 previously), which is not supported. This may be the result of providing different GPU configurations (ConfigProto.gpu_options, for example different visible_device_list) when creating multiple Sessions in the same process. This is not  currently supported, see https://github.com/tensorflow/tensorflow/issues/19083 
[dnn_tensorflow @ 0x7f57a4052500] Failed to create new session with model graph 
[dnn_tensorflow @ 0x7f57a4052500] Failed to load native model 
[livepeer_dnn @ 0x7f57a4058400] could not load DNN model 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator fatal error: unexpected signal during runtime execution 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x476085] 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator runtime stack: 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator runtime.throw(0x1f1d68a, 0x2a) 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator 	/usr/local/go/src/runtime/panic.go:1116 +0x72 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator runtime.sigpanic() 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator 	/usr/local/go/src/runtime/signal_unix.go:726 +0x4ac 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator goroutine 19245 [syscall]: 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator runtime.cgocall(0x11b4810, 0xc00091d810, 0x1d69e20) 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator 	/usr/local/go/src/runtime/cgocall.go:133 +0x5b fp=0xc00091d7e0 sp=0xc00091d7a8 pc=0x4a64bb 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator github.com/livepeer/lpms/ffmpeg._Cfunc_lpms_transcode_new_with_dnn(0xc000a4f240, 0x0) 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator 	_cgo_gotypes.go:370 +0x4a fp=0xc00091d810 sp=0xc00091d7e0 pc=0xd14b6a 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator github.com/livepeer/lpms/ffmpeg.NewTranscoderWithDetector.func5(0xc000a4f240, 0xc0007ad480) 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator 	/go/pkg/mod/github.com/livepeer/lpms@v0.0.0-20210806125031-9fdbf80c8575/ffmpeg/ffmpeg.go:541 +0x4d fp=0xc00091d840 sp=0xc00091d810 pc=0xd1a2ed 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator github.com/livepeer/lpms/ffmpeg.NewTranscoderWithDetector(0x211dfc0, 0xc00012d720, 0x7ffda59a3db6, 0x1, 0x0, 0x0, 0x0) 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator 	/go/pkg/mod/github.com/livepeer/lpms@v0.0.0-20210806125031-9fdbf80c8575/ffmpeg/ffmpeg.go:541 +0x245 fp=0xc00091d8e0 sp=0xc00091d840 pc=0xd17d25 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator github.com/livepeer/go-livepeer/core.NewNvidiaTranscoderWithDetector(0x211dfc0, 0xc00012d720, 0x7ffda59a3db6, 0x1, 0x1, 0x7ffda59a3db6, 0x1, 0xc000aae7b0) 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator 	/build/core/transcoder.go:149 +0x4d fp=0xc00091d940 sp=0xc00091d8e0 pc=0x1103d6d 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator github.com/livepeer/go-livepeer/core.(*LoadBalancingTranscoder).createSession(0xc0002ca000, 0xc00060d860, 0x0, 0x0, 0x0) 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator 	/build/core/lb.go:95 +0x389 fp=0xc00091da78 sp=0xc00091d940 pc=0x10f5789 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator github.com/livepeer/go-livepeer/core.(*LoadBalancingTranscoder).Transcode(0xc0002ca000, 0xc00060d860, 0x2e89680, 0x16ee98, 0x1ffe00) 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator 	/build/core/lb.go:67 +0xd4 fp=0xc00091dae8 sp=0xc00091da78 pc=0x10f5254 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator github.com/livepeer/go-livepeer/core.(*LivepeerNode).transcodeSeg(0xc0005cc420, 0x214a480, 0xc000e704c0, 0x214a480, 0xc000e704c0, 0xc000e70480, 0xc00060d860, 0x781ba5) 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator 	/build/core/orchestrator.go:550 +0x3dc fp=0xc00091de80 sp=0xc00091dae8 pc=0x10fb8bc 
Aug 06 08:33:11 ewr-prod-livepeer-orchestrator-10 orchestrator github.com/livepeer/go-livepeer/core.(*LivepeerNode).transcodeSegmentLoop.func1(0x214a480, 0xc000e704c0, 0x214a480, 0xc000e704c0, 0xc0005cc420, 0xc00060d860, 0xc013210600, 0x214a480, 0xc000e704c0, 

This occurred when using the branch for https://github.com/livepeer/go-livepeer/pull/1979.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the branch for pull request #1979 with the failure path through core/transcoder.go:149 and core/lb.go:95. Trace ffmpeg.NewTranscoderWithDetector and the TensorFlow session creation error, then reproduce the model-loading failure and confirm that transcoding no longer crashes when the device mapping is encountered.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, tensorflow
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.