livepeer / livepeer/go-livepeer
Live-runner: register_runner() only reports one GPU when a single runner manages multiple devices
- Dominant language
- Go
- Stars
- 586
- Forks
- 226
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 19
Description
## Summary
When a single transcode runner process manages multiple GPUs internally (round-robin dispatch across `device_ids`, following the multi-GPU pattern from livepeer/app-examples#25), only **one GPU** shows up in `/discovery`, even though all configured GPUs are actively processing jobs.
## Root cause
In `livepeer-python-gateway`'s `live_runner.py`:
- `register_runner()`'s `gpu` param only accepts a single `LiveRunnerGPU` object (not a list).
- The auto-detect fallback `_detect_gpu_nvidia_smi()` returns just `rows[0]` from `nvidia-smi` when it can't match the current process to a specific device via `--query-compute-apps`.
- Since a runner registers **once** at startup, only one GPU is ever reported, regardless of how many devices the process actually round-robins jobs across.
## Impact
Confirmed on a 2-GPU box (RTX 5090 + RTX 5070 Ti) running the `transcode/ffmpeg` app from app-examples with `device_ids: ['0','1']`:
- `nvidia-smi` shows real, simultaneous enc/dec load on both GPUs.
- `/discovery` only ever lists one of the two — capacity/GPU info for the second device is invisible to clients, even though it's actively serving jobs and being paid for.
## Suggested directions
1. `register_runner()` / `LiveRunnerGPU` could accept a list of GPUs per runner registration.
2. Alternatively, document that the supported multi-GPU pattern is one runner process (one registration) per GPU, rather than a single process spanning multiple `device_ids`.
Happy to provide logs/config or help test a fix.
Contributor guide
Research direction
Start in live_runner.py with register_runner() and _detect_gpu_nvidia-smi(), then trace how the /discovery registration represents GPU data. Compare the multi-device device_ids configuration with the current single-GPU result and determine which supported direction fits. Done means either every actively used GPU is represented in /discovery for one runner, or the one-runner-per-GPU limitation is documented clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100