livepeer / livepeer/go-livepeer

Live-runner: register_runner() only reports one GPU when a single runner manages multiple devices

Open
#3,986 1 comment 0 reactions 0 assignees View on GitHub
status: triage
Dominant language
Go
Stars
586
Forks
226
Avg merge
1d 17h
Merged PRs (30d)
19

Description

## Summary

When a single transcode runner process manages multiple GPUs internally (round-robin dispatch across `device_ids`, following the multi-GPU pattern from livepeer/app-examples#25), only **one GPU** shows up in `/discovery`, even though all configured GPUs are actively processing jobs.

## Root cause

In `livepeer-python-gateway`'s `live_runner.py`:
- `register_runner()`'s `gpu` param only accepts a single `LiveRunnerGPU` object (not a list).
- The auto-detect fallback `_detect_gpu_nvidia_smi()` returns just `rows[0]` from `nvidia-smi` when it can't match the current process to a specific device via `--query-compute-apps`.
- Since a runner registers **once** at startup, only one GPU is ever reported, regardless of how many devices the process actually round-robins jobs across.

## Impact

Confirmed on a 2-GPU box (RTX 5090 + RTX 5070 Ti) running the `transcode/ffmpeg` app from app-examples with `device_ids: ['0','1']`:
- `nvidia-smi` shows real, simultaneous enc/dec load on both GPUs.
- `/discovery` only ever lists one of the two — capacity/GPU info for the second device is invisible to clients, even though it's actively serving jobs and being paid for.

## Suggested directions

1. `register_runner()` / `LiveRunnerGPU` could accept a list of GPUs per runner registration.
2. Alternatively, document that the supported multi-GPU pattern is one runner process (one registration) per GPU, rather than a single process spanning multiple `device_ids`.

Happy to provide logs/config or help test a fix.

Contributor guide

Open the contributing guide

Research direction

Start in live_runner.py with register_runner() and _detect_gpu_nvidia-smi(), then trace how the /discovery registration represents GPU data. Compare the multi-device device_ids configuration with the current single-GPU result and determine which supported direction fits. Done means either every actively used GPU is represented in /discovery for one runner, or the one-runner-per-GPU limitation is documented clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.