livepeer / livepeer/go-livepeer
AI: `-nvidia "all"` is not obvious what it does
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 586
- Forks
- 226
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 19
Description
**Describe the bug**
When the flag `-nvidia "all"` is used for AI inference, the `aiModels.json` must match number of GPUs installed, else it will try run all GPUs with the first item in the list.
Not sure if this is the expected behavior.
**To Reproduce**
1. Install 2 GPUs in a machine
2. Set `-nvidia "all"`
3. Set your `aiModels.json` to include only the LLM model
```
[
{
"pipeline": "llm",
"model_id": "meta-llama/Meta-Llama-3.1-8B-Instruct",
"price_per_unit": 80000000,
"pixels_per_unit": 1000000,
"warm": true
}
]
```
4. If the second GPU has less than 24GB, it will fail to launch to container and time out.
**Expected behavior**
Obviously if we specify the GPUs ie `-nvidia 0,1` then we assume the `aiModels.json` will have two models loaded, it works.
i.e.
```
[
{
"pipeline": "llm",
"model_id": "meta-llama/Meta-Llama-3.1-8B-Instruct",
"price_per_unit": 80000000,
"pixels_per_unit": 1000000,
"warm": true
},
{
"pipeline": "text-to-image",
"model_id": "ByteDance/SDXL-Lightning",
"price_per_unit": 4768371,
"warm": true
}
]
```
I think the documentation needs to be updated, but I am unsure if this is actually the logic for the `"all"` value.
**Set Up**
Slot 0 - 3090 (24GB Ram)
Slot 1 - 2080 ti (11GB Ram)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the documented behavior of the `-nvidia` flag, especially the `"all"` value, alongside the `aiModels.json` example in the repository. Confirm whether the model list must match the number of GPUs and document the behavior and constraints clearly; the issue provides no specific documentation file or test to run.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai, documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100