bigscience-workshop / bigscience-workshop/petals
How to get faster inference?
Open
- Dominant language
- Python
- Stars
- 10.6k
- Forks
- 650
- PR merge metrics
- No merged PRs in 30d
Description
I added my RTX 3080 to swarm using:
conda install pytorch pytorch-cuda=11.7 -c pytorch -c nvidia
pip install git+https://github.com/bigscience-workshop/petals
python -m petals.cli.run_server enoch/llama-65b-hf --adapters timdettmers/guanaco-65b
But I still find my inference to be pretty slow, like 2 mins for 200 tokens. Why is that?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.