bigscience-workshop / bigscience-workshop/petals
IndexError: tuple index out of range
- Dominant language
- Python
- Stars
- 10.6k
- Forks
- 650
- PR merge metrics
- No merged PRs in 30d
Description
To get bootstrapped, I tried to use the example from Readme
```
from transformers import AutoTokenizer
from petals import AutoDistributedModelForCausalLM
import torch
# Choose any model available at https://health.petals.dev
model_name = "petals-team/StableBeluga2" # This one is fine-tuned Llama 2 (70B)
# Connect to a distributed network hosting model layers
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoDistributedModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float32)
# Run the model as if it were on your computer
inputs = tokenizer("A cat sat", return_tensors="pt")["input_ids"]
outputs = model.generate(input_ids=inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0])) # A cat sat on a mat...
```
The ```torch_dtype=torch.float32``` was added due to CPU support warning but apart from that rest is the same as the original example yet I am facing the error and unable to complete the inference.

OS : Ubuntu 22.04
CPU : i7-7700K
GPU: Nvidia 1070
Please guide if i am missing something here.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.