bigscience-workshop / bigscience-workshop/petals

IndexError: tuple index out of range

Open
#532 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10.6k
Forks
650
PR merge metrics
No merged PRs in 30d

Description

To get bootstrapped, I tried to use the example from Readme

```
from transformers import AutoTokenizer
from petals import AutoDistributedModelForCausalLM
import torch

# Choose any model available at https://health.petals.dev
model_name = "petals-team/StableBeluga2" # This one is fine-tuned Llama 2 (70B)

# Connect to a distributed network hosting model layers
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoDistributedModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float32)

# Run the model as if it were on your computer
inputs = tokenizer("A cat sat", return_tensors="pt")["input_ids"]
outputs = model.generate(input_ids=inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0])) # A cat sat on a mat...
```

The ```torch_dtype=torch.float32``` was added due to CPU support warning but apart from that rest is the same as the original example yet I am facing the error and unable to complete the inference.

![Screenshot from 2023-10-25 10-30-08](https://github.com/bigscience-workshop/petals/assets/19311617/e63c4083-df3f-48de-9c4f-4203ecff78d8)

OS : Ubuntu 22.04
CPU : i7-7700K
GPU: Nvidia 1070

Please guide if i am missing something here.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.