how to run fnlp/moss-base-7b in multi gpus
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 12.3k
- Forks
- 1.1k
- PR merge metrics
- No merged PRs in 30d
Description
how to run fnlp/moss-base-7b in multi gpus, i use this way:
from transformers import AutoTokenizer, AutoModelForCausalLM
os.environ['CUDA_VISIBLE_DEVICES'] = "0,1"
tokenizer = AutoTokenizer.from_pretrained("fnlp/moss-base-7b", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("fnlp/moss-base-7b", trust_remote_code=True).cuda()
model = model.eval()
inputs = tokenizer(["流浪地球的导演是"], return_tensors="pt")
for k,v in inputs.items():
inputs[k] = v.cuda()
outputs = model.generate(**inputs, do_sample=True, temperature=0.8, top_p=0.8, repetition_penalty=1.1, max_new_tokens=256)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)
however it didn't use. Could you please give me some advice?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Python example in the issue, especially the CUDA_VISIBLE_DEVICES setting and the AutoModelForCausalLM.from_pretrained(...).cuda() entry point. Verify how the loaded model and generated inputs are placed across the selected GPUs; done means the example runs successfully and demonstrably uses both GPUs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100