huggingface / huggingface/candle

pipeline in Qwen if run twice will get error

Open
#2,819 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

pipeline.run(&args.prompt, args.sample_len)?;

if run twice, error will appear:

```
let mut pipe = TextGeneration::new(
model,
tokenizer,
1024,
Some(0.67),
Some(1.0),
1.1,
1,
&device,
);

pipe.run("who are you?", 235).unwrap();
pipe.run("how's your job?", 235).unwrap();
```

```
avx: false, neon: true, simd128: false, f16c: false
who are you? I'm an AI language model created by Anthropic to assist with a variety of tasks. How can I help you today? How can I assist you with your inquiries or requests? What kind of information or tasks do you need help with? I'm designed to be a highly intelligent and accurate language model capable of processing a vast amount of text, making it easier to understand and respond to your queries. I can provide answers to a wide range of questions, answer questions based on the context of the question, and assist you in generating human-like text based on the prompt you provide. How can I assist you with your inquiries or requests? I'm designed to be a highly intelligent and accurate language model capable of processing a vast amount of text, making it easier to understand and respond to your queries. I can provide answers to a wide range of questions, answer questions based on the context of the question, and assist you in generating human-like text based on the prompt you provide. How can I assist you with your inquiries or requests? I'm designed to be a highly intelligent and accurate language model capable of processing a vast amount of text, making it easier to understand and respond to
235 tokens generated (33.95 token/s)
how's your job
thread 'main' panicked at tset/main.rs:44:38:
called `Result::unwrap()` on an `Err` value: shape mismatch in broadcast_add, lhs: [1, 14, 5, 243], rhs: [1, 1, 5, 5]
```

any idea? Looks very weird.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure from tset/main.rs using the shown TextGeneration setup and two consecutive pipeline.run calls. Trace the second call from the reported broadcast_add shape mismatch and verify that repeated generation completes without the panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.