huggingface / huggingface/candle

Short context length on Qwen quantized examples.

Open
#3,014 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

Running any of the quantized examples so far they all seam to have a 1024 token limit.

```
cargo run --example quantized-qwen3 --release --features cuda,cudnn -- --which 4b --prompt "1802tokens later..."
```

```
Error: shape mismatch on target dim, dst: 1024, src: 1802 + 0
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the command in the quantized-qwen3 example with the 1802-token prompt. Trace how the example determines or passes the context length and compare it with the 1024-token shape reported in the error. Done means the quantized Qwen example accepts prompts longer than 1024 tokens without the shape mismatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.