huggingface / huggingface/candle

qwen3_vl: no way to reset the KV cache, so a model instance is single-use

Open Beginner friendly
#3,901 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

## Summary

`Qwen3VLModel` provides no way to reset its KV cache. Each `Attention` creates `kv_cache: Arc>` in `Attention::new`, and nothing else can reach it, so a second generation on the same instance decodes against the previous conversation's keys and values. Since the model is typically multiple GB, reconstructing it per generation isn't a practical workaround.

## Where

`candle-transformers/src/models/qwen3_vl/text.rs` — `kv_cache` is created in `Attention::new` and used only in `Attention::forward`. `grep -rn "clear_kv_cache\|fn reset" candle-transformers/src/models/qwen3_vl/` returns nothing.

For comparison, 33 model files in the crate expose `pub fn clear_kv_cache`, including `qwen2.rs`, `qwen3.rs`, `qwen3_moe.rs`, `quantized_qwen3.rs`, `paligemma.rs` and `mixformer.rs`.

## Reproduction

Randomly initialised tiny model, CPU, no downloads, runs in well under a second.

`Cargo.toml`:

```toml
[dependencies]
candle-core = "0.10.2"
candle-nn = "0.10.2"
candle-transformers = "0.10.2"
serde_json = "1"
```

`src/main.rs`:

```rust
use candle_core::{DType, Device, Tensor};
use candle_nn::{VarBuilder, VarMap};
use candle_transformers::models::qwen3_vl::{Config, Qwen3VLModel};

const TINY_CONFIG: &str = r#"{
"text_config": {
"head_dim": 8, "vocab_size": 64, "hidden_size": 32,
"intermediate_size": 64, "num_hidden_layers": 2,
"num_attention_heads": 4, "num_key_value_heads": 2,
"hidden_act": "silu", "max_position_embeddings": 128,
"rms_norm_eps": 0.000001, "tie_word_embeddings": false,
"rope_theta": 10000.0, "sliding_window": null
},
"vision_config": {
"depth": 1, "hidden_size": 32, "out_hidden_size": 32,
"intermediate_size": 32, "num_heads": 2
},
"image_token_id": 5, "video_token_id": 6,
"vision_start_token_id": 7, "vision_end_token_id": 8
}"#;

// Text-only forward: no images, so the vision tower is bypassed.
fn forward_text(model: &Qwen3VLModel, tokens: &[u32], offset: usize, device: &Device) -> Tensor {
let input = Tensor::new(tokens, device).unwrap().unsqueeze(0).unwrap();
model
.forward(&input, None, None, None, None,
vec![tokens.len()], vec![vec![]], vec![vec![]], &[offset])
.unwrap()
}

fn main() {
let device = Device::Cpu;
let config: Config = serde_json::from_str(TINY_CONFIG).unwrap();
let varmap = VarMap::new();
let vb = VarBuilder::from_varmap(&varmap, DType::F32, &device);

// Reference: a pristine model answering the second prompt.
let fresh = Qwen3VLModel::new(&config, vb.clone()).unwrap();
let reference = forward_text(&fresh, &[33, 44], 0, &device);

// Same prompt, on a model that already ran a different one.
let reused = Qwen3VLModel::new(&config, vb).unwrap();
let _ = forward_text(&reused, &[11, 22], 0, &device);
let after_reuse = forward_text(&reused, &[33, 44], 0, &device);

let a = reference.flatten_all().unwrap().to_vec1::().unwrap();
let b = after_reuse.flatten_all().unwrap().to_vec1::().unwrap();
let max_diff = a.iter().zip(b.iter()).map(|(x, y)| (x - y).abs()).fold(0.0f32, f32::max);
let scale = a.iter().map(|v| v.abs()).fold(0.0f32, f32::max);
println!("max |fresh - reused| = {max_diff:.6}");
println!("relative = {:.6}", max_diff / scale);
}
```

The second conversation has to use *different* tokens from the first. Repeating the same tokens hides the bug: the stale entries are then bit-identical duplicates of the new keys and values, and attention over duplicated (key, value) pairs gives the same weighted average.

**Expected:** identical logits — same prompt, same weights.

**Actual**, four consecutive runs (weights are randomly initialised, so the magnitude varies, but it is never small):

```
max |fresh - reused| = 1.486256 relative = 0.342524
max |fresh - reused| = 3.680627 relative = 0.997730
max |fresh - reused| = 4.044358 relative = 0.773331
max |fresh - reused| = 1.621907 relative = 0.423533
```

## Suggested fix

Add `pub fn clear_kv_cache(&mut self)` on `Qwen3VLTextModel` forwarding to each layer's `KvCache::reset()`, and expose it on `Qwen3VLModel`, mirroring `qwen2.rs` / `qwen3.rs`. Happy to open a PR if that approach looks right.

## Versions

`candle-transformers` 0.10.2 from crates.io; `qwen3_vl/text.rs` is unchanged on `main` as of 2026-08-17. Reproduced on macOS/aarch64, CPU.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in candle-transformers/src/models/qwen3_vl/text.rs, where Attention creates and uses the KV cache, then compare the cache-clearing methods in qwen2.rs and qwen3.rs. Expose clearing through Qwen3VLTextModel and Qwen3VLModel, and verify the supplied tiny CPU reproduction produces identical logits for a fresh and reset model.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.