huggingface / huggingface/candle

QTensor::quantize on a narrowed view quantizes the parent (silently in release, debug_assert in debug)

Open
#3,864 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

In a **release** build, `QTensor::quantize` on a narrowed-then-`contiguous()`
tensor quantizes the *parent's* data instead of the narrowed rows, with no error.
In a debug build the same code panics on a `debug_assert_eq!` instead.

I am reasonably confident about the mechanism but genuinely unsure whether you
consider this a bug or working-as-intended, so I have written down what I
measured and left the conclusion open — see "Is this a bug?" at the end.

**Environment:** CPU backend, `main` at
`6f74e7c390c717f8fd34f23ce02aceb058173370`, Apple M1 Max / macOS 26.6. Nothing
here is Metal- or arch-specific; both pieces of code are backend-independent, but
CPU is the only backend I ran it on. Q8_0 and Q4_0 both checked.
Reproduction branch:
[`oetiker/candle@repro/narrowed-view-quantize-offset`](https://github.com/oetiker/candle/tree/repro/narrowed-view-quantize-offset)
(commit `34d008cc`, one test in `candle-core/tests/quantized_tests.rs`, no fix).

## What I measured

Four rows of 32 f32; row `r` is filled with the value `r + 1`. Narrow to row 1,
so the correct answer is thirty-two `2.0`s. Reading that same view through the
ordinary tensor API returns `2.0`, so the view itself is fine.

**Release profile** (`cargo test --release`) — no error, wrong data:

```
assertion `left == right` failed: quantize read the parent from element 0 instead of the narrowed view at offset 32
left: [0.99993896, 0.99993896, ... 32 values ...]
right: [2.0, 2.0, ... 32 values ...]
```

`0.99993896` is Q8_0's round-trip of `1.0` — i.e. the parent's **row 0**, not the
row that was narrowed to.

**Debug profile** (`cargo test`) — panics instead, inside the quantizer, at
`candle-core/src/quantized/k_quants.rs:666`:

```
assertion `left == right` failed: size mismatch 128 1 32
left: 1
right: 4
```

`128` is the parent's element count against a 32-element shape.

The reason the two profiles differ is that the guard at `k_quants.rs:666` is a
`debug_assert_eq!`, so it is compiled out in release and the quantizer proceeds.

## Mechanism, as far as I can tell

Two pieces of behaviour compose:

1. `Tensor::contiguous` (`candle-core/src/tensor.rs:2475-2486`) returns
`self.clone()` when `is_contiguous()` is already true. A row-narrowed tensor
*is* contiguous, so `contiguous()` does not copy it — the result keeps the
parent's storage and a non-zero `start_offset()`. (`force_contiguous()` is the
one that always copies.) On its own this seems entirely reasonable for
consumers that honour the layout.

2. `QTensor::quantize` (`candle-core/src/quantized/mod.rs:543-556`) passes
`src.storage()` — the whole parent storage — to the quantizer, and uses
`shape.elem_count()` only to size the destination. The layout offset is not
applied.

Composed, `quantize(&t.narrow(..)?.contiguous()?, ..)` reads the parent's first
`shape.elem_count()` elements from element 0.

`force_contiguous()` produces the correct result, and the test pins that too —
same tensor, same values, right answer once the offset has been resolved by a
copy. That is what makes me fairly sure of the diagnosis rather than just the
symptom.

## Why I noticed

Per-expert quantization of an MoE weight stack is naturally written as "narrow to
expert *i*, make it contiguous, quantize". That code uses only public API and
calls the function whose job is to remove layout concerns. It cost me a full
debugging cycle, most of it spent suspecting an entirely innocent quantization
kernel, because the debug assert fires *inside* the quantizer and its message
mentions neither `narrow` nor `contiguous` nor the offset.

## Is this a bug?

I can see a reasonable position where it is not, and I would rather ask than
assume:

- **If the contract is "`quantize` requires a `src` whose storage matches its
shape, and the `debug_assert` is how that contract is enforced"**, then this is
working as intended and the gap is documentation plus the fact that the release
build does not enforce it. In that case the cheap improvement is promoting the
`debug_assert_eq!` at `k_quants.rs:666` to a real `assert!`, so release fails
loudly rather than returning wrong data — and perhaps a note on `quantize` that
`contiguous()` is not sufficient and `force_contiguous()` is what is wanted.

- **If `quantize` is meant to accept any tensor**, then applying the layout offset
in `QTensor::quantize` would be the narrow fix.

There are wider levers — making `quantize` call `force_contiguous()`, or
revisiting what `contiguous()` guarantees — but the latter has a large blast
radius and I doubt it is the right one.

I am happy to send a PR for whichever of these you prefer, including the
"promote the assert and document it" version if that is the answer.

## Reproducing

```
git clone https://github.com/oetiker/candle && cd candle
git checkout repro/narrowed-view-quantize-offset
cargo test --release -p candle-core --test quantized_tests quantize_a_narrowed_view # wrong data
cargo test -p candle-core --test quantized_tests quantize_a_narrowed_view # debug_assert panic
```

CPU only, no accelerator needed.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reproduction in candle-core/tests/quantized_tests.rs, then read Tensor::contiguous in candle-core/src/tensor.rs:2475-2486 and QTensor::quantize in candle-core/src/quantized/mod.rs:543-556. Check the k_quants.rs:666 guard and run both debug and release reproduction commands. Done means the project’s chosen contract is implemented and narrowed-view quantization no longer silently returns the parent’s data.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.