huggingface / huggingface/peft
KappaTuneSelector does not dequantize bitsandbytes 8-bit weights, computing condition numbers on raw int8 codes
- Dominant language
- Python
- Stars
- 21.7k
- Forks
- 2.5k
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 59
Description
### System Info
- peft: main (`0e8d0ae8`)
- transformers: 5.16.1
- torch: 2.13.0
- Python 3.11
- Affects bitsandbytes 8-bit models (`load_in_8bit=True`); independent of platform
### Who can help?
@benjaminbossan @githubnemo
### Reproduction
`KappaTuneSelector._compute_kappas` tries to dequantize bnb 8-bit weights, but it reads `CB`/`SCB` from the wrong object:
https://github.com/huggingface/peft/blob/0e8d0ae8ab94f189f28b845e293d7452d7892d91/src/peft/helpers.py#L446-L455
```python
weight = module.weight
if bnb is not None:
if hasattr(weight, "quant_state"): # 4-bit
w = bnb.functional.dequantize_4bit(weight.data, weight.quant_state).float()
elif hasattr(weight, "state") and hasattr(weight.state, "CB"): # int8
w = bnb.functional.int8_vectorwise_dequant(weight.state.CB, weight.state.SCB).float()
else:
w = weight.data.detach().float()
```
In bitsandbytes, `CB` and `SCB` are attributes of the `Int8Params` weight itself (`self.CB = CB`, `self.SCB = SCB` in `bitsandbytes/nn/modules.py`). `state` (`MatmulLtState`) belongs to the `Linear8bitLt` *module*, not to its weight. So `hasattr(weight, "state")` is `False` for 8-bit weights, and the int8 branch is never taken.
The code then falls through to `w = weight.data.detach().float()`, which computes the SVD on the raw int8 codes without applying the per-row scales. Because each row is scaled by a different factor, the resulting condition numbers (and therefore the selected targets) differ from those of the actual weights. This happens silently, with no error or warning.
For comparison, PEFT's own `dequantize_bnb_weight` in `src/peft/utils/integrations.py` handles 8-bit correctly by taking the module's `state` and falling back to `weight.SCB`.
The existing GPU test `test_kappatune_with_4bit_model` in `tests/test_gpu_examples.py` only covers 4-bit, so this path isn't exercised.
Note: I found this by reading the PEFT and bitsandbytes source; I don't have a GPU available to run it. A reproducer along these lines should show the problem (untested):
```python
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft.helpers import KappaTuneSelector
model_id = "facebook/opt-125m"
model_8bit = AutoModelForCausalLM.from_pretrained(
model_id, quantization_config=BitsAndBytesConfig(load_in_8bit=True), device_map="cuda"
)
module = model_8bit.model.decoder.layers[0].self_attn.q_proj
print(type(module.weight).__name__, hasattr(module.weight, "state"), hasattr(module.weight, "CB"))
# expected: Int8Params False True -> int8 branch is skipped
model_fp = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float32, device_map="cuda")
kappa_8bit = KappaTuneSelector(model_8bit, show_progress=False)
kappa_fp = KappaTuneSelector(model_fp, show_progress=False)
kappa_8bit._compute_kappas()
kappa_fp._compute_kappas()
name = "model.decoder.layers.0.self_attn.q_proj"
print(kappa_8bit._condition_numbers[name], kappa_fp._condition_numbers[name])
# expected: values differ noticeably, since the 8-bit one is computed on unscaled int8 codes
```
### Expected behavior
For 8-bit bnb models, KappaTune should dequantize the weights (e.g. via `weight.CB`/`weight.SCB`, or by reusing `dequantize_bnb_weight(module.weight, state=module.state)`) before computing condition numbers. Its target selection should then approximately match the selection for the same model loaded unquantized. An 8-bit case could be added next to `test_kappatune_with_4bit_model`.
Contributor guide
Assessment
This issue has not been assessed yet.