huggingface / huggingface/peft
PiSSA / MiCA / CorDA init bypasses quantized-layer guard on 4-bit float-storage layers
- Dominant language
- Python
- Stars
- 21.7k
- Forks
- 2.5k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 60
Description
### Summary
`pissa_init`, `mica_init`, and `corda_init` (`src/peft/tuners/lora/layer.py`) gate on `weight.dtype in [float32, float16, bfloat16]` — but a bitsandbytes 4-bit layer with float quant storage (`bnb_4bit_quant_storage` float) *reports* a float dtype while its data is packed nibbles. The guard passes, SVD runs over packed-nibbles-as-floats, and the residual is written back into the `Params4bit`. The same file already does this correctly in `reset_lora_parameters` (`:386-396`: class detection via `get_bnb_param_type` + `dequantize_module_weight`). (LoftQ was initially suspected too, but `loftq_utils.py:81` dequantizes properly — excluded.)
### Minimal reproduction (CPU; proves the guard bypass — packed-data corruption itself needs a CUDA quant state)
```python
import torch, bitsandbytes as bnb
from torch import nn
from peft import LoraConfig, get_peft_model
from peft.utils.integrations import get_bnb_param_type
class QNet(nn.Module):
def __init__(self):
super().__init__()
self.lin = nn.Linear(16, 16)
self.lin.weight = bnb.nn.Params4bit(self.lin.weight.data, quant_type="nf4")
w = QNet().lin.weight
print(type(w).__name__, w.dtype) # Params4bit torch.float32 <- the confusion
print(get_bnb_param_type(w)) # 4bit (class detection works)
get_peft_model(QNet(), LoraConfig(r=4, target_modules=["lin"], init_lora_weights="pissa"))
print("no TypeError — guard bypassed on a quantized-param class")
```
On a CUDA 4-bit layer the consequence is worse than a bypass: `weight.to(torch.float32)` materializes packed values as floats, the SVD factorizes quantization noise, and `weight.data = residual` stores garbage back into the `Params4bit`.
### Root cause
Three dtype-only guards (`pissa_init`, `mica_init` ~:470, `corda_init` ~:499) where a class check is needed. `get_bnb_param_type` (class-name based, `integrations.py:129`) already exists and the correct dequant path (`dequantize_module_weight`) is used two functions away.
### Expected behavior
SVD-based inits either dequantize-then-proceed (matching `reset_lora_parameters`) or raise the existing `TypeError` on quantized layers — never silently factorize packed storage.
### Proposed fix
Replace the three dtype guards with the `get_bnb_param_type` + `dequantize_module_weight` pattern (or reject quantized layers explicitly), plus a CPU-runnable guard test using a float-storage `Params4bit` exactly as above. Happy to PR after a nod.
Environment: Python 3.12, torch 2.13.0+cpu, bitsandbytes 0.50.2, transformers 5.15.1, peft @ `ab2db1e0`.
Contributor guide
Research direction
Start in src/peft/tuners/lora/layer.py at pissa_init, mica_init, corda_init, and compare their dtype guards with reset_lora_parameters around lines 386-396. Read get_bnb_param_type in integrations.py and dequantize_module_weight, then add the CPU float-storage Params4bit regression test described in the issue. Done means quantized layers are dequantized or rejected consistently, with no packed-storage SVD or residual writeback.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100