huggingface / huggingface/peft

PiSSA / MiCA / CorDA init bypasses quantized-layer guard on 4-bit float-storage layers

Open
#3,742 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
21.7k
Forks
2.5k
Avg merge
4d 16h
Merged PRs (30d)
60

Description

### Summary

`pissa_init`, `mica_init`, and `corda_init` (`src/peft/tuners/lora/layer.py`) gate on `weight.dtype in [float32, float16, bfloat16]` — but a bitsandbytes 4-bit layer with float quant storage (`bnb_4bit_quant_storage` float) *reports* a float dtype while its data is packed nibbles. The guard passes, SVD runs over packed-nibbles-as-floats, and the residual is written back into the `Params4bit`. The same file already does this correctly in `reset_lora_parameters` (`:386-396`: class detection via `get_bnb_param_type` + `dequantize_module_weight`). (LoftQ was initially suspected too, but `loftq_utils.py:81` dequantizes properly — excluded.)

### Minimal reproduction (CPU; proves the guard bypass — packed-data corruption itself needs a CUDA quant state)

```python
import torch, bitsandbytes as bnb
from torch import nn
from peft import LoraConfig, get_peft_model
from peft.utils.integrations import get_bnb_param_type

class QNet(nn.Module):
def __init__(self):
super().__init__()
self.lin = nn.Linear(16, 16)
self.lin.weight = bnb.nn.Params4bit(self.lin.weight.data, quant_type="nf4")

w = QNet().lin.weight
print(type(w).__name__, w.dtype) # Params4bit torch.float32 <- the confusion
print(get_bnb_param_type(w)) # 4bit (class detection works)
get_peft_model(QNet(), LoraConfig(r=4, target_modules=["lin"], init_lora_weights="pissa"))
print("no TypeError — guard bypassed on a quantized-param class")
```

On a CUDA 4-bit layer the consequence is worse than a bypass: `weight.to(torch.float32)` materializes packed values as floats, the SVD factorizes quantization noise, and `weight.data = residual` stores garbage back into the `Params4bit`.

### Root cause

Three dtype-only guards (`pissa_init`, `mica_init` ~:470, `corda_init` ~:499) where a class check is needed. `get_bnb_param_type` (class-name based, `integrations.py:129`) already exists and the correct dequant path (`dequantize_module_weight`) is used two functions away.

### Expected behavior

SVD-based inits either dequantize-then-proceed (matching `reset_lora_parameters`) or raise the existing `TypeError` on quantized layers — never silently factorize packed storage.

### Proposed fix

Replace the three dtype guards with the `get_bnb_param_type` + `dequantize_module_weight` pattern (or reject quantized layers explicitly), plus a CPU-runnable guard test using a float-storage `Params4bit` exactly as above. Happy to PR after a nod.

Environment: Python 3.12, torch 2.13.0+cpu, bitsandbytes 0.50.2, transformers 5.15.1, peft @ `ab2db1e0`.

Contributor guide

Open the contributing guide

Research direction

Start in src/peft/tuners/lora/layer.py at pissa_init, mica_init, corda_init, and compare their dtype guards with reset_lora_parameters around lines 386-396. Read get_bnb_param_type in integrations.py and dequantize_module_weight, then add the CPU float-storage Params4bit regression test described in the issue. Done means quantized layers are dequantized or rejected consistently, with no packed-storage SVD or residual writeback.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.