huggingface / huggingface/peft
Conv LoRA adapters do not preserve dilation and padding mode
- Dominant language
- Python
- Stars
- 21.7k
- Forks
- 2.5k
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 59
Description
# Description
LoRA convolutional branches do not appear to preserve all relevant convolutional parameters from the base convolutional layer. In particular, ``dilation`` and ``padding_mode`` are not propagated when the LoRA convolutional layers are constructed, which can lead to incompatible spatial output shapes for dilated convolutions and inconsistent boundary behaviour for non-default padding modes.
# Reproduction
The following minimal script reproduces the shape mismatch:
```python
"""Reproduce a shape mismatch in a LoRA-wrapped dilated Conv2d."""
import torch
from torch import nn
from peft import LoraConfig, get_peft_model
class TinyConvModel(nn.Module):
def __init__(self):
super().__init__()
self.conv = nn.Conv2d(
in_channels=1,
out_channels=1,
kernel_size=3,
padding=2,
dilation=2,
padding_mode="reflect",
)
def forward(self, x):
return self.conv(x)
def main():
model = TinyConvModel()
inputs = torch.randn(1, 1, 8, 8)
config = LoraConfig(r=1, lora_alpha=1, target_modules=["conv"])
model = get_peft_model(model, config)
model(inputs)
if __name__ == "__main__":
main()
```
Full traceback:
```
Traceback (most recent call last):
File "C:\Users\tomas\peft\reproduce_lora_conv2d_dilation.py", line 36, in
main()
~~~~^^
File "C:\Users\tomas\peft\reproduce_lora_conv2d_dilation.py", line 32, in main
model(inputs)
~~~~~^^^^^^^^
File "C:\Users\tomas\miniconda3\Lib\site-packages\torch\nn\modules\module.py", line 1779, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "C:\Users\tomas\miniconda3\Lib\site-packages\torch\nn\modules\module.py", line 1790, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\tomas\peft\src\peft\peft_model.py", line 1005, in forward
return self.get_base_model()(*args, **kwargs)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "C:\Users\tomas\miniconda3\Lib\site-packages\torch\nn\modules\module.py", line 1779, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "C:\Users\tomas\miniconda3\Lib\site-packages\torch\nn\modules\module.py", line 1790, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\tomas\peft\reproduce_lora_conv2d_dilation.py", line 22, in forward
return self.conv(x)
~~~~~~~~~^^^
File "C:\Users\tomas\miniconda3\Lib\site-packages\torch\nn\modules\module.py", line 1779, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "C:\Users\tomas\miniconda3\Lib\site-packages\torch\nn\modules\module.py", line 1790, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\tomas\peft\src\peft\tuners\lora\layer.py", line 1797, in forward
result = result + lora_B(lora_A(dropout(x))) * scaling
~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RuntimeError: The size of tensor a (8) must match the size of tensor b (10) at non-singleton dimension 3
```
# Contribution
I believe the issue should not arise frequently at all, because LoRA may be more commonly applied to linear layers, and most convolutional layers use default dilation and padding. Still, I think preserving the full convolutional configurations seems necessary for correctness in some cases.
I have already a local implementation that appears to fix the issue. If the maintainers agree that this is an appropriate fix, I would like to submit a small PR.
Contributor guide
Research direction
Start by running the minimal PyTorch reproduction in the issue, then inspect src/peft/tuners/lora/layer.py around the convolutional forward path and layer construction. Trace which base convolution parameters are carried into the LoRA branches. Done means the dilated, reflect-padded example runs without a shape mismatch and the LoRA branches preserve the stated convolution settings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100