huggingface / huggingface/peft

Conv LoRA adapters do not preserve dilation and padding mode

Open
#3,697 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
21.7k
Forks
2.5k
Avg merge
4d 12h
Merged PRs (30d)
59

Description

# Description

LoRA convolutional branches do not appear to preserve all relevant convolutional parameters from the base convolutional layer. In particular, ``dilation`` and ``padding_mode`` are not propagated when the LoRA convolutional layers are constructed, which can lead to incompatible spatial output shapes for dilated convolutions and inconsistent boundary behaviour for non-default padding modes.

# Reproduction

The following minimal script reproduces the shape mismatch:

```python
"""Reproduce a shape mismatch in a LoRA-wrapped dilated Conv2d."""

import torch
from torch import nn

from peft import LoraConfig, get_peft_model

class TinyConvModel(nn.Module):
def __init__(self):
super().__init__()
self.conv = nn.Conv2d(
in_channels=1,
out_channels=1,
kernel_size=3,
padding=2,
dilation=2,
padding_mode="reflect",
)

def forward(self, x):
return self.conv(x)

def main():
model = TinyConvModel()
inputs = torch.randn(1, 1, 8, 8)

config = LoraConfig(r=1, lora_alpha=1, target_modules=["conv"])
model = get_peft_model(model, config)

model(inputs)

if __name__ == "__main__":
main()
```

Full traceback:

```
Traceback (most recent call last):
File "C:\Users\tomas\peft\reproduce_lora_conv2d_dilation.py", line 36, in
main()
~~~~^^
File "C:\Users\tomas\peft\reproduce_lora_conv2d_dilation.py", line 32, in main
model(inputs)
~~~~~^^^^^^^^
File "C:\Users\tomas\miniconda3\Lib\site-packages\torch\nn\modules\module.py", line 1779, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "C:\Users\tomas\miniconda3\Lib\site-packages\torch\nn\modules\module.py", line 1790, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\tomas\peft\src\peft\peft_model.py", line 1005, in forward
return self.get_base_model()(*args, **kwargs)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "C:\Users\tomas\miniconda3\Lib\site-packages\torch\nn\modules\module.py", line 1779, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "C:\Users\tomas\miniconda3\Lib\site-packages\torch\nn\modules\module.py", line 1790, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\tomas\peft\reproduce_lora_conv2d_dilation.py", line 22, in forward
return self.conv(x)
~~~~~~~~~^^^
File "C:\Users\tomas\miniconda3\Lib\site-packages\torch\nn\modules\module.py", line 1779, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "C:\Users\tomas\miniconda3\Lib\site-packages\torch\nn\modules\module.py", line 1790, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\tomas\peft\src\peft\tuners\lora\layer.py", line 1797, in forward
result = result + lora_B(lora_A(dropout(x))) * scaling
~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RuntimeError: The size of tensor a (8) must match the size of tensor b (10) at non-singleton dimension 3
```

# Contribution

I believe the issue should not arise frequently at all, because LoRA may be more commonly applied to linear layers, and most convolutional layers use default dilation and padding. Still, I think preserving the full convolutional configurations seems necessary for correctness in some cases.

I have already a local implementation that appears to fix the issue. If the maintainers agree that this is an appropriate fix, I would like to submit a small PR.

Contributor guide

Open the contributing guide

Research direction

Start by running the minimal PyTorch reproduction in the issue, then inspect src/peft/tuners/lora/layer.py around the convolutional forward path and layer construction. Trace which base convolution parameters are carried into the LoRA branches. Done means the dilated, reflect-padded example runs without a shape mismatch and the LoRA branches preserve the stated convolution settings.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.