huggingface / huggingface/peft

Avoid duplicated shared projections in RandLoRA adapter checkpoints

Open
#3,709 2 comments 0 reactions 1 assignee Claimed by @BenjaminBossan View on GitHub
Dominant language
Python
Stars
21.7k
Forks
2.5k
Avg merge
4d 12h
Merged PRs (30d)
59

Description

### System Info

- OS: Ubuntu 22.04.5 LTS
- Kernel: Linux 5.15.0-190-generic x86_64
- Python: 3.10.20
- PEFT: 0.20.1.dev0
- PEFT commit: 279fccce200686af324611d7c49e3adab60485e1
- PyTorch: 2.8.0+cu128
- Transformers: 5.8.1
- safetensors: 0.8.0
- Accelerate: 1.13.0
- CUDA: 12.8
- GPU: NVIDIA A100 80GB PCIe
- NVIDIA driver: 580.173.02

### Who can help?

@BenjaminBossan

### Reproduction

RandLoRA's shared `randlora_A` and `randlora_B` projections are exposed in the adapter state dict both at the model level and through every targeted layer.

The following reproducer was run against `upstream/main` at commit `279fccce200686af324611d7c49e3adab60485e1`:

```python
import os
import tempfile

from safetensors.torch import save_file

from peft import RandLoraConfig, get_peft_model, get_peft_model_state_dict
from tests.test_randlora import MLP

model = get_peft_model(
MLP(),
RandLoraConfig(
target_modules=["lin0", "lin1", "lin2"],
r=4,
init_weights=False,
save_projection=True,
),
)

state_dict = get_peft_model_state_dict(model)

with tempfile.TemporaryDirectory() as tmp:
save_file(state_dict, os.path.join(tmp, "adapter.safetensors"))
```

The state dict contains 14 keys, including four aliases for each shared projection:

```text
randlora_A:
base_model.model.lin0.randlora_A
base_model.model.lin1.randlora_A
base_model.model.lin2.randlora_A
base_model.randlora_A

randlora_B:
base_model.model.lin0.randlora_B
base_model.model.lin1.randlora_B
base_model.model.lin2.randlora_B
base_model.randlora_B
```

All aliases in each group point to the same storage.

The complete error is:

```text
Traceback (most recent call last):
File "", line 37, in
File ".../site-packages/safetensors/torch.py", line 324, in save_file
_flatten_as_ptr(tensors, keep_references_alive), filename, metadata=metadata
File ".../site-packages/safetensors/torch.py", line 569, in _flatten_as_ptr
_evaluate_tensors_for_save(tensors)
File ".../site-packages/safetensors/torch.py", line 557, in _evaluate_tensors_for_save
raise RuntimeError(
RuntimeError:
Some tensors share memory, this will lead to duplicate memory on disk and potential differences when loading them again:
[{'base_model.model.lin1.randlora_A', 'base_model.randlora_A',
'base_model.model.lin2.randlora_A', 'base_model.model.lin0.randlora_A'},
{'base_model.model.lin0.randlora_B', 'base_model.randlora_B',
'base_model.model.lin1.randlora_B', 'base_model.model.lin2.randlora_B'}].
A potential way to correctly save your model is to use `save_model`.
More information at https://huggingface.co/docs/safetensors/torch_shared_tensors
```

With one target layer, the same issue occurs with two aliases per projection.

With `save_projection=False`, the projections are non-persistent and direct saving succeeds.

The existing test suite currently passes:

```text
10 passed, 4 warnings
```

However, it does not test direct safetensors serialization of shared RandLoRA projections.

### Expected behavior

The shared RandLoRA projections should be serialized only once per adapter.

A suitable checkpoint should:

- contain one canonical `randlora_A` tensor;
- contain one canonical `randlora_B` tensor;
- allow direct use of `safetensors.save_file(get_peft_model_state_dict(model))`;
- avoid writing the same projection data repeatedly through layer aliases;
- preserve forward outputs after save/load;
- remain compatible with existing checkpoints containing duplicated aliases;
- preserve the existing behavior when `save_projection=False`.

Current measurements for the three-target reproducer are:

| Measurement | Current behavior |
| --- | ---: |
| Unique tensor storage | 3,048 bytes |
| State-dict tensor bytes including aliases | 8,808 bytes |
| `save_pretrained` tensor payload | 8,808 bytes |
| Saved safetensors file size | 10,168 bytes |
| Projection entries in the saved file | 8 |

`PeftModel.save_pretrained` currently avoids the exception by cloning shared tensors, but the duplicated projection payload is still written to the checkpoint.

This unresolved behavior was also mentioned in [#3490](https://github.com/huggingface/peft/pull/3490), where duplicated RandLoRA shared weights were listed as an outstanding issue.

The proposed implementation would retain the model-level projections as canonical checkpoint entries, remove redundant layer-level aliases during saving, and restore the shared references when loading. The implementation should also keep backward compatibility with existing duplicated RandLoRA checkpoints.

The proposed tests are:

- direct `safetensors.save_file(get_peft_model_state_dict(model))` succeeds;
- exactly one canonical A/B projection pair is serialized;
- `PeftModel.save_pretrained` no longer writes repeated projection payloads;
- checkpoint round-trip preserves model outputs;
- existing duplicated RandLoRA checkpoints remain loadable;
- `save_projection=False` behavior remains unchanged.

This issue was prepared with AI assistance. I reviewed the reproducer, source behavior, and proposed compatibility requirements myself.

Please assign this issue to me. I will take responsibility for resolving this issue and submit a PR addressing it. I would appreciate it if a maintainer could review it when convenient, and I am happy to make any changes based on feedback.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.