huggingface / huggingface/peft

Add Astra (Activation-Space Tail-Eigenvector LoRA) initialization

Open
#3,662 2 comments 0 reactions 1 assignee Claimed by @BenjaminBossan View on GitHub
Dominant language
Python
Stars
21.7k
Forks
2.5k
Avg merge
4d 12h
Merged PRs (30d)
59

Description

### Feature request

## Description

I would like to propose adding **Astra** (Activation-Space Tail-Eigenvector Low-Rank Adaptation) as a LoRA
initialization method to PEFT. Astra is published at ACL 2026 Findings ([[arXiv:2602.19111](https://arxiv.org/abs/2602.19111)](https://arxiv.org/abs/2602.19111)) with a reference implementation at [[LyoAI/Astra](https://github.com/LyoAI/Astra)](https://github.com/LyoAI/Astra).

As with PiSSA/CorDA, the model output is unchanged at the start of training, but the trainable update is constrained to a task-aware activation subspace. In the paper, Astra outperforms LoRA/PiSSA/CorDA-style baselines on 16 NLU/NLG benchmarks (GLUE-style tasks, math reasoning, code generation, commonsense reasoning) at reduced rank budgets, and all results are reproducible from the public code.

## Proposed code

```python
from peft import LoraConfig, get_peft_model
from peft.tuners.lora.astra import preprocess_astra
from peft.tuners.lora.config import AstraConfig

lora_config = LoraConfig(
init_lora_weights="astra",
r=128,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
astra_config=AstraConfig(astra_method="ipm"), # or "kpm"
)
preprocess_astra(model, lora_config, run_model=run_model) # collects covariance + eigendecomposition
model = get_peft_model(model, lora_config)
```

Implementation outline

`AstraConfig` sub-config (`cache_file`, `covariance_file`, `astra_method`, `verbose`,
`use_float16_for_covariance`, `prune_temporary_fields`) + `LoraConfig(astra_config=...)`.

- `preprocess_astra()` in `src/peft/tuners/lora/astra.py` with forward hooks on module outputs.
- `LoraLayer.astra_init()` and an `init_lora_weights="astra"` branch in `update_layer()`.
- Tests in `tests/test_initialization.py` (identity transform, cache/covariance reuse, hook cleanup, temporary field
pruning, IPM/KPM) and `tests/test_config.py` (nested config roundtrip).
- Docs entry + `examples/astra_finetuning/` example.

## Motivation

PEFT already ships several data- and weight-driven initializations (PiSSA, CorDA, EVA, LoRA-GA, ...). Astra
complements them with an output-activation-eigenspace initialization that is cheap to compute (one eigendecomposition
per target layer, no SVD of the weight, no covariance inverse) and, per the paper, gives better convergence and
final quality at the same or lower rank. The integration is small and follows the existing CorDA pattern almost
one-to-one, so maintenance cost should be low.

## Additional context

- Paper: https://arxiv.org/abs/2602.19111 (ACL 2026 Findings)
- Code: https://github.com/LyoAI/Astra
- The reference implementation currently pins `peft==0.14.0` and uses a custom `AstraLayer`; the proposed integration
reuses the stock PEFT LoRA layers and needs no custom layer class.

### Your contribution

I'm one of the Astra authors and would be happy to submit the PR myself. I'll open the PR as soon as the approach is approved, and I'm also willing to help maintain this integration afterwards.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.