TrainLoraNode: expose LoRA alpha instead of hardcoding alpha=1.0 for new adapters
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Feature Idea
`TrainLoraNode` hardcodes `alpha=1.0` when creating a new trainable LoRA
adapter, while `rank` is configurable from the node UI. I would like `alpha`
exposed as a node input.
For a new adapter the code path (`comfy_extras/nodes_train.py`) effectively
does:
```python
adapter_cls.create_train(
module.weight,
rank=rank,
alpha=1.0,
)
```
Since LoRA scaling is `alpha / rank`, this means a `rank=16` run trains with a
scaling of `0.0625`, whereas the common convention `alpha = rank` gives `1.0`.
This is **not** a claim that `alpha=1` is mathematically invalid — it is a
valid parameterization. The request is that an important LoRA parameter whose
hardcoded value materially changes training behaviour should be visible and
configurable, rather than fixed at a value that differs from most reference
implementations and makes learning rates non-comparable across trainers.
Note that adapter *loading* already supports reading an `.alpha` value from
saved weights, so alpha exists in the adapter representation; it is simply not
configurable when a new adapter is initialized.
**Why it matters in practice.** I compared two single-image overfit runs on
Z-Image Base — rank 16, 250 steps, AdamW, bf16, 0.25 MP, `offloading=false`,
same image, same caption, same generation seeds — differing in the new-adapter
alpha:
Measured relative weight perturbation `|dW|/|W|` on four inspected layers:
```
alpha=1 1.55e-03 8.82e-04 9.71e-04 3.16e-03
alpha=16 1.48e-02 1.12e-02 1.21e-02 1.84e-02
```
With `alpha=rank` the adapter began reproducing subject-specific detail from
the training image across fixed generation seeds, which it had not done before.
Worth noting for anyone reasoning about this: the amplification is **not** 16x.
AdamW's adaptive normalization partly cancels the gradient rescaling — the
`|down|` magnitudes actually *decreased* under `alpha=16` — and the measured
net effect on `dW` was roughly 5.8x to 10x depending on the layer. So this
changes the optimization trajectory, not merely a scalar on the output.
I am not arguing `alpha=rank` should necessarily become the default. The point
is that alpha is behaviorally significant and should not be hidden behind a
fixed constant.
**Suggested implementation**
- expose `alpha` as a numeric `TrainLoraNode` input;
- optionally offer a convenience default such as `alpha = rank`;
- keep `alpha=1.0` as the default if backward compatibility is preferred;
- continue storing the chosen alpha in the saved LoRA so inference and
continued training use the same scaling.
This would also make configurations directly comparable with Diffusers, kohya
and other LoRA trainers, where rank and alpha are specified separately.
### Existing Solutions
I am not aware of a custom node that exposes alpha for ComfyUI's built-in
`TrainLoraNode`. External trainers (kohya sd-scripts, Diffusers training
scripts, ai-toolkit) all expose rank and alpha as separate parameters, which is
the behaviour being requested here.
### Other
**Environment**
- ComfyUI commit `4da9e2dbead52fc1e68beae33fe3d7ad63b63241`, base tag `v0.33.3`
- `comfy_extras/nodes_train.py` as of commit `d0fec2ef` (2026-07-21)
- PyTorch 2.10.0, macOS 26.6.2, Apple Silicon, device MPS
**Possibly related**
- #12191 — "LoRA strength scaling appears inconsistent: object LoRA
(r=16 / alpha=64) requires strength ~3.0 to match trained geometry". Different
context, but the same underlying point that rank/alpha scaling conventions
need to be explicit to be interpretable.
Contributor guide
Research direction
Read comfy_extras/nodes_train.py, starting at TrainLoraNode and the adapter_cls.create_train call, then trace how rank becomes a node input and how adapter parameters are saved. Inspect the existing loading of .alpha; done means alpha is configurable for new adapters, the selected value is preserved in saved LoRA data, and the chosen default behavior is clear.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100