Comfy-Org / Comfy-Org/ComfyUI

TrainLoraNode: expose LoRA alpha instead of hardcoding alpha=1.0 for new adapters

Open
#15,848 0 comments 0 reactions 0 assignees View on GitHub
Feature
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Feature Idea

`TrainLoraNode` hardcodes `alpha=1.0` when creating a new trainable LoRA
adapter, while `rank` is configurable from the node UI. I would like `alpha`
exposed as a node input.

For a new adapter the code path (`comfy_extras/nodes_train.py`) effectively
does:

```python
adapter_cls.create_train(
module.weight,
rank=rank,
alpha=1.0,
)
```

Since LoRA scaling is `alpha / rank`, this means a `rank=16` run trains with a
scaling of `0.0625`, whereas the common convention `alpha = rank` gives `1.0`.

This is **not** a claim that `alpha=1` is mathematically invalid — it is a
valid parameterization. The request is that an important LoRA parameter whose
hardcoded value materially changes training behaviour should be visible and
configurable, rather than fixed at a value that differs from most reference
implementations and makes learning rates non-comparable across trainers.

Note that adapter *loading* already supports reading an `.alpha` value from
saved weights, so alpha exists in the adapter representation; it is simply not
configurable when a new adapter is initialized.

**Why it matters in practice.** I compared two single-image overfit runs on
Z-Image Base — rank 16, 250 steps, AdamW, bf16, 0.25 MP, `offloading=false`,
same image, same caption, same generation seeds — differing in the new-adapter
alpha:

Measured relative weight perturbation `|dW|/|W|` on four inspected layers:

```
alpha=1 1.55e-03 8.82e-04 9.71e-04 3.16e-03
alpha=16 1.48e-02 1.12e-02 1.21e-02 1.84e-02
```

With `alpha=rank` the adapter began reproducing subject-specific detail from
the training image across fixed generation seeds, which it had not done before.

Worth noting for anyone reasoning about this: the amplification is **not** 16x.
AdamW's adaptive normalization partly cancels the gradient rescaling — the
`|down|` magnitudes actually *decreased* under `alpha=16` — and the measured
net effect on `dW` was roughly 5.8x to 10x depending on the layer. So this
changes the optimization trajectory, not merely a scalar on the output.

I am not arguing `alpha=rank` should necessarily become the default. The point
is that alpha is behaviorally significant and should not be hidden behind a
fixed constant.

**Suggested implementation**

- expose `alpha` as a numeric `TrainLoraNode` input;
- optionally offer a convenience default such as `alpha = rank`;
- keep `alpha=1.0` as the default if backward compatibility is preferred;
- continue storing the chosen alpha in the saved LoRA so inference and
continued training use the same scaling.

This would also make configurations directly comparable with Diffusers, kohya
and other LoRA trainers, where rank and alpha are specified separately.

### Existing Solutions

I am not aware of a custom node that exposes alpha for ComfyUI's built-in
`TrainLoraNode`. External trainers (kohya sd-scripts, Diffusers training
scripts, ai-toolkit) all expose rank and alpha as separate parameters, which is
the behaviour being requested here.

### Other

**Environment**

- ComfyUI commit `4da9e2dbead52fc1e68beae33fe3d7ad63b63241`, base tag `v0.33.3`
- `comfy_extras/nodes_train.py` as of commit `d0fec2ef` (2026-07-21)
- PyTorch 2.10.0, macOS 26.6.2, Apple Silicon, device MPS

**Possibly related**

- #12191 — "LoRA strength scaling appears inconsistent: object LoRA
(r=16 / alpha=64) requires strength ~3.0 to match trained geometry". Different
context, but the same underlying point that rank/alpha scaling conventions
need to be explicit to be interpretable.

Contributor guide

Open the contributing guide

Research direction

Read comfy_extras/nodes_train.py, starting at TrainLoraNode and the adapter_cls.create_train call, then trace how rank becomes a node input and how adapter parameters are saved. Inspect the existing loading of .alpha; done means alpha is configurable for new adapters, the selected value is preserved in saved LoRA data, and the chosen default behavior is clear.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.