BOFT with Constraint Loads Incorrectly
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
### Custom Node Testing
- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)
### Expected Behavior
A BOFT (trained using LyCORIS and Kohya's SD-Scripts) with a `constraint` set to a non-zero value should load and run inference correctly.
### Actual Behavior
Such a BOFT does load without an error, but does not generate images that resemble the samples produced during training. There's a problem with how Comfy is loading the BOFT model.
### Steps to Reproduce
1. Train a BOFT with Kohya's SD-Scripts with a configuration that includes:
```TOML
network_module = "lycoris.kohya"
network_args = [
"algo=boft",
"constraint=0.001",
"conv_dim=10",
"conv_alpha=10",
]
network_dim = 10
network_alpha = 10
```
2. Generate samples to confirm training is working correctly.
3. Load the BOFT in ComfyUI using the vanilla node.
4. Generate an image and compare to the Kohya samples.
### Debug Logs
```powershell
This bug does not produce meaningful logs.
```
### Other
The problem appears to be an oversight in the [reimplemention of LyCORIS's BOFT module](https://github.com/Comfy-Org/ComfyUI/blob/v0.3.77/comfy/weight_adapter/boft.py). LyCORIS performs a scaling of the constraint internally while training, but saves the `constraint` argument as it was provided into the model when it is exported:
```python
self.constraint = constraint * out_dim
self.register_buffer("alpha", torch.tensor(constraint))
```
This same scaling is not reproduced by Comfy when it loads the BOFT:
```python
if alpha > 0: # alpha in boft/bboft is for constraint
q_norm = torch.norm(q) + 1e-8
if q_norm > alpha:
normed_q = q * alpha / q_norm
```
I replaced the above with the following to fix my local copy:
```python
# alpha in boft/bboft is for constraint
# scale by output dim to match LyCORIS
constraint_val = alpha * weight.shape[0]
if constraint_val > 0:
q_norm = torch.norm(q) + 1e-8
if q_norm > constraint_val:
normed_q = q * constraint_val / q_norm
```
And now the BOFT generates imagery that resembles the training samples.
Contributor guide
Research direction
The affected implementation is comfy/weight_adapter/boft.py; start by reading its BOFT constraint handling and reproduce the issue with the LyCORIS/Kohya configuration in the report. Compare ComfyUI output with the Kohya samples after accounting for the output dimension, and confirm that a non-zero constraint produces matching inference behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 80/100