Comfy-Org / Comfy-Org/ComfyUI

BOFT with Constraint Loads Incorrectly

Open Beginner friendly
#15,066 0 comments 0 reactions 0 assignees View on GitHub
Potential Bug
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 7h
Merged PRs (30d)
158

Description

### Custom Node Testing

- [x] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

A BOFT (trained using LyCORIS and Kohya's SD-Scripts) with a `constraint` set to a non-zero value should load and run inference correctly.

### Actual Behavior

Such a BOFT does load without an error, but does not generate images that resemble the samples produced during training. There's a problem with how Comfy is loading the BOFT model.

### Steps to Reproduce

1. Train a BOFT with Kohya's SD-Scripts with a configuration that includes:
```TOML
network_module = "lycoris.kohya"
network_args = [
"algo=boft",
"constraint=0.001",
"conv_dim=10",
"conv_alpha=10",
]
network_dim = 10
network_alpha = 10
```
2. Generate samples to confirm training is working correctly.
3. Load the BOFT in ComfyUI using the vanilla node.
4. Generate an image and compare to the Kohya samples.

### Debug Logs

```powershell
This bug does not produce meaningful logs.
```

### Other

The problem appears to be an oversight in the [reimplemention of LyCORIS's BOFT module](https://github.com/Comfy-Org/ComfyUI/blob/v0.3.77/comfy/weight_adapter/boft.py). LyCORIS performs a scaling of the constraint internally while training, but saves the `constraint` argument as it was provided into the model when it is exported:
```python
self.constraint = constraint * out_dim
self.register_buffer("alpha", torch.tensor(constraint))
```
This same scaling is not reproduced by Comfy when it loads the BOFT:
```python
if alpha > 0: # alpha in boft/bboft is for constraint
q_norm = torch.norm(q) + 1e-8
if q_norm > alpha:
normed_q = q * alpha / q_norm
```
I replaced the above with the following to fix my local copy:
```python
# alpha in boft/bboft is for constraint
# scale by output dim to match LyCORIS
constraint_val = alpha * weight.shape[0]
if constraint_val > 0:
q_norm = torch.norm(q) + 1e-8
if q_norm > constraint_val:
normed_q = q * constraint_val / q_norm
```
And now the BOFT generates imagery that resembles the training samples.

Contributor guide

Open the contributing guide

Research direction

The affected implementation is comfy/weight_adapter/boft.py; start by reading its BOFT constraint handling and reproduce the issue with the LyCORIS/Kohya configuration in the report. Compare ComfyUI output with the Kohya samples after accounting for the output dimension, and confirm that a non-zero constraint produces matching inference behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
80/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.