lllyasviel / lllyasviel/stable-diffusion-webui-forge

[Bug]: Fix TypeError: 'NoneType' object is not iterable when generating img2img with controlllite

Open
#768 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

### Checklist

- [ ] The issue exists after disabling all extensions
- [ ] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [X] The issue exists in the current version of the webui
- [ ] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

When dealing with intermediate image sizes such as upscaling 1248×1248 by 1.5 times, Controlllite throws a NoneType object error. This issue occurs around line 234 in torch.cat within extensions-builtin/sd_forge_controlllite/lib_controllllite/lib_controllllite.py due to a mismatch in tensor sizes between cx and x.
I resolved this by using padding as a simple fix. While padding may affect the performance of the generated images, it is less problematic than having Controlllite not function at all. Here is the code:
```
if cx.shape[1] < x.shape[1]:
cx_pad = torch.zeros(cx.shape[0], x.shape[1]-cx.shape[1], cx.shape[2], dtype=cx.dtype, device=cx.device)
cx = torch.cat([cx, cx_pad], dim=1)
```

### Steps to reproduce the problem

1.5x upscaling of a 1248x1248 image using the controllllite model for img2img.

### What should have happened?

controllllite should have worked.

### What browsers do you use to access the UI ?

_No response_

### Sysinfo

[sysinfo-2024-05-29-09-21.json](https://github.com/lllyasviel/stable-diffusion-webui-forge/files/15482072/sysinfo-2024-05-29-09-21.json)

### Console logs

```Shell
https://pastebin.com/gxT0wyCL
```

### Additional information

invoking it from [EasySdxlWebUi](https://github.com/Zuntan03/EasySdxlWebUi)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in extensions-builtin/sd_forge_controlllite/lib_controllllite/lib_controllllite.py around line 234, where torch.cat is reported to fail. Reproduce with img2img using the controllllite model and 1.5x upscaling of a 1248x1248 image, then compare the cx and x tensor shapes. Done means controllllite completes generation for this case without the reported error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.