lllyasviel / lllyasviel/stable-diffusion-webui-forge

Interrogate Clip error - Expected all tensors to be on the same device

Open
#2,306 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

I am getting the following error when trying to use Interrogate.

```
[Unload] Trying to free 1024.00 MB for cuda:0 with 2 models keep loaded ... Current free memory is 2864.60 MB ... Done.
*** Error interrogating
Traceback (most recent call last):
File "D:\webui_forge_cu121_torch231\webui\modules\interrogate.py", line 191, in interrogate
caption = self.generate_caption(pil_image)
File "D:\webui_forge_cu121_torch231\webui\modules\interrogate.py", line 181, in generate_caption
caption = self.blip_model.generate(gpu_image, sample=False, num_beams=int(shared.opts.interrogate_clip_num_beams), min_length=int(shared.opts.interrogate_clip_min_length), max_length=shared.opts.interrogate_clip_max_length)
File "D:\webui_forge_cu121_torch231\webui\repositories\BLIP\models\blip.py", line 129, in generate
image_embeds = self.visual_encoder(image)
File "D:\webui_forge_cu121_torch231\system\python\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "D:\webui_forge_cu121_torch231\system\python\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "D:\webui_forge_cu121_torch231\webui\repositories\BLIP\models\vit.py", line 185, in forward
x = torch.cat((cls_tokens, x), dim=1)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument tensors in method wrapper_CUDA_cat)

```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with modules/interrogate.py at interrogate and generate_caption, then follow the call into repositories/BLIP/models/blip.py and vit.py. Reproduce Interrogate with the reported setup and inspect where the image and model tensors are placed. Done means the caption generation no longer fails with a CPU/cuda:0 device mismatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.