huggingface / huggingface/controlnet_aux

Errors and Solutions Encountered When Using SamDetector.

Open
#91 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
497
Forks
104
PR merge metrics
No merged PRs in 30d

Description

I encountered an error while using SamDetector, and the error message is as follows:
```
Traceback (most recent call last):
File "c:\PythonProgram\diffweb\test_seg.py", line 12, in <module>
processed = sam(image)
File "C:\PythonProgram\diffweb\lib\site-packages\controlnet_aux\segment_anything\__init__.py", line 76, in __call__
masks = self.mask_generator.generate(input_image)
// ... Omitted intermediate error information ...
File "C:\PythonProgram\diffweb\lib\site-packages\controlnet_aux\segment_anything\modeling\tiny_vit_sam.py", line 274, in forward
(q @ k.transpose(-2, -1)) * self.scale
```
`RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!`
After some investigation, I found that this issue was caused by some parts of the model using GPU (cuda:0) for computation, while other parts were using the CPU. This led to a device mismatch problem when executing the attention mechanism.
To solve this issue, I added the following code before the line:
```
attn = (
(q @ k.transpose(-2, -1)) * self.scale
+
(self.attention_biases[:, self.attention_bias_idxs] if self.training else self.ab)
)
```
`self.ab = self.ab.to('cuda:0')
`By doing this, all tensors in the model are computed on the GPU, thereby avoiding the device mismatch problem.
I hope this solution will be helpful to others.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.