lllyasviel / lllyasviel/ControlNet

Complex types are unsupported on MPS

Open
#450 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
34.1k
Forks
3k
PR merge metrics
No merged PRs in 30d

Description

I'm trying to use inpaint+lama and get the following error during preview:

````python
Traceback (most recent call last):
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/gradio/routes.py", line 422, in run_predict
output = await app.get_blocks().process_api(
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1323, in process_api
result = await self.call_function(
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1051, in call_function
prediction = await anyio.to_thread.run_sync(
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
File "/Users/mariodian/Source/AI/automatic/extensions-builtin/sd-webui-controlnet/scripts/controlnet_ui/controlnet_ui_group.py", line 636, in run_annotator
result, is_image = preprocessor(
File "/Users/mariodian/Source/AI/automatic/extensions-builtin/sd-webui-controlnet/scripts/utils.py", line 76, in decorated_func
return cached_func(*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/extensions-builtin/sd-webui-controlnet/scripts/utils.py", line 64, in cached_func
return func(*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/extensions-builtin/sd-webui-controlnet/scripts/global_state.py", line 35, in unified_preprocessor
return preprocessor_modules[preprocessor_name](*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/extensions-builtin/sd-webui-controlnet/scripts/processor.py", line 489, in lama_inpaint
prd_color = model_lama(img_res)
File "/Users/mariodian/Source/AI/automatic/extensions-builtin/sd-webui-controlnet/annotator/lama/__init__.py", line 54, in __call__
result = self.model(image_feed)[0]
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/extensions-builtin/sd-webui-controlnet/annotator/lama/saicinpainting/training/modules/ffc.py", line 419, in forward
return self.model(input)
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/extensions-builtin/sd-webui-controlnet/annotator/lama/saicinpainting/training/modules/ffc.py", line 337, in forward
x_l, x_g = self.conv1((x_l, x_g))
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/extensions-builtin/sd-webui-controlnet/annotator/lama/saicinpainting/training/modules/ffc.py", line 304, in forward
x_l, x_g = self.ffc(x)
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/extensions-builtin/sd-webui-controlnet/annotator/lama/saicinpainting/training/modules/ffc.py", line 275, in forward
out_xg = self.convl2g(x_l) * l2g_gate + self.convg2g(x_g)
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/extensions-builtin/sd-webui-controlnet/annotator/lama/saicinpainting/training/modules/ffc.py", line 198, in forward
output = self.fu(x)
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1502, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/mariodian/Source/AI/automatic/extensions-builtin/sd-webui-controlnet/annotator/lama/saicinpainting/training/modules/ffc.py", line 86, in forward
ffted = torch.fft.rfftn(x, dim=fft_dim, norm=self.fft_norm)
TypeError: Complex types are unsupported on MPS
````

and this error during image generation:

````
13:18:06-665653 ERROR Running script process: /Users/mariodian/Source/AI/automatic/extensions-builtin/sd-webui-controlnet/scripts/controlnet.py: TypeError
╭───────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────────╮
│ /Users/mariodian/Source/AI/automatic/modules/scripts.py:406 in process │
│ │
│ 405 │ │ │ │ args = p.per_script_args.get(script.title(), p.script_args[script.args_f │
│ ❱ 406 │ │ │ │ script.process(p, *args, **kwargs) │
│ 407 │ │ │ │ s.append(f'{script.title()}:{round(time.time()-t0, 2)}s') │
│ │
│ /Users/mariodian/Source/AI/automatic/extensions-builtin/sd-webui-controlnet/scripts/controlnet.py:829 in process │
│ │
│ 828 │ │ │ logger.debug(f"Use numpy seed {seed}.") │
│ ❱ 829 │ │ │ detected_map, is_image = preprocessor( │
│ 830 │ │ │ │ input_image, │
│ │
│ ... 24 frames hidden ... │
│ │
│ /Users/mariodian/Source/AI/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1511 in _call_impl │
│ │
│ 1510 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1511 │ │ │ return forward_call(*args, **kwargs) │
│ 1512 │ │ # Do not call functions when jit is used │
│ │
│ /Users/mariodian/Source/AI/automatic/extensions-builtin/sd-webui-controlnet/annotator/lama/saicinpainting/training/modules/ffc.py:86 in forward │
│ │
│ 85 │ │ fft_dim = (-3, -2, -1) if self.ffc3d else (-2, -1) │
│ ❱ 86 │ │ ffted = torch.fft.rfftn(x, dim=fft_dim, norm=self.fft_norm) │
│ 87 │ │ ffted = torch.stack((ffted.real, ffted.imag), dim=-1) │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: Complex types are unsupported on MPS
````

# Specs

HW: mac mini m2 pro, 32gb ram
OS: macOS ventura 13.4
SD webui: latest [vladmandic/automatic](https://github.com/vladmandic/automatic)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the inpaint+lama preview and image-generation paths on the reported MPS setup. Start at extensions-builtin/sd-webui-controlnet/annotator/lama/saicinpainting/training/modules/ffc.py:86, where torch.fft.rfftn raises the reported error, and trace how that path is reached from the ControlNet preprocessor. Done means both reported operations no longer fail with the MPS complex-type error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.