lllyasviel / lllyasviel/ControlNet

depth_leres++ can't run in M1 Pro

Open
#484 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
34.1k
Forks
3k
PR merge metrics
No merged PRs in 30d

Description

A1111 version: v1.5
Controlnet version: v1.1.233
torch version: v2.0.1
torchversion version: 0.15.2
MetalPerformanceShaders version: v1.0

2023-07-27 10:32:50,095 - ControlNet - INFO - Preview Resolution = 512
Traceback (most recent call last):
File "/Users/simplemin/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/routes.py", line 422, in run_predict
output = await app.get_blocks().process_api(
File "/Users/simplemin/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1323, in process_api
result = await self.call_function(
File "/Users/simplemin/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1051, in call_function
prediction = await anyio.to_thread.run_sync(
File "/Users/simplemin/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/Users/simplemin/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "/Users/simplemin/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
File "/Users/simplemin/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet_ui/controlnet_ui_group.py", line 606, in run_annotator
result, is_image = preprocessor(
File "/Users/simplemin/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/utils.py", line 79, in decorated_func
return cached_func(*args, **kwargs)
File "/Users/simplemin/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/utils.py", line 67, in cached_func
return func(*args, **kwargs)
File "/Users/simplemin/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/global_state.py", line 35, in unified_preprocessor
return preprocessor_modules[preprocessor_name](*args, **kwargs)
File "/Users/simplemin/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/processor.py", line 211, in leres
result = model_leres(img, thr_a, thr_b, boost=boost)
File "/Users/simplemin/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/leres/init.py", line 79, in apply_leres
depth = estimateboost(input_image, model, 0, pix2pixmodel, max(width, height))
File "/Users/simplemin/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/leres/leres/depthmap.py", line 418, in estimateboost
whole_estimate = doubleestimate(img, net_receptive_field_size, whole_image_optimal_size, pix2pixsize, model, model_type, pix2pixmodel)
File "/Users/simplemin/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/leres/leres/depthmap.py", line 132, in doubleestimate
estimate1 = singleestimate(img, size1, model, net_type)
File "/Users/simplemin/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/leres/leres/depthmap.py", line 156, in singleestimate
return estimateleres(img, model, msize, msize)
File "/Users/simplemin/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/leres/leres/depthmap.py", line 43, in estimateleres
prediction = model.depth_model(img_torch)
File "/Users/simplemin/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/simplemin/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/leres/leres/multi_depth_model_woauxi.py", line 32, in forward
lateral_out = self.encoder_modules(x)
File "/Users/simplemin/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/simplemin/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/leres/leres/network_auxi.py", line 96, in forward
x = self.encoder(x) # 1/32, 1/16, 1/8, 1/4
File "/Users/simplemin/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/simplemin/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/leres/leres/Resnext_torch.py", line 223, in forward
return self._forward_impl(x)
File "/Users/simplemin/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/leres/leres/Resnext_torch.py", line 199, in _forward_impl
x = self.conv1(x)
File "/Users/simplemin/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/simplemin/stable-diffusion-webui/extensions-builtin/Lora/networks.py", line 371, in network_Conv2d_forward
return torch.nn.Conv2d_forward_before_network(self, input)
File "/Users/simplemin/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 463, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/Users/simplemin/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (MPSFloatType) and weight type (torch.FloatTensor) should be the same

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with annotator/leres/leres/depthmap.py and annotator/leres/leres/Resnext_torch.py, following the traceback into the first convolution in the MPS execution path. Reproduce depth_leres++ on an M1 Pro with the listed versions; done means the preprocessor completes without the MPSFloatType and torch.FloatTensor mismatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.