deforum / deforum/sd-webui-deforum

[Bug]: Video masks won't work if resolution set to non-divisible by 64 values

Open
#678 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.9k
Forks
407
PR merge metrics
No merged PRs in 30d

Description

### Have you read the latest version of the FAQ?

- [X] I have visited the FAQ page right now and my issue is not present there

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

### Are you using the latest version of the Deforum extension?

- [X] I have Deforum updated to the lastest version and I still have the issue.

### What happened?

Deforum throws an exception if I manually set the W and H values in the Run tab and try to use a video mask.

### Steps to reproduce the problem

1. It is assumed that the video and video mask being used are of the exact same dimensions. In my case it was 852 x 480
2. Set W & H manually to the width and height of your input video.
3. Set Noise type to Uniform (bug happens with Uniform and Perlin) because Uniform lets you use non-64 divisible W & H
4. Set animation mode to Video Input
5. Specify input video in Init tab and check Use Mask Video
6. Specify path to mask video
7. Set up mask in Mask Init
8. Leave default prompts
9. Hit Generate

### What should have happened?

Video should have been generated, respecting the video mask specified.

### WebUI and Deforum extension Commit IDs

webui commit id - https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/5ab7f213bec2f816f9c5644becb32eb72c8ffb89
deforum exten commit id - e392e0ac

### Torch version

2.0.0+cu118

### On which platform are you launching the webui with the extension?

Local PC setup (Windows)

### Deforum settings

[mask_bug_settings.txt](https://github.com/deforum-art/sd-webui-deforum/files/11413064/mask_bug_settings.txt)

### Webui core settings

[ui-config.json.txt](https://github.com/deforum-art/sd-webui-deforum/files/11413071/ui-config.json.txt)

### Console logs

```Shell
venv "D:\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Commit hash: 5ab7f213bec2f816f9c5644becb32eb72c8ffb89
Installing requirements

Installing requirements for TemporalKit extension

Launching Web UI with arguments: --opt-sdp-no-mem-attention --ckpt-dir D:\sd-models\Stable-diffusion
No module 'xformers'. Proceeding without it.
ControlNet v1.1.134
ControlNet v1.1.134
Loading weights [e72b33b7a2] from D:\sd-models\Stable-diffusion\cheeseDaddysLandscapes_20.safetensors
Creating model from config: D:\stable-diffusion-webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading VAE weights specified in settings: D:\stable-diffusion-webui\models\VAE\vae-ft-mse-840000-ema-pruned.safetensors
Applying scaled dot product cross attention optimization (without memory efficient attention).
Textual inversion embeddings loaded(1): EasyNegative
Textual inversion embeddings skipped(3): nartfixer, nfixer, nrealfixer
Model loaded in 2.4s (load weights from disk: 0.1s, create model: 0.2s, apply weights to model: 0.5s, apply half(): 0.4s, move model to device: 0.4s, load textual inversion embeddings: 0.7s).
*Deforum ControlNet support: enabled*
Running on local URL: http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 7.5s (import torch: 0.9s, import gradio: 0.6s, import ldm: 0.3s, other imports: 0.5s, load scripts: 1.1s, load SD checkpoint: 2.8s, create ui: 1.1s, gradio launch: 0.1s).
reading custom settings from mask_bug_settings.txt
Deforum extension for auto1111 webui, v2.4b
Git commit: e392e0ac (Fri May 5 13:51:02 2023)
Exporting Video Frames (1 every 2) frames to D:\a1111_outputs\img2img-images\Deforum_masktest_20230506191610\inputframes...
Trying to extract frames from video with input FPS of 30.0. Please wait patiently.
Extracted 180 frames from video in 0.23 seconds!
Loading 90 input frames from D:\a1111_outputs\img2img-images\Deforum_masktest_20230506191610\inputframes and saving video frames to D:\a1111_outputs\img2img-images\Deforum_masktest_20230506191610
Exporting Video Frames (1 every 2) frames to D:\a1111_outputs\img2img-images\Deforum_masktest_20230506191610\maskframes...
Trying to extract frames from video with input FPS of 30.0. Please wait patiently.
Extracted 179 frames from video in 0.19 seconds!
Saving animation frames to:
D:\a1111_outputs\img2img-images\Deforum_masktest_20230506191610
Animation frame: 0/90
Seed: 951618567
Prompt: tiny cute swamp bunny, highly detailed, intricate, ultra hd, sharp photo, crepuscular rays, in focus, by tomasz alen kopera
Neg Prompt: nsfw, nude nsfw, nude nsfw, nude
Using video init frame D:\a1111_outputs\img2img-images\Deforum_masktest_20230506191610\inputframes\mask_test_orig000000001.jpg
use_alpha_as_mask==True: Using the alpha channel from the init image as a mask, but the alpha channel is blank.
ignoring alpha as mask.
mask is blank. ignoring mask
╭─────┬───┬───────╮
│Steps│CFG│Denoise│
├─────┼───┼───────┤
│ 25 │7.0│ 0.35 │
╰─────┴───┴───────╯
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:01<00:00, 6.59it/s]
Deforum progress: 1%|█ | 9/810 [00:00<01:00, 13.16it/s]
*START OF TRACEBACK*
Traceback (most recent call last):
File "D:\stable-diffusion-webui\extensions\deforum-for-automatic1111-webui\scripts\deforum.py", line 113, in run_deforum
render_input_video(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root.animation_prompts, root)#TODO: prettify code
File "D:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts\deforum_helpers\render_modes.py", line 53, in render_input_video
render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, animation_prompts, root)
File "D:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts\deforum_helpers\render.py", line 559, in render_animation
image = do_overlay_mask(args, anim_args, image, frame_idx)
File "D:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts\deforum_helpers\masks.py", line 31, in do_overlay_mask
img = Image.composite(img, current_frame, current_mask)
File "D:\stable-diffusion-webui\venv\lib\site-packages\PIL\Image.py", line 3341, in composite
image.paste(image1, None, mask)
File "D:\stable-diffusion-webui\venv\lib\site-packages\PIL\Image.py", line 1731, in paste
self.im.paste(im, box, mask.im)
ValueError: images do not match
*END OF TRACEBACK*

User friendly error message: ead of \.mages do not match. Check your schedules/ init values please. Also make sure you don't have a backwards slash in any of your PATHs - use / insted
```

### Additional information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in scripts/deforum_helpers/masks.py at do_overlay_mask and trace its call from render.py:559 during render_animation. Reproduce the issue with the attached settings and 852x480 input and mask videos. Done means video-mask generation completes without the PIL “images do not match” error and respects the specified mask.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, computer-vision
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.