lllyasviel / lllyasviel/stable-diffusion-webui-forge

Video cannot be generated with animatediff

Open
#2,290 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
13k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

When "sd-forge-animatediff" is installed and used with controlnet openpose, 6 images and a skeleton are displayed and no video is generated.
If I generate the video using only the prompts without the controlnet, the video will be exactly as set up.
The version of the forge is commit hash 29be1da7cf2b5dccfc70fbdd33eb35c56a31ffb7.
Please help.

> Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf61.7.100
Duration: 00:00:01.94, start: 0.000000, bitrate: 496 kb/s
Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt470bg/bt470bg/smpte170m, progressive), 512x512 [SAR 1:1 DAR 1:1], 362 kb/s, 16 fps, 16 tbr, 16384 tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : Lavc61.19.100 libx264
Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
[vf#0:0 @ 00000206a9319b80] No information about the input framerate is available. Falling back to a default value of 25fps. Use the -r option if you want a different framerate.
Output #0, image2, to 'C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge\tmp\animatediff-frames\inshot_2-e6589af9\%09d.jpg':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf61.7.100
Stream #0:0(und): Video: mjpeg, yuv420p(pc, bt470bg/bt470bg/smpte170m, progressive), 512x512 [SAR 1:1 DAR 1:1], q=2-31, 200 kb/s, 25 fps, 25 tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : Lavc61.19.100 mjpeg
Side data:
cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
[out#0/image2 @ 00000206a8bdee80] video:314KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame= 6 fps=0.0 q=1.0 Lsize=N/A time=00:00:00.24 bitrate=N/A dup=2 drop=0 speed=3.92x
2024-11-09 12:00:16,934 - AnimateDiff - INFO - AnimateDiff + ControlNet will generate 6 frames.
*** Error running before_process: C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py
Traceback (most recent call last):
File "C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge\modules\scripts.py", line 795, in before_process
script.before_process(p, *script_args)
File "C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py", line 64, in before_process
params.prompt_scheduler = AnimateDiffPromptSchedule(p, params)
File "C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff_prompt.py", line 15, in __init__
self.parse_prompt(p, params)
File "C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff_prompt.py", line 59, in parse_prompt
assert int(frame) < params.video_length, \
AssertionError: invalid prompt travel frame number: 12 >= number of frames (6)

---
2024-11-09 12:00:16,937 - ControlNet - INFO - ControlNet Input Mode: InputMode.BATCH
2024-11-09 12:00:16,938 - ControlNet - INFO - Try to read image: C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge/tmp/animatediff-frames\inshot_2-e6589af9\000000001.jpg
2024-11-09 12:00:16,946 - ControlNet - INFO - Try to read image: C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge/tmp/animatediff-frames\inshot_2-e6589af9\000000002.jpg
2024-11-09 12:00:16,951 - ControlNet - INFO - Try to read image: C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge/tmp/animatediff-frames\inshot_2-e6589af9\000000003.jpg
2024-11-09 12:00:16,955 - ControlNet - INFO - Try to read image: C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge/tmp/animatediff-frames\inshot_2-e6589af9\000000004.jpg
2024-11-09 12:00:16,958 - ControlNet - INFO - Try to read image: C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge/tmp/animatediff-frames\inshot_2-e6589af9\000000005.jpg
2024-11-09 12:00:16,962 - ControlNet - INFO - Try to read image: C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge/tmp/animatediff-frames\inshot_2-e6589af9\000000006.jpg
0%| | 0/6 [00:00

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing AnimateDiff with ControlNet openpose using the reported Forge commit and six-frame input. Read extensions/sd-forge-animatediff/scripts/animatediff.py, animatediff_prompt.py, and animatediff_infv2v.py at the traceback locations, then trace how the prompt scheduler is initialized before cfg_denoiser callbacks. Done means this combination generates a video without the reported callback errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.