lllyasviel / lllyasviel/stable-diffusion-webui-forge
Video cannot be generated with animatediff
- Dominant language
- Python
- Stars
- 13k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
When "sd-forge-animatediff" is installed and used with controlnet openpose, 6 images and a skeleton are displayed and no video is generated.
If I generate the video using only the prompts without the controlnet, the video will be exactly as set up.
The version of the forge is commit hash 29be1da7cf2b5dccfc70fbdd33eb35c56a31ffb7.
Please help.
> Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf61.7.100
Duration: 00:00:01.94, start: 0.000000, bitrate: 496 kb/s
Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt470bg/bt470bg/smpte170m, progressive), 512x512 [SAR 1:1 DAR 1:1], 362 kb/s, 16 fps, 16 tbr, 16384 tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : Lavc61.19.100 libx264
Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
Metadata:
handler_name : SoundHandler
vendor_id : [0][0][0][0]
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
[vf#0:0 @ 00000206a9319b80] No information about the input framerate is available. Falling back to a default value of 25fps. Use the -r option if you want a different framerate.
Output #0, image2, to 'C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge\tmp\animatediff-frames\inshot_2-e6589af9\%09d.jpg':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf61.7.100
Stream #0:0(und): Video: mjpeg, yuv420p(pc, bt470bg/bt470bg/smpte170m, progressive), 512x512 [SAR 1:1 DAR 1:1], q=2-31, 200 kb/s, 25 fps, 25 tbn (default)
Metadata:
handler_name : VideoHandler
vendor_id : [0][0][0][0]
encoder : Lavc61.19.100 mjpeg
Side data:
cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: N/A
[out#0/image2 @ 00000206a8bdee80] video:314KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame= 6 fps=0.0 q=1.0 Lsize=N/A time=00:00:00.24 bitrate=N/A dup=2 drop=0 speed=3.92x
2024-11-09 12:00:16,934 - AnimateDiff - INFO - AnimateDiff + ControlNet will generate 6 frames.
*** Error running before_process: C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py
Traceback (most recent call last):
File "C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge\modules\scripts.py", line 795, in before_process
script.before_process(p, *script_args)
File "C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff.py", line 64, in before_process
params.prompt_scheduler = AnimateDiffPromptSchedule(p, params)
File "C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff_prompt.py", line 15, in __init__
self.parse_prompt(p, params)
File "C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge\extensions\sd-forge-animatediff\scripts\animatediff_prompt.py", line 59, in parse_prompt
assert int(frame) < params.video_length, \
AssertionError: invalid prompt travel frame number: 12 >= number of frames (6)
---
2024-11-09 12:00:16,937 - ControlNet - INFO - ControlNet Input Mode: InputMode.BATCH
2024-11-09 12:00:16,938 - ControlNet - INFO - Try to read image: C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge/tmp/animatediff-frames\inshot_2-e6589af9\000000001.jpg
2024-11-09 12:00:16,946 - ControlNet - INFO - Try to read image: C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge/tmp/animatediff-frames\inshot_2-e6589af9\000000002.jpg
2024-11-09 12:00:16,951 - ControlNet - INFO - Try to read image: C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge/tmp/animatediff-frames\inshot_2-e6589af9\000000003.jpg
2024-11-09 12:00:16,955 - ControlNet - INFO - Try to read image: C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge/tmp/animatediff-frames\inshot_2-e6589af9\000000004.jpg
2024-11-09 12:00:16,958 - ControlNet - INFO - Try to read image: C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge/tmp/animatediff-frames\inshot_2-e6589af9\000000005.jpg
2024-11-09 12:00:16,962 - ControlNet - INFO - Try to read image: C:\StableDiffusion WebUI Forge\stable-diffusion-webui-forge/tmp/animatediff-frames\inshot_2-e6589af9\000000006.jpg
0%| | 0/6 [00:00
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing AnimateDiff with ControlNet openpose using the reported Forge commit and six-frame input. Read extensions/sd-forge-animatediff/scripts/animatediff.py, animatediff_prompt.py, and animatediff_infv2v.py at the traceback locations, then trace how the prompt scheduler is initialized before cfg_denoiser callbacks. Done means this combination generates a video without the reported callback errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100