deforum / deforum/sd-webui-deforum

Use Inpainting for Consistency on Single Prompt Animations

Open
#120 3 comments 0 reactions 0 assignees View on GitHub
2D 3D duplicate enhancement help wanted
Dominant language
Python
Stars
2.9k
Forks
407
PR merge metrics
No merged PRs in 30d

Description

I believe that Deforum videos of a single prompt would be more consistent if there was a mode that instead of generating a new frame, inpainted the missing details. For example for a 2D camera that is zooming in what it would do is keep all of the image the exact same (maybe with noise scheduling settings) except for the center which it would inpaint at a strength proportional to the zooming speed, so that the zoomed area would be in proper resolution. For a 3D rotating camera as another example, it would find the areas that MiDaS shows weren't in view last frame.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how single-prompt animations generate each frame and how the 2D and 3D camera paths expose new areas. Inspect the existing MiDaS-related depth handling for the rotating-camera case. Done means a new mode can preserve existing image regions while inpainting newly exposed details with strength tied to zoom or camera movement.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-graphics, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.