Control net for video conversion

Nobody has claimed this yet.

I have observed that there are many videos using SD for animation (such as Rock-Paper-Scissors) that suffer from flickering issues, meaning there are inconsistencies in details between frames. I wonder if it's possible to train a control net to directly address this problem. For example:

During the training phase, the input control condition consists of k+1 consecutive images, where the first k images are extracted from the video's n-k to n-1 frames, and the last image is obtained by preprocessing the nth frame from the video (using tools like OpenPose or HED). The training objective is to restore the original nth frame image.

In the inference phase, the input ontrocl condition's first k images are generated by SD from the n-k to n-1 frames of the video, while the last image is preprocessed from the nth frame of the original video.

Alternatively, we could use a double control net, where one net is dedicated to inputting the previous k frames to control the details.

I hope that this approach can help resolve the flickering issue in SD-generated videos.

The issue names no files, tests, or entry points. Start by reviewing the repository's ControlNet training and inference entry points, then determine whether the proposed temporal-video conditioning fits the existing design. Done would require an agreed implementation scope and validation that generated video details remain consistent across frames.

Written by the indexing model from the issue text.