lllyasviel / lllyasviel/ControlNet

Allow the models to be chained in a sequence. (How to solve the animation issues)

Open
#122 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
34.1k
Forks
3k
PR merge metrics
No merged PRs in 30d

Description

It's possible to send the result of a model to a secondary and tertiary model pass, where each one has a diferent picture?

That way I could render a picture as a 3D animation in blender and then do img2img and then simply use it initially with the open pose model, then later use a segmentation pass with some mask render from a 3D render using flat unshaded mask animations, then give it later to a model that can take some concept art frame made with a normal text2image prompt and some lora, and use some edge detection model to guide a final polish step.
Maybe as a final step, check between frames using some cache, where each step of the difusion process, is done to all the pictures as a batch system where a frame influences the next frame and maybe some middle value between the previous and next smooth blending operation is applied between the frames.
Then all the animation is done in batch, until all the frames are basically generated in a batch, similar to training with epochs.
That would be my initial idea to solve the animation temporal coherence issue.

Sorry if my english is poor BTW.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or implementation entry points are named. Start by reviewing the proposed model chaining, multi-stage conditioning, and batch animation workflow; this issue would be done only after the project defines a concrete scope and verifies the resulting frames maintain temporal coherence.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.