lllyasviel / lllyasviel/FramePack
Proposal for a Video Merging Workflow
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
Thank you for the excellent solution.
As a method for generating more dynamic and controllable videos, I propose a video merging workflow. (Note: I haven't personally tested this workflow, so I'm not sure if it will work as intended.)
The video merging workflow is essentially the same as taking the final frame of a video generated by framepack, feeding it back into framepack to generate the next segment of the video, and then combining the two segments using video editing software.
The idea is to give framepack any number of sections, each with its own prompt.
Example:
Section 1: Jumping (1 sec)
Section 2: Wings sprout (2 sec)
Section 3: Flying through the sky (2 sec)
By generating a video for each section and then combining all the segments at the end, I believe we can control the flow of complex video narratives.
Additionally, since framepack proposes the final frame of the video first, I think it's possible to have it suggest the final frame of each section before generating the actual video for that section.
This allows for quick verification of whether the video sequence matches your intended flow.
If you're not satisfied, you can regenerate just the final frame for that section.
Once all the final frames are approved, you proceed to generate the videos for each section.
If you're not happy with a particular section's video, you can regenerate it as well.
Sections that are fine can be kept as is.
After completing all sections, you merge them into a single video.
That is the overall process.
While this can already be replicated manually using video editing software, I believe having an integrated batch generation workflow would make the process far more efficient.
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files, tests, or entry points; start by locating the current FramePack generation flow and how generated video segments are saved. Define the batch workflow and its completion criteria around per-section prompts, final-frame review, selective regeneration, and final merging.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100