lllyasviel / lllyasviel/FramePack

FramePack F1: The longer the video, the much less forward movement of the object

Open
#552 26 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
17.3k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

For some reason, if the video (960x360, 30fps -> upscale 2x to 1920x720, as it has to be exactly this resolution and FPS) is longer than lets say 2 seconds, then **THE FORWARD MOVEMENT IS LESS AND LESS VISIBLE** - if it is those 2 seconds, the movement is immediate right from the very first frame.
https://github.com/user-attachments/assets/4bef5b06-d3f2-4bf5-80e8-016172a39445

So if I make 15 seconds video, then the movement of the person starts extremely slowly (basically no forward movement) at the beginning and got faster towards the end - there simply is no consistency in the speed of the movement at all.
https://github.com/user-attachments/assets/8ebafbe9-27a7-474d-8517-3bb32eb5d7b1

But if I dare to make it really long, like in this case 37 seconds, then the person just kind of shaking in the place and only the last maybe 3 seconds it moves forward, this is really bad IMHO.
https://github.com/user-attachments/assets/1bbf035a-48cd-43ce-86c4-611ceafcdf8e

I already saw a reply from the Developer to some other user about basically the same issue where he says it basically is the problem of the prompt, where you have to have some sort of infinitive statements. So after discussion and help of ChatGPT I went from this...

```
A confident man in a black leather outfit wearing a prescription glasses with thick black frame walks steadily towards the camera in the high noon thru a cyberpunk city street.

The scene is illuminated by cool blue and pink neon lights, creating a futuristic city ambiance.

The man's expression is serious yet approachable as he looks straight ahead.

His body moves gracefully and expressively, emphasizing his face expression with natural gestures.

The background features open cyberpunk city street and glowing rectangular panels, enhancing the night setting.

It is day scene.

The overall mood is dynamic and engaging, capturing the man's charisma and the immersive environment.
```

...to this:

```
A confident man in a black leather outfit wearing prescription glasses with thick black frame is repeatedly, endlessly, continuously walking toward the camera again and again without stopping, from the beginning to the end of the video, starting at full speed from the very first frame, with a steady, unwavering pace through a cyberpunk city street in the high noon.

The scene is constantly illuminated by cool blue and pink neon lights, keeping a futuristic city ambiance without change.

The man's expression remains serious yet approachable as he looks straight ahead the whole time.

His body moves nonstop in a flowing, graceful, expressive manner, emphasizing his face expression with natural gestures that never pause or fade, initiated instantly at full intensity from the start.

The background remains open with cyberpunk elements and glowing rectangular panels, maintaining the night illusion.

It is a day scene.

The overall mood stays dynamic and engaging, with no interruption of motion, constantly capturing the man's charisma and the immersive city setting.
```

But the result was - as you can see - absolutely the same: less and less forward movement or even no movement for long videos except the last few seconds.

Can someone really explain to me why is this happening, and how to actually make FramePack F1 respect the prompt and have the long videos with the forward movement of the person right from the very beginning, please?

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file or test is identified in the report. Start by reproducing the behavior at 960x360 and 30fps, upscaled to 1920x720, across the 2-, 15-, and 37-second examples using both prompts. Compare motion from the first frame through the final frames; done means identifying the cause and establishing consistent forward movement for long videos.

Written by the indexing model from the issue text.

Assessment

Tech stack
machine-learning, python
Domain
ai, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.