Attention is all you need. Consistent video animation (simplified version of RERENDER A VIDEO: ZERO-SHOT TEXT-GUIDED VIDEO-TO-VIDEO TRANSLATION)

https://anonymous-31415926.github.io/ - looked at this paper (RERENDER A VIDEO: ZERO-SHOT TEXT-GUIDED VIDEO-TO-VIDEO TRANSLATION)

The bottom line is that the input to each frame is the previous frame and a certain crossframe with textures, colors and shapes: The framework includes two parts: key frame translation and full video translation. The first part uses an adapted diffusion model to generate key frames, with hierarchical cross-frame constraints applied to enforce coherence in shapes, textures and colors.

In short, I didn’t fucking understand, except that if you get confused, you can get an awesomely consistent video.

I set up a deforum, a bunch of controlnet models and started experimenting. First I came up with the following combination of models:
- reference only
- reference attan
- open pose/face
- softage/head
All 4 through a loopback (pointing to the previous frame)

Generation speed dropped from 6 iterations/sec to 1
However, the authors of the paper write that they also need 16 video memory

But the generations were still inconsistent. Then I threw a reference on the very first frame:
- reference only on the first frame

And then a miracle happened. The girl stopped mutating like crazy

https://github.com/deforum-art/deforum-stable-diffusion/assets/417177/79428ba9-c9d0-4dcd-b3f7-402889ff3071

I decided to test the theory that in fact it is not the number of models, but the quality. You have to give two signals with a reference (slightly different) so that the model understands what frames unite. I threw out Oppose and Softage and Reference Atan and left only 2 controllers, both on the reference. The first takes the last frame. The second takes the very first frame. And the combination gave even more consistent results. And by the way, the speed has doubled to 2 iterations per second, because 2 models instead of five:

https://github.com/deforum-art/deforum-stable-diffusion/assets/417177/b3d16ab9-1e89-4318-a05c-1a30292c2fe5

Actually, I think it's even simpler than that. You need to give two frames, the last and penultimate, so that the model rolls animation over them. This will not have to adjust each frame by hand, and will smooth it out more properly, because the difference between the first and last frame increases with the distance covered by the animation. In general, you get such a ZERO-SHOT VIDEO-TO-VIDEO of shit and sticks, just as we like. Configure deforum for those who want to repeat it attached
[deforum_settings_cntrl (1).txt](https://github.com/deforum-art/deforum-stable-diffusion/files/11749466/deforum_settings_cntrl.1.txt)

And to understand the difference, the version without the controllet. When the model starts to dance (zero zero knocked out)

https://github.com/deforum-art/deforum-stable-diffusion/assets/417177/7489f9a3-4f44-4363-8d3e-ecc823687c6e

I haven't tested video2video by mask and so on, but I'm sure it should work. So, attention is all you need, just add two frames instead of one

https://dump.video/i/B1PLxztF.mp4