huggingface / huggingface/diffusers

AnimateDiff SparseCtrl RGB does not work as expected

Đang mở
#9,508 9 bình luận 0 reaction 0 người được giao Xem trên GitHub
advanced bug contributions-welcome help wanted stale
Ngôn ngữ chính
Python
Star
34.5k
Fork
7.3k
Merge trung bình
3 ngày 3 giờ
Pull request đã merge (30 ngày)
91

Mô tả

Relevant comments are [this](https://github.com/huggingface/diffusers/pull/8897#issuecomment-2255416318) and [this](https://github.com/huggingface/diffusers/pull/8897#issuecomment-2255478105).

AnimateDiff SparseCtrl RGB does not work similar to other implementations and cannot replicate their outputs. This makes me believe that there is something incorrect with our SparseControlNet or MotionAdapter implementation.

When comparing the results of the [original](https://github.com/guoyww/AnimateDiff)/[Comfy](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved) implementation to Diffusers implementation, one can notice that if an image is used with an unrelated prompt, the Diffusers implementation ignores the image and just follows the prompt whereas the other implementations try to incorporate both.

Since the original and Comfy implementations produce this behaviour consistently, this seems more like a problem with Diffusers implementation. However, I've not been able to spot differences in implementation just by comparing the code visually. I also tried matching outputs layerwise and it seemed to be alright (although I didn't investigate this as deeply as I should have due to other priorities).

If someone from the community actively following/using the AnimateDiff implementations can help determine the cause of this bug, it would be really awesome and helpful.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Start with the linked comments on PR 8897 and reproduce the AnimateDiff SparseCtrl RGB comparison using the original and Comfy implementations as references. Investigate the Diffusers SparseControlNet and MotionAdapter behavior with an unrelated prompt, and compare the conditioning results. Done means identifying and correcting the cause so the image influences the output rather than being ignored.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python, pytorch
Lĩnh vực
computer-vision, machine-learning
Loại issue
Lỗi
Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức độ hoạt động
Đình trệ
Độ rõ ràng
Cần làm rõ
Mức phù hợp với người mới
35/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.