huggingface / huggingface/diffusers

[New Pipeline/Model] Add JoyAI-Echo multi-shot audio-video generation pipeline

Open
#13,909 0 comments 0 reactions 0 assignees View on GitHub
feature-request models pipelines
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

### Model/Pipeline/Scheduler description

**We are the JoyAI Team (JD.com)**, proposing the integration of JoyAI-Echo into Diffusers.

**JoyAI-Echo** is a unified framework for long-form audio-visual generation, designed to support minute-level video creation with synchronized audio, strong temporal consistency, and real-time interaction.

Key innovations:
- **Cross-modal audio-visual memory bank**: preserves character appearance and voice timbre across long sequences (up to minutes)
- **DMD-distilled few-step inference**: ~7.5× faster than baseline while improving alignment and visual quality
- **Joint audio-video generation**: a single pipeline produces synchronized video and audio
- **Multi-shot story generation**: generates coherent sequences of shots from prompt lists

The architecture builds on LTX-2 and adds the JoyAI-Echo DMD denoising schedule plus a paired audio-video memory bank for cross-shot consistency.

### Open source status

- [x] The model implementation is available.
- [x] The model weights are available (Only relevant if addition is not a scheduler).

### Provide useful links for the implementation

- GitHub Repository: https://github.com/jd-opensource/JoyAI-Echo
- HuggingFace Weights: https://huggingface.co/jdopensource/JoyAI-Echo
- Diffusers implementation PR: (will link after PR is created)

### Additional context

We (JoyAI Team) previously contributed JoyAI-Image-Edit to Diffusers (PR #13444, merged). This follows the same pattern — official team providing a complete, tested implementation.

Contributor guide

Open the contributing guide

Research direction

Start with the JoyAI-Echo repository and HuggingFace weights linked in the issue, then compare their implementation with Diffusers' existing LTX-2 support. The issue describes a multi-shot audio-video pipeline with memory-bank and DMD scheduling changes, but names no Diffusers files or tests. Done would mean a complete, tested integration and a linked implementation pull request.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
audio-video-rtc, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
40/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.