agentscope-ai / agentscope-ai/agentscope

feat(tts): support voice cloning and local TTS

Offen
#2,190 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
31.5k
Forks
3.5k
Ø Merge
1 T. 23 Std.
Gemergte PRs (30 T.)
95

Beschreibung

## Background

AgentScope already provides a unified `TTSModelBase` abstraction and supports multiple API-based TTS providers. However, two important capabilities are still missing:

1. An end-to-end API-based voice cloning workflow that creates and manages custom voices from reference audio.
2. Local TTS inference for offline and privacy-sensitive deployments, including local voice cloning.

This issue tracks the overall implementation of voice cloning and local TTS support in AgentScope.

## Scope

- Support API-based voice cloning:
- Create a custom voice from reference audio.
- Manage provider-side custom voices where supported.
- Use the returned voice ID directly with existing TTS models.
- Support local TTS inference:
- Provide a lightweight preset-voice engine.
- Provide at least one zero-shot local voice-cloning engine.
- Support CPU, CUDA, and MPS where supported by the upstream engine.
- Expose consistent capability metadata, such as `voice_cloning`, through TTS model cards.
- Keep local model dependencies optional and lazily loaded.
- Add unit tests, documentation, and runnable examples.
- Document privacy, consent, and responsible-use requirements for reference audio.

## Out of Scope

- STT or full realtime speech-to-speech interaction.
- Training or fine-tuning a new speaker model.
- Hosting or distributing cloned voices as a standalone service.

## Definition of Done

- Both sub-issues are completed.
- API and local voices can be selected through the existing TTS configuration flow.
- Voice-cloning capabilities are discoverable from model metadata.
- Existing TTS providers and `TTSMiddleware` continue to work without regression.
- Documentation includes API-based and fully local examples.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.