mudler / mudler/LocalAI

Voice Library Improvements

Open
#11,617 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
49.2k
Forks
4.5k
Avg merge
1d 3m
Merged PRs (30d)
239

Description

Is your feature request related to a problem? Please describe.
audio.cpp backend recently added support for multiple reference inputs https://github.com/0xShug0/audio.cpp/pull/282 for fish audio, and the fish speech backend it already have support for this too https://docs.fish.audio/api-reference/sdk/javascript/api-reference#ttsrequest

I think the voice library should be updated instead of having "saved voices" it should have "saved personalities". These personalities should be able to hold multiple references audio and texts and for any backend that supports multiple audio's it should include all of them when generating the TTS

Describe the solution you'd like
I'd like to be able to manage a "personality" with multiple reference audio's and texts so the output at the end can be better.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the voice library and the existing Fish speech backend, then compare the multiple-reference behavior described in the audio.cpp pull request and Fish Audio TTS request documentation. Done means personalities can manage multiple reference audios and texts, and backends that support multiple audios use all of them during TTS generation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.