mudler / mudler/LocalAI

audio-cpp using a fish-audio doesn't work with referance audio/text

Open
#11,626 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/backends bug unconfirmed
Dominant language
Go
Stars
49.2k
Forks
4.5k
Avg merge
1d 3m
Merged PRs (30d)
239

Description

LocalAI version:
localai/localai:master-gpu-hipblas
sha256:f3ff922777a09ec7f63093df7482ebb15d3f94c0f8611d49143fe455b4f7144d

Environment, CPU architecture, OS, and Version:
I'm doing this on unraid 7.3, using portainer.
AMD Ryzen 9 5900X
Radeon AI PRO R9700
Local AI is setup using distribution mode with nodes.

Describe the bug
When I try and use TTS with a voice that was a referance audio with text using audio-cpp backend with a fish-audio family model I get this error Error: rpc error: code = Internal desc = Fish Audio prepare with inline reference audio requires reference_text option

To Reproduce
My model yaml looks like this.

name: audio-cpp-chatterbox
backend: audio-cpp

flash_attention: true

known_usecases:
    - tts
    - audio_transform

tts:
    voice_cloning: true

parameters:
    model: audio-cpp/Fish-Audio-S2-Pro-GGUF/fish-audio-s2-pro-q8_0.gguf

the model is downloaded from https://huggingface.co/audio-cpp/audio.cpp-gguf/tree/main/Fish-Audio-S2-Pro-GGUF

Inside voice library I click "Use in Text to Speech" and then when I try and generate anything it will error Error: rpc error: code = Internal desc = Fish Audio prepare with inline reference audio requires reference_text option

Expected behavior
to be able to use different models with audio-cpp like the fish-audio family of models

Logs
node-backend-logs-audio-cpp-chatterbox-2026-08-20.json

Additional context
There is another PR that is touching ref audio/text inside audio-cpp https://github.com/mudler/LocalAI/pull/11620 if changes are done from master then there might be merge conflicts.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the audio-cpp backend's handling of inline reference audio and reference text, using the provided model YAML and logs to reproduce the Fish Audio error. Review PR #11620 because it touches the same reference-audio/text area; done means the voice-library TTS flow works with Fish Audio models without requiring an unavailable reference_text option.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, go
Domain
audio-video-rtc, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.