[bug] Improve the timing and quality of voice cloning

Open
#14,480 62 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
csharp
Domain
audio-video-rtc

Research direction

The issue does not name files, tests, or an entry point. Start by locating the dubbing TTS speech-processing and timing code, then review how sentence duration, speech speed, and clipped final words are handled; done should mean dubbing completes every sentence without cutting off its ending.

Written by the indexing model from the issue text.

Description

I'll start by saying that I've tried many TTS models for voice cloning and dubbing.

I've encountered this common problem: when dubbing a sentence, the last word spoken is often cut off, or the speech stops before the sentence ends.

I'm sure TTS models are excellent, because I've also tried them to generate simple text-to-speech. So the model itself works well.

The problem is text-to-speech during dubbing. More stringent rules should be set, such as requiring it to finish all words correctly and not cut them off. Furthermore, for example, you could leave some leeway in the timing of sentences, so that it has a safety margin to finish words and sentences.

I don't know if this is feasible, but I'm noticing that it's a shame to have all these TTS models that clone excellently, and then end up with incorrect results. Even re-dubbing often fails to produce a correct and decently pronounced sentence.

I therefore believe that, with current TTS models, the level is already very high, and there's no point in adding new ones if we don't focus on and improve the final result.

I therefore ask that we move on to a careful review of the final result, improving the speech processing. Now, in my opinion, is the time to focus. I'm sure that improving this phase will improve the processing across all the various TTS models.

I would also recommend improving the speech speed: I've often noticed that the speaker speaks slowly, and as a result, there's not enough time to pronounce all the words in the sentence.
In fact, I would even consider setting a default speed-up rule to ensure that words aren't cut off (I don't know, it's just an idea).

Unfortunately, the result isn't usable today.

Dominant language
C#
Stars
14.3k
Forks
1.3k
Avg merge
1h 19m
Merged PRs (30d)
782

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from SubtitleEdit/subtitleedit

All issues in SubtitleEdit/subtitleedit

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.