bytedance / bytedance/MegaTTS3

Part of sentences are stripped from the audio output.

Open
#83 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.1k
Forks
474
PR merge metrics
No merged PRs in 30d

Description

I'm executing this CMD with the default English_prompt.wav

`
python ./tts/infer_cli.py --input_wav ./assets/English_prompt.wav --input_text "As his long promised tariff threat turned into reality this week, top human advisers began fielding a wave of calls from business leaders, particularly in the automotive sector, along with lawmakers who were sounding the alarm." --output_dir ./gen --p_w 2.0 --t_w 3.0
`

the result is the part of text are stripped "...along with lawmakers who were sounding the alarm"

![Image](https://github.com/user-attachments/assets/0259aff7-b541-45b1-b88c-4dbfc60c6568)

then I run again with my own local voice

`
python ./tts/infer_cli.py --input_wav ./assets/sample_zzz_en.wav --input_text "As his long promised tariff threat turned into reality this week, top human advisers began fielding a wave of calls from business leaders, particularly in the automotive sector, along with lawmakers who were sounding the alarm." --output_dir ./gen --p_w 2.0 --t_w 3.0
`

now the first part of the text are stripped "As his long promised tariff threat turned into reality this week,.."

![Image](https://github.com/user-attachments/assets/866266f2-25b9-4dab-bdea-005c562d6579)

here's the generated audio sample

https://drive.google.com/file/d/1m7tB2YqaEchS1Qskp-7pmH_JDpEEC_Nw/view?usp=sharing

as for the log for the later cmd:

~~~
| loaded 'dur_model' from './checkpoints\duration_lm/model_only_last.ckpt'.
| Missing keys: 0, Unexpected keys: 0
| loaded 'dit' from './checkpoints\diffusion_transformer/model_only_last.ckpt'.
| Missing keys: 0, Unexpected keys: 9
| loaded 'model' from './checkpoints\aligner_lm/model_only_last.ckpt'.
| Missing keys: 0, Unexpected keys: 0
D:\DEVSDK\anaconda3\envs\megatts3-env\lib\site-packages\torch\nn\utils\weight_norm.py:143: FutureWarning: `torch.nn.utils.weight_norm` is deprecated in favor of `torch.nn.utils.parametrizations.weight_norm`.
WeightNorm.apply(module, name, dim)
| loaded 'model_gen' from './checkpoints\wavvae/decoder.ckpt'.
| Missing keys: 74, Unexpected keys: 0
2025-04-12 18:00:00,588 WETEXT INFO found existing fst: D:\DEVSDK\anaconda3\envs\megatts3-env\lib\site-packages\tn\zh_tn_tagger.fst
2025-04-12 18:00:00,588 WETEXT INFO D:\DEVSDK\anaconda3\envs\megatts3-env\lib\site-packages\tn\zh_tn_verbalizer.fst
2025-04-12 18:00:00,588 WETEXT INFO skip building fst for zh_normalizer ...
2025-04-12 18:00:01,268 WETEXT INFO found existing fst: D:\DEVSDK\anaconda3\envs\megatts3-env\lib\site-packages\tn\en_tn_tagger.fst
2025-04-12 18:00:01,268 WETEXT INFO D:\DEVSDK\anaconda3\envs\megatts3-env\lib\site-packages\tn\en_tn_verbalizer.fst
2025-04-12 18:00:01,268 WETEXT INFO skip building fst for en_normalizer ...
| Start processing ./assets/sample_zzz_en.wav+As his long promised tariff threat turned into reality this week, top human advisers began fielding a wave of calls from business leaders, particularly in the automotive sector, along with lawmakers who were sounding the alarm.
D:\Repo_AI\bytedanceMegaTTS3\tts\frontend_function.py:49: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(dtype=self.precision, enabled=True):
D:\Repo_AI\bytedanceMegaTTS3\tts\frontend_function.py:88: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(dtype=self.precision, enabled=True):
D:\Repo_AI\bytedanceMegaTTS3\tts\frontend_function.py:30: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(dtype=self.precision, enabled=True):
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:152468 for open-end generation.
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
D:\Repo_AI\bytedanceMegaTTS3\tts\frontend_function.py:104: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(dtype=self.precision, enabled=True):
D:\Repo_AI\bytedanceMegaTTS3\tts\infer_cli.py:232: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(dtype=self.precision, enabled=True):
D:\Repo_AI\bytedanceMegaTTS3\tts\frontend_function.py:30: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(dtype=self.precision, enabled=True):
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:152468 for open-end generation.
D:\Repo_AI\bytedanceMegaTTS3\tts\frontend_function.py:104: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(dtype=self.precision, enabled=True):
D:\Repo_AI\bytedanceMegaTTS3\tts\infer_cli.py:232: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(dtype=self.precision, enabled=True):
D:\Repo_AI\bytedanceMegaTTS3\tts\frontend_function.py:30: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(dtype=self.precision, enabled=True):
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:152468 for open-end generation.
D:\Repo_AI\bytedanceMegaTTS3\tts\frontend_function.py:104: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(dtype=self.precision, enabled=True):
D:\Repo_AI\bytedanceMegaTTS3\tts\infer_cli.py:232: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(dtype=self.precision, enabled=True):
D:\Repo_AI\bytedanceMegaTTS3\tts\utils\audio_utils\io.py:35: RuntimeWarning: invalid value encountered in cast
wavfile.write(bytes_io, sr, wav.astype(np.int16))
| Saving results to ./gen/[P]As his long promised.wav
~~~

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the command in tts/infer_cli.py with both referenced prompt WAV files and compare the generated audio with the full input text. Then inspect tts/frontend_function.py and tts/utils/audio_utils/io.py, including the logged generation and saving paths. Done means the output preserves the complete sentence for both prompt voices without the reported truncation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
audio-video-rtc, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.