mudler / mudler/magpie-tts.cpp
Arabic TTS reads no digits at all, not even Arabic-Indic digits
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 13
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
Problem
When using Arabic TTS (ar-MSA), no digit system works — not ASCII (0-9), not Arabic-Indic (٠-٩). All digits are silently dropped.
Example
Arabic-Indic digits (٠-٩) — native Arabic digits
./magpie-cli say --model ... --lang ar-MSA
--text "وزارة العمل تعلن عن ٨٥٥ فرصة عمل حتى سن ٥٠ عاما بـ ٣ مدن"
Audio Output: "وزارة العمل تعلن عن فرصة عمل حتى سن عاما بـ مدن" (ALL numbers missing)
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '٨' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '٥' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '٥' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '٥' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '٠' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char 'ـ' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '٣' skipped
ASCII digits (0-9)
./magpie-cli say --model ... --lang ar-MSA
--text "وزارة العمل تعلن عن 855 فرصة عمل"
Audio Output: "وزارة العمل تعلن عن فرصة عمل" (numbers missing)
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '8' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '5' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '5' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '5' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '0' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char 'ـ' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '3' skipped
Possible Root Cause [suggested by opencode/nemotron-3-ultra]
The ArabicCharsTokenizer (charset v1) uses a fixed character vocabulary loaded from the GGUF model. This vocabulary appears to not include any digit characters at all — not even Arabic-Indic digits (٠-٩, U+0660–U+0669).
The tokenizer does per-character lookup (st.in_vocab(c) at tokenizer.cpp:580) and drops any character not in the vocabulary with a warning log.
Expected Behavior
Arabic TTS should read numbers written in Arabic-Indic digits (٠-٩) at minimum, since that's the standard digit system for Arabic text.
Affected Languages
- ar-MSA (Modern Standard Arabic)
it uses the same ArabicCharsTokenizer with "fixed charset v1" (per architecture doc).
Current Workaround
None — there is no way to synthesize spoken numbers in Arabic with the current model/tokenizer. Numbers must be written out in words (e.g., "ثمانمائة وخمسة وخمسون" instead of "855" or "٨٥٥").
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ArabicCharsTokenizer and the per-character lookup at tokenizer.cpp:580, then consult the architecture documentation for the fixed charset v1 model vocabulary. Reproduce the ar-MSA examples with both Arabic-Indic and ASCII digits and trace the unknown-character warnings. Done means digits are no longer dropped and Arabic TTS synthesizes the numbers in those inputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100