mudler / mudler/magpie-tts.cpp

Arabic TTS reads no digits at all, not even Arabic-Indic digits

Open
#1 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
13
Forks
4
PR merge metrics
No merged PRs in 30d

Description

Problem
When using Arabic TTS (ar-MSA), no digit system works — not ASCII (0-9), not Arabic-Indic (٠-٩). All digits are silently dropped.
Example
Arabic-Indic digits (٠-٩) — native Arabic digits
./magpie-cli say --model ... --lang ar-MSA
--text "وزارة العمل تعلن عن ٨٥٥ فرصة عمل حتى سن ٥٠ عاما بـ ٣ مدن"
Audio Output: "وزارة العمل تعلن عن فرصة عمل حتى سن عاما بـ مدن" (ALL numbers missing)

[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '٨' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '٥' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '٥' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '٥' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '٠' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char 'ـ' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '٣' skipped

ASCII digits (0-9)
./magpie-cli say --model ... --lang ar-MSA
--text "وزارة العمل تعلن عن 855 فرصة عمل"
Audio Output: "وزارة العمل تعلن عن فرصة عمل" (numbers missing)
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '8' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '5' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '5' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '5' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '0' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char 'ـ' skipped
[magpie] tokenizer arabic_MSA_chartokenizer: unknown char '3' skipped

Possible Root Cause [suggested by opencode/nemotron-3-ultra]
The ArabicCharsTokenizer (charset v1) uses a fixed character vocabulary loaded from the GGUF model. This vocabulary appears to not include any digit characters at all — not even Arabic-Indic digits (٠-٩, U+0660–U+0669).
The tokenizer does per-character lookup (st.in_vocab(c) at tokenizer.cpp:580) and drops any character not in the vocabulary with a warning log.
Expected Behavior
Arabic TTS should read numbers written in Arabic-Indic digits (٠-٩) at minimum, since that's the standard digit system for Arabic text.
Affected Languages

  • ar-MSA (Modern Standard Arabic)

it uses the same ArabicCharsTokenizer with "fixed charset v1" (per architecture doc).
Current Workaround
None — there is no way to synthesize spoken numbers in Arabic with the current model/tokenizer. Numbers must be written out in words (e.g., "ثمانمائة وخمسة وخمسون" instead of "855" or "٨٥٥").

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ArabicCharsTokenizer and the per-character lookup at tokenizer.cpp:580, then consult the architecture documentation for the fixed charset v1 model vocabulary. Reproduce the ar-MSA examples with both Arabic-Indic and ASCII digits and trace the unknown-character warnings. Done means digits are no longer dropped and Arabic TTS synthesizes the numbers in those inputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.