huggingface / huggingface/blog
How to introduce new alphabets in Whisper fine-tuning
- Dominant language
- Jupyter Notebook
- Stars
- 3.5k
- Forks
- 1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 19
Description
Dear @sanchit-gandhi,
I was following your tutorial, [Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers](https://huggingface.co/blog/fine-tune-whisper), to fine-tune Whisper with a dataset in the Amharic language. Amharic is used in Whisper training as speech-translation only, [Amharic audio -> corresponding English translation text]. Hence the Amharic alphabets are unseen in Whisper training.
The dataset I am trying to fine-tune with is [Amharic audio -> corresponding text in Amharic characters]. It consists of 92.28 hours (32901 instances) for training and 9.12 hours (3139 instances) for the testing set.
My data sources are:
1. https://github.com/getalp/ALFFA_PUBLIC/tree/master/ASR/AMHARIC and
2. https://www.findke.ovgu.de/findke/en/Research/Data+Sets/Amharic+Speech+Corpus.html
I tried the tiny, base, and small model sizes. In my first run with whisper-small, I observed a bad performance but when tried to play around with some parameters, including the model size, I was unable to run the code even.
I am not quite sure how to introduce the Amharic language characters other than giving the corresponding text as I have seen in the Hindi example.
I would appreciate your comment regarding the language whose characters were not seen in the Whisper training because it was treated as a speech translation only.
Thank you!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked Fine-Tune Whisper For Multilingual ASR tutorial, then compare its Hindi example with the Amharic dataset and the reported tiny, base, and small model runs. Done means providing a clear, reproducible explanation of how to handle unseen Amharic characters during fine-tuning, including the relevant parameter or preprocessing guidance.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100