huggingface / huggingface/blog

How to introduce new alphabets in Whisper fine-tuning

Open
#1,702 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
3.5k
Forks
1k
Avg merge
1d 20h
Merged PRs (30d)
19

Description

Dear @sanchit-gandhi,
I was following your tutorial, [Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers](https://huggingface.co/blog/fine-tune-whisper), to fine-tune Whisper with a dataset in the Amharic language. Amharic is used in Whisper training as speech-translation only, [Amharic audio -> corresponding English translation text]. Hence the Amharic alphabets are unseen in Whisper training.
The dataset I am trying to fine-tune with is [Amharic audio -> corresponding text in Amharic characters]. It consists of 92.28 hours (32901 instances) for training and 9.12 hours (3139 instances) for the testing set.
My data sources are:
1. https://github.com/getalp/ALFFA_PUBLIC/tree/master/ASR/AMHARIC and
2. https://www.findke.ovgu.de/findke/en/Research/Data+Sets/Amharic+Speech+Corpus.html

I tried the tiny, base, and small model sizes. In my first run with whisper-small, I observed a bad performance but when tried to play around with some parameters, including the model size, I was unable to run the code even.
I am not quite sure how to introduce the Amharic language characters other than giving the corresponding text as I have seen in the Hindi example.
I would appreciate your comment regarding the language whose characters were not seen in the Whisper training because it was treated as a speech translation only.
Thank you!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked Fine-Tune Whisper For Multilingual ASR tutorial, then compare its Hindi example with the Amharic dataset and the reported tiny, base, and small model runs. Done means providing a clear, reproducible explanation of how to handle unseen Amharic characters during fine-tuning, including the relevant parameter or preprocessing guidance.

Written by the indexing model from the issue text.

Assessment

Domain
machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.