huggingface / huggingface/swift-transformers
Whisper normalization for evals
- Dominant language
- Swift
- Stars
- 1.4k
- Forks
- 209
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 4
Description
The `transformers` version of the Whisper tokenizer has an `EnglishTextNormalizer` (https://github.com/huggingface/transformers/blob/d9deddb4c18410a14952537a91099319ecedb869/src/transformers/models/whisper/tokenization_whisper.py#L529) that is initialized with the contents of [this file](https://github.com/huggingface/transformers/blob/d9deddb4c18410a14952537a91099319ecedb869/src/transformers/models/whisper/tokenization_whisper.py#L42). There's also a `BasicTextNormalizer` and some additional stuff.
These normalizers are not applied during regular use of the tokenizer. They can be enabled by [passing custom flags to `decode`](https://github.com/huggingface/transformers/blob/d9deddb4c18410a14952537a91099319ecedb869/src/transformers/models/whisper/tokenization_whisper.py#L662). This usually happens during quality evaluation, as [explained in this PR](https://github.com/huggingface/transformers/pull/28136), or as seen in the [Open ASR leaderboard](https://github.com/huggingface/open_asr_leaderboard/blob/main/normalizer/normalizer.py), which contains a [hardcoded version of the English normalization file](https://github.com/huggingface/open_asr_leaderboard/blob/main/normalizer/english_abbreviations.py).
It'd be interesting to add these features as opt-in capabilities, but they are really not required until we want to run evaluations in Swift. Opening this issue for future reference.
h/t @ZachNagengast for his help diving into this.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.