huggingface / huggingface/swift-transformers

Whisper normalization for evals

Open
#47 0 comments 1 reaction 0 assignees View on GitHub
tokenization
Dominant language
Swift
Stars
1.4k
Forks
209
Avg merge
2d 7h
Merged PRs (30d)
4

Description

The `transformers` version of the Whisper tokenizer has an `EnglishTextNormalizer` (https://github.com/huggingface/transformers/blob/d9deddb4c18410a14952537a91099319ecedb869/src/transformers/models/whisper/tokenization_whisper.py#L529) that is initialized with the contents of [this file](https://github.com/huggingface/transformers/blob/d9deddb4c18410a14952537a91099319ecedb869/src/transformers/models/whisper/tokenization_whisper.py#L42). There's also a `BasicTextNormalizer` and some additional stuff.

These normalizers are not applied during regular use of the tokenizer. They can be enabled by [passing custom flags to `decode`](https://github.com/huggingface/transformers/blob/d9deddb4c18410a14952537a91099319ecedb869/src/transformers/models/whisper/tokenization_whisper.py#L662). This usually happens during quality evaluation, as [explained in this PR](https://github.com/huggingface/transformers/pull/28136), or as seen in the [Open ASR leaderboard](https://github.com/huggingface/open_asr_leaderboard/blob/main/normalizer/normalizer.py), which contains a [hardcoded version of the English normalization file](https://github.com/huggingface/open_asr_leaderboard/blob/main/normalizer/english_abbreviations.py).

It'd be interesting to add these features as opt-in capabilities, but they are really not required until we want to run evaluations in Swift. Opening this issue for future reference.

h/t @ZachNagengast for his help diving into this.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.