bugbakery / bugbakery/transcribee

Non speech text

Open
#25 1 comment 0 reactions 0 assignees View on GitHub
discuss
Dominant language
TypeScript
Stars
515
Forks
39
Avg merge
19h 36m
Merged PRs (30d)
15

Description

There are several non speech segments generated by whisper.
Broadly these come in three classes:
- Punctuation
- Special tokens (like `start of transcript`, `end of transcript`, etc)
- Special annotations (like `*Musik*`, `...`, etc)

The first two classes are easy to detect and if necessary for some processing step split out. Punctuation is sampled from a limited class of possible characters (like `.`, ` `, `,`, `-`, ...) and the special tokens have specific token IDs that can be filtered out.

The third class however is generated just like "normal" transcript text. The official whisper implementation seems to have some heuristics to filter them: https://github.com/openai/whisper/blob/ad3250a846fe7553a25064a2dc593e492dadf040/whisper/tokenizer.py#L237
However this also just looks like some basic heuristic, that will not work for `*Musik*` for example.

Do we care about this third class? Are there cases where we want to filter these?
One case where I think it could be useful to filter them is the alignment, but it is to be determined how important this is.

Contributor guide

Open the contributing guide

Research direction

Start with the referenced Whisper tokenizer.py heuristics and compare their handling of punctuation, special tokens, and annotation-like text such as `*Musik*`. Investigate how these segments affect alignment in this project. Done means establishing whether the third class should be filtered and defining the intended behavior and scope.

Written by the indexing model from the issue text.

Assessment

Domain
ai
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.