bugbakery / bugbakery/transcribee
Better handling for invalid utf-8
- Dominant language
- TypeScript
- Stars
- 515
- Forks
- 39
- Avg merge
- 19h 36m
- Merged PRs (30d)
- 15
Description
Currently we store all text of a `Atom` as utf-8. This interacts badly with whisper, as not every token generated by it is a valid utf-8 sequence.
There are two cases:
1. Back to back tokens generated by whisper are valid utf-8, but they are not valid utf-8 on their own.
2. The tokens are just completely invalid.
The first case is currently handled by combining the tokens generated by whisper into a `Atom` until the combined text is valid utf-8. This does however not solve the second case and will just cause the whole `Paragraph` to be empty. (As no `Atom` will ever be emitted for a segment.)
Furthermore the handling for the first case assumes these issues are always contained in a single `Segment`. This might not always be true.
I see two ways forward:
1. Save all text as bytes and only decode as utf-8 string whenever necessary.
2. Add more sophisticated handling for the cases where the generated byte stream is not valid utf-8.
The first option would allow "lossless" storage of everything generated by whisper, however it is unclear how to interpret the invalid utf-8 sequences.
Contributor guide
Research direction
Trace how whisper output moves through Atom, Segment, and Paragraph handling. Compare the two proposed approaches: byte storage with decoding when needed, or more sophisticated invalid UTF-8 handling. Done means completely invalid tokens no longer make the whole Paragraph empty, and invalid sequences spanning Segments are handled correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100