bugbakery / bugbakery/transcribee

Better handling for invalid utf-8

Open
#21 0 comments 0 reactions 0 assignees View on GitHub
discuss worker
Dominant language
TypeScript
Stars
515
Forks
39
Avg merge
19h 36m
Merged PRs (30d)
15

Description

Currently we store all text of a `Atom` as utf-8. This interacts badly with whisper, as not every token generated by it is a valid utf-8 sequence.
There are two cases:
1. Back to back tokens generated by whisper are valid utf-8, but they are not valid utf-8 on their own.
2. The tokens are just completely invalid.

The first case is currently handled by combining the tokens generated by whisper into a `Atom` until the combined text is valid utf-8. This does however not solve the second case and will just cause the whole `Paragraph` to be empty. (As no `Atom` will ever be emitted for a segment.)

Furthermore the handling for the first case assumes these issues are always contained in a single `Segment`. This might not always be true.

I see two ways forward:
1. Save all text as bytes and only decode as utf-8 string whenever necessary.
2. Add more sophisticated handling for the cases where the generated byte stream is not valid utf-8.

The first option would allow "lossless" storage of everything generated by whisper, however it is unclear how to interpret the invalid utf-8 sequences.

Contributor guide

Open the contributing guide

Research direction

Trace how whisper output moves through Atom, Segment, and Paragraph handling. Compare the two proposed approaches: byte storage with decoding when needed, or more sophisticated invalid UTF-8 handling. Done means completely invalid tokens no longer make the whole Paragraph empty, and invalid sequences spanning Segments are handled correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.