mlc-ai / mlc-ai/tokenizers-cpp
Abort is not a great error handling strategy
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 512
- Forks
- 132
- PR merge metrics
- No merged PRs in 30d
Description
Calling these two functions with illegal data causes a stack-trace/abort in Rust code (when it tries to unwrap a failing error object):
::tokenizers::Tokenizer::FromBlobJSON(json_blob)
::tokenizers::Tokenizer::FromBlobByteLevelBPE(vocab_blob, merges_blob, added_tokens)
The sentence piece initializer doesn't fail but it is probably just silently not parsing the protobuf:
::tokenizers::Tokenizer::FromBlobSentencePiece(model_blob)
Such functions in a library aren't a great design: it would be better to at least have a simple way to marshal the failure to create to the user. For code that is trying to be light-weight and not use exceptions, I would probably use a signature like:
static std::unique_ptr<Tokenizer> FromBlobJSON(const std::string& json_blob, std::string &error_message);
And define it such that returning a nullptr indicates error and the library makes some attempt to populate error_message in this case. Since you likely already have things calling these and those aren't expecting a nullptr return, maybe rename the to a WithError suffix or something.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting Tokenizer::FromBlobJSON, FromBlobByteLevelBPE, and FromBlobSentencePiece, along with their existing callers. Reproduce the illegal-data cases described in the issue and determine how failures are currently unwrapped or ignored. Done means these initializers provide a documented failure path without aborting, while preserving or deliberately updating existing caller behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, rust
- Domain
- api
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100