mlc-ai / mlc-ai/tokenizers-cpp

Abort is not a great error handling strategy

Open
#50 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
512
Forks
132
PR merge metrics
No merged PRs in 30d

Description

Calling these two functions with illegal data causes a stack-trace/abort in Rust code (when it tries to unwrap a failing error object):

::tokenizers::Tokenizer::FromBlobJSON(json_blob)
::tokenizers::Tokenizer::FromBlobByteLevelBPE(vocab_blob, merges_blob, added_tokens)

The sentence piece initializer doesn't fail but it is probably just silently not parsing the protobuf:

::tokenizers::Tokenizer::FromBlobSentencePiece(model_blob)

Such functions in a library aren't a great design: it would be better to at least have a simple way to marshal the failure to create to the user. For code that is trying to be light-weight and not use exceptions, I would probably use a signature like:

static std::unique_ptr<Tokenizer> FromBlobJSON(const std::string& json_blob, std::string &error_message);

And define it such that returning a nullptr indicates error and the library makes some attempt to populate error_message in this case. Since you likely already have things calling these and those aren't expecting a nullptr return, maybe rename the to a WithError suffix or something.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting Tokenizer::FromBlobJSON, FromBlobByteLevelBPE, and FromBlobSentencePiece, along with their existing callers. Reproduce the illegal-data cases described in the issue and determine how failures are currently unwrapped or ignored. Done means these initializers provide a documented failure path without aborting, while preserving or deliberately updating existing caller behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, rust
Domain
api
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.