mlc-ai / mlc-ai/tokenizers-cpp
Finish implementation of MLC_ENABLE_SENTENCEPIECE_TOKENIZER
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 512
- Forks
- 132
- PR merge metrics
- No merged PRs in 30d
Description
There is a cmake option called MLC_ENABLE_SENTENCEPIECE_TOKENIZER but as today, when OFF, it does nt totally disable/skip sentencepiece, eg that line still links with the sentpiece lib:
https://github.com/mlc-ai/tokenizers-cpp/blob/f77710965a3bcae85b7a00bdddbfc1adadef0e32/CMakeLists.txt#L186
Would you mind if I add some small retouches to really fully disable sentpiece when that option is OFF?
Best regards
WT
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in CMakeLists.txt around line 186 and trace how MLC_ENABLE_SENTENCEPIECE_TOKENIZER controls SentencePiece references. Verify the CMake configuration with the option OFF and confirm that SentencePiece is fully skipped, including its library link; the issue does not name a test to run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, cpp
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100