mlc-ai / mlc-ai/tokenizers-cpp
Building from the release tag v0.1.1 will fail with errors
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 512
- Forks
- 132
- PR merge metrics
- No merged PRs in 30d
Description
I tried building from the release tag v0.1.1 like this:
git clone --depth 1 --branch v0.1.1 --recurse-submodules https://github.com/mlc-ai/tokenizers-cpp tokenizers-cpp-0.1.1
cmake -S tokenizers-cpp-0.1.1 -B tokenizers-cpp-0.1.1/build -DCMAKE_BUILD_TYPE=RelWithDebInfo
cmake --build tokenizers-cpp-0.1.1/build
, and the build fails with the error below.
error[E0308]: mismatched types
--> src/lib.rs:78:53
|
78 | let mut tokenizer = Tokenizer::new(BPE::new(vocab, merges));
| -------- ^^^^^ expected `AHashMap<String, u32>`, found `HashMap<String, u32>`
| |
| arguments to this function are incorrect
|
= note: expected struct `ahash::hash_map::AHashMap<std::string::String, u32>`
found struct `HashMap<std::string::String, u32>`
It seems that the submodules referenced in this release tag are not compatible with the source code at the same tag.
The build succeeds at the current main, but building from main is error-prone, so I ended up building from the current commit of main, but I'm not sure if I'm picking up something that is not supposed to be in the v0.1.1 release.
git clone https://github.com/mlc-ai/tokenizers-cpp tokenizers-cpp-0.1.1
cd tokenizers-cpp-0.1.1
git checkout acbdc5a2
git submodule update --init
cd ..
It might be a good idea to move the release tag to the commit that fixed those dependencies.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure using the v0.1.1 clone and CMake commands in the issue, then inspect src/lib.rs at line 78 and the submodule revisions referenced by that tag. Compare them with the current main commit and its submodules. Done means a clean checkout of v0.1.1 builds successfully, or the release tag is corrected as appropriate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, cpp, git, rust
- Domain
- build-system, release
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100