mlc-ai / mlc-ai/tokenizers-cpp

Building from the release tag v0.1.1 will fail with errors

Open
#94 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
512
Forks
132
PR merge metrics
No merged PRs in 30d

Description

I tried building from the release tag v0.1.1 like this:

git clone --depth 1 --branch v0.1.1 --recurse-submodules https://github.com/mlc-ai/tokenizers-cpp tokenizers-cpp-0.1.1
cmake -S tokenizers-cpp-0.1.1 -B tokenizers-cpp-0.1.1/build -DCMAKE_BUILD_TYPE=RelWithDebInfo
cmake --build tokenizers-cpp-0.1.1/build

, and the build fails with the error below.

error[E0308]: mismatched types
   --> src/lib.rs:78:53
    |
 78 |         let mut tokenizer = Tokenizer::new(BPE::new(vocab, merges));
    |                                            -------- ^^^^^ expected `AHashMap<String, u32>`, found `HashMap<String, u32>`
    |                                            |
    |                                            arguments to this function are incorrect
    |
    = note: expected struct `ahash::hash_map::AHashMap<std::string::String, u32>`
               found struct `HashMap<std::string::String, u32>`

It seems that the submodules referenced in this release tag are not compatible with the source code at the same tag.

The build succeeds at the current main, but building from main is error-prone, so I ended up building from the current commit of main, but I'm not sure if I'm picking up something that is not supposed to be in the v0.1.1 release.

git clone https://github.com/mlc-ai/tokenizers-cpp tokenizers-cpp-0.1.1
cd tokenizers-cpp-0.1.1
git checkout acbdc5a2
git submodule update --init
cd ..

It might be a good idea to move the release tag to the commit that fixed those dependencies.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure using the v0.1.1 clone and CMake commands in the issue, then inspect src/lib.rs at line 78 and the submodule revisions referenced by that tag. Compare them with the current main commit and its submodules. Done means a clean checkout of v0.1.1 builds successfully, or the release tag is corrected as appropriate.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp, git, rust
Domain
build-system, release
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.