mlc-ai / mlc-ai/tokenizers-cpp
Problem resolving some symbols when using the library in an Android C++ project (I am compiling using ndk)
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 512
- Forks
- 132
- PR merge metrics
- No merged PRs in 30d
Description
I was able to successfully compile the library but when I use it as indicated in the example folder I am having the following errors:
ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::SentencePieceProcessor()
referenced by sentencepiece_tokenizer.cc:18 (src/sentencepiece_tokenizer.cc:18)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::SentencePieceTokenizer(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::LoadFromSerializedProto(std::__ndk1::basic_string_view<char, std::__ndk1::char_traits >)
referenced by sentencepiece_tokenizer.cc:19 (src/sentencepiece_tokenizer.cc:19)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::SentencePieceTokenizer(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
ld: error: undefined symbol: sentencepiece::util::Status::~Status()
referenced by sentencepiece_tokenizer.cc:19 (src/sentencepiece_tokenizer.cc:19)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::SentencePieceTokenizer(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
referenced by sentencepiece_tokenizer.cc:24 (src/sentencepiece_tokenizer.cc:24)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Encode(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
referenced by sentencepiece_tokenizer.cc:24 (src/sentencepiece_tokenizer.cc:24)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Encode(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
referenced 2 more times
ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::~SentencePieceProcessor()
referenced by sentencepiece_tokenizer.cc:20 (src/sentencepiece_tokenizer.cc:20)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::SentencePieceTokenizer(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
referenced by sentencepiece_tokenizer.cc:16 (src/sentencepiece_tokenizer.cc:16)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::~SentencePieceTokenizer()) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::Encode(std::__ndk1::basic_string_view<char, std::__ndk1::char_traits >, std::__ndk1::vector<int, std::__ndk1::allocator >*) const
referenced by sentencepiece_tokenizer.cc:24 (src/sentencepiece_tokenizer.cc:24)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Encode(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
ld: error: undefined symbol: sentencepiece::util::Status::IgnoreError()
referenced by sentencepiece_tokenizer.cc:24 (src/sentencepiece_tokenizer.cc:24)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Encode(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
referenced by sentencepiece_tokenizer.cc:30 (src/sentencepiece_tokenizer.cc:30)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Decode(std::__ndk1::vector<int, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::Decode(std::__ndk1::vector<int, std::__ndk1::allocator > const&, std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator >*) const
referenced by sentencepiece_tokenizer.cc:30 (src/sentencepiece_tokenizer.cc:30)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Decode(std::__ndk1::vector<int, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::GetPieceSize() const
referenced by sentencepiece_tokenizer.cc:35 (src/sentencepiece_tokenizer.cc:35)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::GetVocabSize()) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::IdToPiece(int) const
referenced by sentencepiece_tokenizer.cc:40 (src/sentencepiece_tokenizer.cc:40)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::IdToToken(int)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::PieceToId(std::__ndk1::basic_string_view<char, std::__ndk1::char_traits >) const
referenced by sentencepiece_tokenizer.cc:42 (src/sentencepiece_tokenizer.cc:42)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::TokenToId(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
When I check inside the library the symbols are properly defined.
In my code I am just doing the same as in the example folder, so I am not invoking directly the symbols that are not recognized. The ones that I am using (FromBlobSentencePiece, for example) are correctly identified. What could be the error?
One things which is curious for me is: why the compiler of my program is complaining about the src/sentencepiece_tokenizer.cc file if I am just using the static library (the .a file) through the tokenizers_cpp.h file provided by the library?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Android example and the library-linking setup, then inspect src/sentencepiece_tokenizer.cc, tokenizers_cpp.h, and the generated static .a file. Reproduce the NDK link command and compare which SentencePiece symbols are available and linked. Done means the Android C++ example links successfully and runs without the reported undefined-symbol errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, cpp
- Domain
- build-system, mobile-dev
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100