mlc-ai / mlc-ai/tokenizers-cpp

Problem resolving some symbols when using the library in an Android C++ project (I am compiling using ndk)

Open
#31 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
512
Forks
132
PR merge metrics
No merged PRs in 30d

Description

I was able to successfully compile the library but when I use it as indicated in the example folder I am having the following errors:

ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::SentencePieceProcessor()

referenced by sentencepiece_tokenizer.cc:18 (src/sentencepiece_tokenizer.cc:18)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::SentencePieceTokenizer(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a

ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::LoadFromSerializedProto(std::__ndk1::basic_string_view<char, std::__ndk1::char_traits >)

referenced by sentencepiece_tokenizer.cc:19 (src/sentencepiece_tokenizer.cc:19)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::SentencePieceTokenizer(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a

ld: error: undefined symbol: sentencepiece::util::Status::~Status()

referenced by sentencepiece_tokenizer.cc:19 (src/sentencepiece_tokenizer.cc:19)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::SentencePieceTokenizer(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
referenced by sentencepiece_tokenizer.cc:24 (src/sentencepiece_tokenizer.cc:24)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Encode(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
referenced by sentencepiece_tokenizer.cc:24 (src/sentencepiece_tokenizer.cc:24)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Encode(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
referenced 2 more times

ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::~SentencePieceProcessor()

referenced by sentencepiece_tokenizer.cc:20 (src/sentencepiece_tokenizer.cc:20)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::SentencePieceTokenizer(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
referenced by sentencepiece_tokenizer.cc:16 (src/sentencepiece_tokenizer.cc:16)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::~SentencePieceTokenizer()) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a

ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::Encode(std::__ndk1::basic_string_view<char, std::__ndk1::char_traits >, std::__ndk1::vector<int, std::__ndk1::allocator >*) const

referenced by sentencepiece_tokenizer.cc:24 (src/sentencepiece_tokenizer.cc:24)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Encode(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a

ld: error: undefined symbol: sentencepiece::util::Status::IgnoreError()

referenced by sentencepiece_tokenizer.cc:24 (src/sentencepiece_tokenizer.cc:24)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Encode(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
referenced by sentencepiece_tokenizer.cc:30 (src/sentencepiece_tokenizer.cc:30)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Decode(std::__ndk1::vector<int, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a

ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::Decode(std::__ndk1::vector<int, std::__ndk1::allocator > const&, std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator >*) const

referenced by sentencepiece_tokenizer.cc:30 (src/sentencepiece_tokenizer.cc:30)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::Decode(std::__ndk1::vector<int, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a

ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::GetPieceSize() const

referenced by sentencepiece_tokenizer.cc:35 (src/sentencepiece_tokenizer.cc:35)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::GetVocabSize()) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a

ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::IdToPiece(int) const

referenced by sentencepiece_tokenizer.cc:40 (src/sentencepiece_tokenizer.cc:40)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::IdToToken(int)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a

ld: error: undefined symbol: sentencepiece::SentencePieceProcessor::PieceToId(std::__ndk1::basic_string_view<char, std::__ndk1::char_traits >) const

referenced by sentencepiece_tokenizer.cc:42 (src/sentencepiece_tokenizer.cc:42)
sentencepiece_tokenizer.cc.o:(tokenizers::SentencePieceTokenizer::TokenToId(std::__ndk1::basic_string<char, std::__ndk1::char_traits, std::__ndk1::allocator > const&)) in archive ./src/tokenizers-cpp/libtokenizers_cpp.a
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

When I check inside the library the symbols are properly defined.

In my code I am just doing the same as in the example folder, so I am not invoking directly the symbols that are not recognized. The ones that I am using (FromBlobSentencePiece, for example) are correctly identified. What could be the error?

One things which is curious for me is: why the compiler of my program is complaining about the src/sentencepiece_tokenizer.cc file if I am just using the static library (the .a file) through the tokenizers_cpp.h file provided by the library?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Android example and the library-linking setup, then inspect src/sentencepiece_tokenizer.cc, tokenizers_cpp.h, and the generated static .a file. Reproduce the NDK link command and compare which SentencePiece symbols are available and linked. Done means the Android C++ example links successfully and runs without the reported undefined-symbol errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, cpp
Domain
build-system, mobile-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.